The magic of systemd-creds

Updated on: November 1, 2022

Carl Tashian

TL;DR The systemd suite's new systemd-creds feature fills an important security gap: It allows you to protect service credentials using a TPM, rather than storing them on disk in plaintext. This is perfect for storing a service's private keys, TLS certificates, account credentials, or passwords! But there are several subtleties around how to set it up and use it securely. In this tutorial, we'll use systemd-creds to secure the CA password for step-ca Certificate Authority server on a Google Cloud VM.

Background

Like the Secure Enclave on Apple devices, Trusted Platform Module (TPM) chips protect private keys at the hardware level. And the TPM 2.0 interface is now pretty much ubiquitous in server hardware and cloud VMs. Google Cloud rolled out virtual TPMs as part of its Shielded VM features in 2018, and since 2020 vTPMs have been enabled by default. In May, Amazon rolled out its NitroTPM support for EC2 instances. TPMs are everywhere!

In Linux, the systemd suite has added a lot of TPM2 support over the past year or so. Services managed by systemd can now leverage a TPM for protecting credentials, bind them to your specific device and software configuration. You can encrypt certificates, passwords, PEM keys, or any other sensitive material needed to start up a server.

So, let's use this feature to encrypt a CA password for the step-ca online Certificate Authority daemon.

Prerequisites

For this project you will need:

A Linux machine (or virtual environment) with TPM 2.0 support:

$ sudo dmesg | grep TPM
[    0.018942] ACPI: TPM2 0x00000000BEB7B000 000034 (v04 GOOGLE          00000001 GOOG 00000001)
[    0.018968] ACPI: Reserving TPM2 table memory at [mem 0xbeb7b000-0xbeb7b033]
[    3.676084] systemd[1]: systemd 251.2-1-arch running in system mode (+PAM +AUDIT -SELINUX -APPARMOR -IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified)
[    6.842149] tpm_tis MSFT0101:00: 2.0 TPM (device-id 0x9009, rev-id 0)

For this tutorial, I'll be using a Google Cloud Platform account and project. To follow along, you'll need the gcloud command on your local computer (brew install google-cloud-sdk) that you've authenticated to your Google Cloud Platform account.
You'll need systemd 250+, preferably 251. If you're running systemd 251, you can confirm your system is ready to use systemd credentials by running systemd-creds has-tpm2. It should return a simple yes.

For production, use an environment with UEFI Secure Boot enabled:

$ sudo dmesg | grep Secure
[    0.000000] secureboot: Secure boot enabled
[    0.000000] Kernel is locked down from EFI Secure Boot mode; see man kernel_lockdown.7
[    0.023818] secureboot: Secure boot enabled

(For testing, it's OK if secure boot isn't enabled.)

For production, use an encrypted /var. By default systemd-creds splits its master private key across both the TPM and a file on disk, and requires both keys to decrypt a service credential.

(Why not just use the TPM alone? You can. But in many cases an attacker could install the same bootloader and OS kernel on another volume on your machine, and gain access to the TPM key.)

Diving In

So that anyone can try this out, we opted to use Google Cloud for this example.

Unfortunately I wasn't able to find a cloud VM image that met all of these criteria (as of June 2022, systemd 251 is still very new), so I opted for Arch Linux without UEFI Secure Boot support. I wouldn't use this in production, but for the purpose of this tutorial it's okay; the instructions are the same.

1. Create an Arch Linux VM.

Let's make an e2-medium VM that has vTPM support enabled:

$ gcloud compute instances create arch-test \
   --project=systemd-creds \
   --zone=us-west1-c \
   --machine-type=e2-medium \
   --network-interface=network-tier=PREMIUM,subnet=default \
   --maintenance-policy=MIGRATE \
   --provisioning-model=STANDARD \
   --service-account=20EXAMPLE009-compute@developer.gserviceaccount.com \
   --scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/trace.append \
   --image-project=arch-linux-gce \
   --image-family=arch \
   --no-shielded-secure-boot \
   --shielded-vtpm \
   --shielded-integrity-monitoring \
   --reservation-affinity=any

2. Connect, update the system, and restart

$ gcloud compute ssh arch-test
[carl@arch-test ~]$ sudo pacman -Syu
...
[carl@arch-test ~]$ sudo reboot
Connection closed by remote host.

After it restarts, reconnect. You should see that TPM 2.0 is enabled and detected, and that you're running systemd 251 or higher:

[carl@arch-test ~]$ sudo dmesg | grep TPM
[    0.018548] ACPI: TPM2 0x00000000BEB7B000 000034 (v04 GOOGLE          00000001 GOOG 00000001)
[    0.018574] ACPI: Reserving TPM2 table memory at [mem 0xbeb7b000-0xbeb7b033]
[    3.051060] systemd[1]: systemd 251.2-1-arch running in system mode (+PAM +AUDIT -SELINUX -APPARMOR -IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified)
[    5.977329] tpm_tis MSFT0101:00: 2.0 TPM (device-id 0x9009, rev-id 0)

That's great. But, is systemd ready to use its TPM support? Run:

sudo systemd-creds has-tpm2

Output:

partial
-firmware
+driver
+system

Without Secure Boot, we have only "partial" TPM2 support here. We'll proceed with partial support, but in a production environment you'd want to confirm secure boot and full TPM2 support.

3. Configure `systemd-creds`

We need to set up the host credentials for systemd-creds in order to use it. Run:

sudo systemd-creds setup

Output:

Credential secret file '/var/lib/systemd/credential.secret' is not located on encrypted media, using anyway.
4096 byte credentials host key set up.

In this step, systemd-creds creates a key that's split between the TPM and the filesystem. This file will indeed be encrypted at rest by GCP, but Linux can't see that.

3. Install dependencies.

The tpm2-abrmd service is needed. This service is the TPM2 access broker. It allows different applications to share the TPM device.

sudo pacman -Sy vim tpm2-abrmd
systemctl enable --now tpm2-abrmd.service

4. Install the `step` and `step-ca` binaries.

sudo pacman -Sy step-ca step-cli
sudo ln -s /usr/bin/step-cli /usr/local/bin/step

5. Initialize your CA

To initialize your CA, run the following. Change CA_NAME, CA_HOSTNAMES (the DNS names your CA will answer on), and EMAIL to match your local setup.

export STEPPATH=/etc/step-ca
< /dev/urandom tr -dc A-Za-z0-9 | head -c40 > password.txt
CA_NAME="Smallstep"
CA_HOSTNAMES="127.0.0.1,localhost"
EMAIL="carl@smallstep.com"
sudo -E step ca init --ssh --name "$CA_NAME" \
     --dns "$CA_HOSTNAMES" \
     --address ":443" --provisioner "$EMAIL" \
     --password-file "password.txt"
sudo useradd --system --home /etc/step-ca --shell /bin/false step
sudo chown -R step:step /etc/step-ca

6. Add a systemd service for your CA.

We've created a service unit for step-ca, which you can download:

sudo curl -sL https://raw.githubusercontent.com/smallstep/certificates/master/systemd/step-ca.service \
     -o /etc/systemd/system/step-ca.service

7. Encrypt your CA credentials.

In this step we're going to make a drop-in override for the step-ca.service. The override unit will contain our encrypted credential.

sudo mkdir /etc/systemd/system/step-ca.service.d
cat<<EOF | sudo tee /etc/systemd/system/step-ca.service.d/overrides.conf
[Unit]
ConditionFileNotEmpty=
[Service]
ExecStart=
ExecStart=/usr/bin/step-ca config/ca.json --password-file \${CREDENTIALS_DIRECTORY}/step-ca
EOF
sudo systemd-creds encrypt --name=step-ca -p password.txt - | sudo tee -a /etc/systemd/system/step-ca.service.d/overrides.conf
sudo systemctl daemon-reload

8. Start `step-ca.service`

The service will start up and read its credentials from the TPM.

$ sudo systemctl start step-ca
$ sudo systemctl status step-ca
● step-ca.service - step-ca service
     Loaded: loaded (/etc/systemd/system/step-ca.service; disabled; vendor preset: disabled)
    Drop-In: /etc/systemd/system/step-ca.service.d
             └─overrides.conf
     Active: active (running) since Wed 2022-06-15 23:33:30 UTC; 2s ago
       Docs: https://smallstep.com/docs/step-ca
             https://smallstep.com/docs/step-ca/certificate-authority-server-production
   Main PID: 2176 (step-ca)
      Tasks: 7 (limit: 4694)
     Memory: 19.2M
        CPU: 133ms
     CGroup: /system.slice/step-ca.service
             └─2176 /usr/bin/step-ca config/ca.json --password-file /run/credentials/step-ca.service/step-ca
Jun 15 23:33:31 arch-test step-ca[2176]: badger 2022/06/15 23:33:31 INFO: All 0 tables opened in 0s
Jun 15 23:33:31 arch-test step-ca[2176]: 2022/06/15 23:33:31 Starting Smallstep CA/0.20.0 (linux/amd64)
Jun 15 23:33:31 arch-test step-ca[2176]: 2022/06/15 23:33:31 Documentation: https://u.step.sm/docs/ca
Jun 15 23:33:31 arch-test step-ca[2176]: 2022/06/15 23:33:31 Community Discord: https://u.step.sm/discord
Jun 15 23:33:31 arch-test step-ca[2176]: 2022/06/15 23:33:31 Config file: config/ca.json
Jun 15 23:33:31 arch-test step-ca[2176]: 2022/06/15 23:33:31 The primary server URL is https://127.0.0.1:443
Jun 15 23:33:31 arch-test step-ca[2176]: 2022/06/15 23:33:31 Root certificates are available at https://127.0.0.1:443/roots.pem
Jun 15 23:33:31 arch-test step-ca[2176]: 2022/06/15 23:33:31 Additional configured hostnames: localhost
Jun 15 23:33:31 arch-test step-ca[2176]: 2022/06/15 23:33:31 X.509 Root Fingerprint: c8719fad9b7639dbc0b78fdc92def5b4b7c83725ac4a84d2fbdc271c991d8f
Jun 15 23:33:31 arch-test step-ca[2176]: 2022/06/15 23:33:31 Serving HTTPS on :443 ...

9. Finishing up

Finally, make an offline copy of the contents of password.txt.

Once you have, you can delete the plaintext file:

shred -u password.txt

A side note on PCRs

Platform Configuration Registers (PCRs) are system integrity registers, consisting of 24 protected hashes stored in a TPM.

PCRs are zeroed out when the system boots, and only operation that can be performed on a PCR is "extend." Extending is a one-way function that appends a passed-in hash to the PCR's current hash, hashes the result, and replaces the PCR with it.

PCRs record system properties into well-known PCRs, mostly during boot time. Values such as the hardware configuration, hashes of the machine's trusted firmware, and the UEFI code signing certificate database (things like the Machine Owner Key other key credentials needed for proof of a Secure Boot).

PCRs merit a deeper discussion beyond the scope of this tutorial. But, for the purpose of systemd-creds, the encryption key is bound to PCR 7 by default. You may want to bind it to PCRs 7 and 14, using --tpm2-pcrs=7+14 when running systemd-creds setup.

By binding your key to TPM PCRs 7 and 14, you are trusting the kernel binaries provided by your OS distro. An attacker could install the same OS on another volume and potentially unseal your systemd-creds TPM-stored key. This is why the master credential key for systemd-creds is split across the TPM and disk by default.

For a list of well-known PCR values in Linux, see the shim bootloader README.tpm file and the docs for systemd-cryptenroll Also check out Lennart Poettering's blog post authenticated boot and disk encryption for his take on the brittleness of PCRs and why you should choose carefully.

If you really want to lock things down, you could create your own code signing CA and add it to the certificate database in the Linux shim bootloader, and then sign your own kernel images. But that's a lot of work.

Other important caveats

Because our systemd credential is encrypted, it's okay to store it in a world-readable systemd unit file. For production, you can store encrypted credentials in the credentials directory, outside of unit files and with tighter permissions.
When the service is running, the credentials live on disk so they can be read by the service. The systemd service should use mount namespacing so that the credentials directory is only visible to that service. Mount namespacing is the same mechanism containers use to keep their filesystems isolated from the host environment. Use PrivateMounts=true to enable mount namespacing for a service.

Sources:

Carl Tashian (Website, LinkedIn) is an engineer, writer, exec coach, and startup all-rounder. He's currently an Offroad Engineer at Smallstep. He co-founded and built the engineering team at Trove, and he wrote the code that opens your Zipcar. He lives in San Francisco with his wife Siobhan and he loves to play the modular synthesizer 🎛️🎚️

The magic of systemd-creds

Background

Prerequisites

Diving In

1. Create an Arch Linux VM.

2. Connect, update the system, and restart

3. Configure `systemd-creds`

3. Install dependencies.

4. Install the `step` and `step-ca` binaries.

5. Initialize your CA

6. Add a systemd service for your CA.

7. Encrypt your CA credentials.

8. Start `step-ca.service`

9. Finishing up

A side note on PCRs

Other important caveats

Managed workloads and devices

Further Reading

Unlocking Zero Trust Security: Apple MDA for MDM Device Enrollment and Web Application Security

Certificate Based Wifi Authentication With RADIUS and EAP-TLS

You shouldn’t run NSA-grade Wi-Fi at home. Here’s how to do it

The magic of systemd-creds

Background

Prerequisites

Diving In

1. Create an Arch Linux VM.

2. Connect, update the system, and restart

3. Configure systemd-creds

3. Install dependencies.

4. Install the step and step-ca binaries.

5. Initialize your CA

6. Add a systemd service for your CA.

7. Encrypt your CA credentials.

8. Start step-ca.service

9. Finishing up

A side note on PCRs

Other important caveats

Managed workloads and devices

Further Reading

Unlocking Zero Trust Security: Apple MDA for MDM Device Enrollment and Web Application Security

Certificate Based Wifi Authentication With RADIUS and EAP-TLS

You shouldn’t run NSA-grade Wi-Fi at home. Here’s how to do it

3. Configure `systemd-creds`

4. Install the `step` and `step-ca` binaries.

8. Start `step-ca.service`