Embarrassingly easy private certificate management for VMs on AWS, GCP, and Azure

Getting a certificate from step-ca using IID-based authentication

The basic architecture of step-ca's IID-based authentication is pretty simple:

To get a certificate from step-ca using an IID we need to:

• Generate a key pair and a certificate signing request (CSR) with the workload's name
• Obtain an IID from the metadata API to authenticate to step-ca
• Submit the CSR & IID to step-ca via HTTPS POST to obtain a certificate
• Store the certificate and private key somewhere our workload can find it

While this is all standards-based and simple in theory, in practice there's a lot of arcane implementation detail to get right. Luckily, the step command line utility works seamlessly with step-ca to do all of this for us.

To demonstrate, assume we have step-ca running on AWS with hostname ca.local (see getting started).

To enable IID-based authentication we'll simply configure step-ca, adding an AWS-type provisioner.

Find your AWS account ID to restrict access to our VMs:

On the host running step-ca add an AWS provisioner to your configuration by running:

    $ step ca provisioner add "AWS IID Provisioner" \
          --type AWS \
          --aws-account 123456789042

(Re)start step-ca to pick up this new configuration:

    $ step-ca $(step path)/config/ca.json

With the step-ca server configured and running, let's use step ca bootstrap to configure a new VM instance to trust and connect to it:

    $ step ca bootstrap \
           --ca-url https://ca.local \
           --fingerprint f501ed49263c1369bd490a85660ddd4388d4175e0337100a11d4e82eae496499
    The root certificate has been saved in ~/.step/certs/root_ca.crt.
    Your configuration has been saved in ~/.step/config/defaults.json.

The --fingerprint is for the root certificate used by step-ca. Find it by running step certificate fingerprint $(step path)/certs/root_ca.crt on your CA. After bootstrapping we're ready to get a certificate. If we pass our AWS IID provisioner name to step ca certificate (via --provisioner), step will automatically use IID-based authentication to get a certificate:

    $ step ca certificate foo.local foo.crt foo.key \
           --provisioner "AWS IID Provisioner"
    ✔ Key ID: AWS IID Provisioner (AWS)
    ✔ CA: https://ca.local
    ✔ Certificate: foo.crt
    ✔ Private Key: foo.key

The first positional argument to step ca certificate specifies our workload's name -- the certificate subject -- in this case, foo.local. The next two positional arguments specify which files to write the certificate and private key, respectively.

We can use step certificate inspect to check our work:

    $ step certificate inspect --short foo.crt
    X.509v3 TLS Certificate (ECDSA P-256) [Serial: 4555...1939]
      Subject:     foo.local
                   ip-172-31-65-180.us-east-1.compute.internal
                   172.31.65.180
      Issuer:      My Intermediate CA
      Provisioner: AWS IID Provisioner [ID: 8074...3263]
      Valid from:  2019-07-08T21:39:40Z
              to:  2019-07-09T21:39:40Z

Congratulations, that's it. Tell your workload to use foo.crt and foo.key, configure clients to trust certificates signed by $(step path)/certs/root_ca.crt, and you're good to go. You now have a strong standards-based mechanism to authenticate workloads and encrypt traffic using TLS.

Example configurations for GCP and Azure are available in the step certificates docs. Instead of account IDs, the GCP IID implementation restricts access by project and/or service account. Azure restricts access by tenant. The step CLI abstracts away the remaining differences.

Depending on your situation and religion, you might toss these commands into a startup script, bake them into an AMI, or use configuration management for the last mile of automation. Speaking of religion: if you're using kubernetes then IID-based authentication isn't right for you. Use autocert or our cert-manager integration instead.

A few more features

Subject Names

In the example above we got a certificate identifying our workload using the logical service name foo.local. Standard HTTPS implementations require this name to appear in the authority portion of a URL. That is, this certificate will only work if we connect to the service using a URL like https://foo.local/some/path.

The name foo.local doesn't appear anywhere in the IID. There's no check to ensure that this particular VM should be allowed to obtain a certificate with this particular name. In fact, by default, step-ca will allow your VMs to obtain a certificate binding any name. This is a pretty broad privilege -- a VM that's supposed to run bar.local could maliciously impersonate foo.local -- but this isn't a problem if you trust all of the processes and users that have access to your VMs.

If you'd rather, you can add "disableCustomSANs": true to your provisioner configuration to lock down step-ca so that it will only issue certificates binding a cloud-specific instance identifier (e.g., i-05e360db6e50b5eda), internal hostname (e.g., ip-172-31-65-180.us-east-1.compute.internal), and internal IP address (e.g., 172.31.65.180):

    ubuntu@ip-172-31-65-180:~$ NAME=$(curl -s http://169.254.169.254/1.0/meta-data/instance-id)
    ubuntu@ip-172-31-65-180:~$ step ca certificate $NAME vm.crt vm.key \
      --provisioner "AWS IID Provisioner"
    ✔ Key ID: AWS IID Provisioner (AWS)
    ✔ CA: https://ca.local
    ✔ Certificate: vm.crt
    ✔ Private Key: vm.key

Here, we've passed the instance ID (i.e., i-05e360db6e50b5eda) to step ca certificate as the certificate's subject (its common name), which we obtain via introspection from the metadata API. With custom subject alternative names (SANs) disabled, step-ca automatically extracts the instance's internal hostname and IP from the IID and includes them as SANs in the certificate:

    ubuntu@ip-172-31-65-180:~$ step certificate inspect --short vm.crt
    X.509v3 TLS Certificate (ECDSA P-256) [Serial: 9391...0237]
      Subject:     i-05e360db6e50b5eda
                   ip-172-31-65-180.us-east-1.compute.internal
                   172.31.65.180
      Issuer:      My Intermediate CA
      Provisioner: AWS IID Provisioner [ID: 8074...3263]
      Valid from:  2019-07-08T19:00:48Z
              to:  2019-07-09T19:00:48Z

With custom SANs disabled for an IID provisioner this is the only sort of certificate step-ca will sign. That is, step-ca will only sign certificates binding your VM's instance identifier, host name, and IP address. This is much more restrictive, and will require that you connect to workloads using URLs like https://ip-172-31-65-180.us-east-1.compute.internal or https://172.31.65.180.

If you're interested in more sophisticated enrollment and access control please open an issue or let us know so we can understand your requirements.

Multiple Certificates

If you try to get two certificates on the same VM you'll find that the CA returns Unauthorized:

    $ step ca certificate foo.local foo.crt foo.key
    ✔ Key ID: AWS IID Provisioner (AWS)
    ✔ CA: https://ca.local
    ✔ Certificate: foo.crt
    ✔ Private Key: foo.key
    
    $ step ca certificate bar.local bar.crt bar.key
    ✔ Key ID: AWS IID Provisioner (AWS)
    ✔ CA: https://ca.local
    Unauthorized

This is another security feature. Since any user or process on a VM can obtain an IID, they can also obtain a certificate. To prevent malicious certificate requests the CA will only issue one certificate per VM by default. If you obtain a genuine certificate early in your VM's life cycle (e.g., in a startup script) any subsequent requests will fail. So this feature mitigates some threats.

Unfortunately, this also makes it impossible to use IIDs to issue different certificates to two or more workloads running on the same VM. To turn this feature off, set "disableTrustOnFirstUse": true in your provisioner configuration and restart step-ca.

Renewal & Revocation

You can renew and revoke a certificate that was issued using IID-based authentication the same way you would any other certificate issued by step-ca, with step ca renew and step ca revoke:

    $ step ca renew --force foo.crt foo.key
    Your certificate has been saved in foo.crt.
    
    $ step ca revoke --cert foo.crt --key foo.key
    Certificate with Serial Number 4555...1939 has been revoked.
    
    $ step ca renew --force foo.crt foo.key
    error renewing certificate: Unauthorized

This handy incantation of step ca renew can address a couple obvious operational considerations, too:

    $ step ca renew --daemon --exec "kill -HUP $SOME_PID" foo.crt foo.key
    INFO: 2019/07/18 14:22:18 first renewal in 15h50m43s

Here we've told step to stay running, renewing your certificate periodically, and notifying $SOME_PID after each renewal by sending it a SIGHUP. If the CA goes down for a period of time and a renewal fails, step ca renew will simply try again.

For more thoughts and best practices around renewal & revocation see our blog post on passive revocation.

Summary

So there you have it. IID-based authentication makes it easier than ever to get certificates to cloud VMs. All you need to do is spin up an instance running 'step-ca' and add a few shell commands to your CD pipeline. The hard stuff is handled for you by your cloud provider, step, and step-ca. You can get up and running in an afternoon.