Introducing Smallstep ACME RA: Automating internal TLS with ACME + Google CAS

Carl Tashian

Follow Smallstep

We're excited to announce our new HSM-backed cloud ACME server, the Smallstep ACME Registration Authority (RA) for Google CA Services (CAS). This product aims to make your internal PKI easier to use, more secure, and simpler to scale:

• An ACME interface to Google CAS. Our ACME server makes internal automated certificate enrollment and renewal simpler, by bringing the ACME protocol (used by Let's Encrypt; RFC8555) to your internal Google CAS environment. It supports all of the challenge types that Let's Encrypt supports, so it's straightforward to add ACME support to existing services.

  For internal services, ACME is a great way to automate TLS certificate enrollment and renewal, due to its broad support across languages and platforms.

• Leverage Google Security & Scalability. Our ACME server connects to your Google CAS instance, which acts as your CA and issues Google-signed certificates. In this scenario, our ACME server acts narrowly as a registration authority, sitting between Google CAS and your ACME clients.

  Google CAS is a highly-available, scalable private CA that is backed by Hardware Security Modules (HSM). It uses FIPS 140-2 Level 3 validated HSMs.

• Easy automated enrollment and renewal. Automation with ACME clients obviates the need for manual SSL certificate renewal, while preventing outages related to certificate expiry.

If you want internal ACME support, consider running Google CAS with Smallstep's ACME RA. Together you can think of the two as an HSM-backed cloud ACME server.

Our click-to-deploy image is available in the Google cloud marketplace.

We plan to add support for more backends soon: Microsoft Active Directory Certificate Services (AD CS), Hashicorp Vault, and AWS Certificate Manager Private CA (ACM PCA).

How ACME works

ACME is a JSON API that runs mostly over HTTPS. To get a certificate issued by an ACME server, a client must prove that it controls the requested domain name(s). It does this by responding to ACME challenges from the server.

A typical ACME challenge flow looks like this:

1. The ACME client generates a Certificate Signing Request (CSR) and a private key. It contacts the ACME server and requests a certificate for the intended domain name.
2. To verify that the client owns the domain name, the ACME server responds with one or more challenges. The challenges are just random values. There are three challenge types that work at increasing levels of intimacy with the service that will ultimately use the certificate:

   • http-01 — the challenge value is placed at a well-known URL on an HTTP server at a domain named in the certificate request.

   • dns-01 — a DNS TXT record must be created that matches the challenge value, confirming that the entity requesting the certificate has control over DNS for a domain named in the certificate request.

     With http-01 challenge type, you need routability between the ACME server and the enrolling host. With dns-01, all you need is for everyone to share the same DNS servers, and you can use public or private DNS. The downside of dns-01 is that you have to trust DNS—which is notoriously difficult to secure.

   • tls-alpn-01— the challenge value is added to the initial TLS handshake (using the Application-Layer Protocol Negotiation (ALPN) TLS extension) of a server answering at a domain named in the certificate request.

     http-01 uses HTTP and runs on port 80. If you believe in running HTTPS everywhere (and you should), then it's annoying to have to run an HTTP server just to get a certificate. So tls-alpn-01 helps solve for this because it only uses HTTPS. tls-alpn-01 also solves for multi-tenant web servers. You could have wildcard DNS for a bunch of different web server names, and still get certificates for those servers.

Once challenges have been met for each DNS name listed on the certificate, the client can retrieve its signed certificate from the server.

Later, the client returns to the ACME server to renew its certificate using the same approach.

Let's try it out

In these tutorials we'll configure ACME clients for use with Google CAS and the Smallstep ACME server.

Prerequisites

These examples require some setup. You will need:

• A Google Cloud Platform account with a project you can use for this tutorial.
• A Google CAS CA instance, and a Smallstep ACME RA for CAS server that connects to your CAS CA. See our complete setup instructions.
• You may want to create a new GCP project for testing.

ACME + Kubernetes cert-manager

In this example, we'll configure Kubernetes cert-manager to get a certificate from an internal ACME server, using cert-manager's ACME issuer.

Before you begin

This example uses the ACME dns-01 challenge type, so you will need a Google Cloud DNS zone inside your GCP project. You can use a public or private zone. Unlike Let's Encrypt, private ACME works just fine within private DNS zones.

1. Create a cluster

For this tutorial, I created a Google Compute Engine VM running a kind cluster:

$ kind cluster create

I'm using kind for testing, but pretty much any Kubernetes cluster will do as long as the cluster's DNS resolver can resolve names in your Cloud DNS zone.

2. Set up cert-manager to trust your internal CA

Let's install Kubernetes cert-manager and patch it so that it will trust your internal ACME CA. First, install cert-manager:

$ kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v1.0.4/cert-manager.yaml

Next, create a ConfigMap that contains your ACME server's CA certificate. To find your certificate's PEM file, select your CA in the Google Cloud CAS Console, and view your CA certificate under the Actions menu.

Create a file called internal-ca.yaml, replacing the certificate shown here with your own:

apiVersion: v1
data:
  internal-ca.pem: |
    -----BEGIN CERTIFICATE-----
    [REPLACE with your CA certificate]
    -----END CERTIFICATE-----
kind: ConfigMap
  metadata:
  name: ca-pemstore
  namespace: cert-manager
  resourceVersion: "9978"
  selfLink: /api/v1/namespaces/cert-manager/configmaps/ca-pemstore
-

Apply it:

$ kubectl apply -f internal-ca.yaml

To inject this ConfigMap into cert-manager, we need to patch the cert-manager Deployment to add the CA certificate as a container volume mount.

Create a file called cm-ca-patch.yaml:

spec:
  template:
    spec:
      containers:
      - args:
        - --v=2
        - --cluster-resource-namespace=$(POD_NAMESPACE)
        - --leader-election-namespace=kube-system
        name: cert-manager
        volumeMounts:
        - name: ca-pemstore
          mountPath: /etc/ssl/certs/internal-ca.pem
          subPath: internal-ca.pem
          readOnly: false
        resources: {}
      volumes:
        - name: ca-pemstore
          configMap:
            # Provide the name of the ConfigMap containing the files you want
            # to add to the container
            name: ca-pemstore

Apply the patch:

$ kubectl patch deployment cert-manager -n cert-manager --patch "$(cat cm-ca-patch.yaml)"

3. Create a GCP service account and import its credentials

Let's create a Google Cloud Platform service account with the roles/dns.admin role. Replace the PROJECT_ID here with your own:

$ export PROJECT_ID=step-cas-test
$ gcloud iam service-accounts create dns01-solver \
   --project $PROJECT_ID --display-name "dns01-solver"
$ gcloud projects add-iam-policy-binding $PROJECT_ID \
   --member serviceAccount:dns01-solver@$PROJECT_ID.iam.gserviceaccount.com \
   --role roles/dns.admin

Now import the service account's credentials as a Kubernetes secret:

$ gcloud iam service-accounts keys create key.json \
   --iam-account dns01-solver@$PROJECT_ID.iam.gserviceaccount.com
$ kubectl create secret generic clouddns-dns01-solver-svc-acct \
   --from-file=key.json

Finally, let's create an cert-manager Issuer to perform dns-01 ACME challenges. Make a new file called acme-issuer.yaml:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: acme-issuer
spec:
  acme:
    <strong>email: carl@smallstep.com</strong>
    <strong>server: https://ca.smallstep.internal/acme/acme/directory</strong>
    privateKeySecretRef:
      name: acme-issuer-account-key
    solvers:
    - dns01:
      cloudDNS:
      # Your Google Cloud Platform project ID
      <strong>project: step-cas-test</strong>
      # Your Google CloudDNS zone name we will use for DNS01 challenges
      <strong>hostedZoneName: step-cas-internal</strong>
      serviceAccountSecretRef:
        name: clouddns-dns01-solver-svc-acct
        key: key.json

Replace the values for email, server URL, project and hostedZoneName with your own. Your Smallstep ACME endpoint will always take the form of https://[ACME RA hostname]/acme/acme/directory.

Apply it:

$ kubectl apply -f acme-issuer.yaml

You now have an automated ACME certificate manager running inside your Kubernetes cluster.

4. Issue a test certificate

Let's get a test certificate from our ACME CA, using a Certificate object. Create a file called tls-certificate.yaml:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: k8s-internal
  namespace: default
spec:
  secretName: k8s-internal-tls
  issuerRef:
    name: acme-issuer
  dnsNames:
  - k8s.smallstep.internal

Replace the dnsNames with a DNS name that's inside your zone.

Apply it:

$ kubectl apply -f tls-certificate.yaml

You can check the status with kubectl get certificaterequest or kubectl describe certificate:

$ kubectl get certificaterequest
NAME                    READY   AGE
k8s-internal-nzbnm      True    7s
$ kubectl describe certificate k8s-internal
Name:         k8s-internal
Namespace:    default
...
Kind:         Certificate
Metadata:
  Creation Timestamp:  2020-11-03T23:06:46Z
...
Spec:
  Dns Names:
    k8s.smallstep.internal
  Issuer Ref:
    Name:       acme-issuer
  Secret Name:  k8s-internal-tls
Status:
  Conditions:
    Last Transition Time:  2020-11-03T23:11:01Z
    Message:               Certificate is up to date and has not expired
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2020-11-04T23:11:01Z
  Not Before:              2020-11-03T23:11:01Z
  Renewal Time:            2020-11-04T15:11:01Z
  Revision:                1
Events:
  Type    Reason     Age    From          Message
  ----    ------     ----   ----          -------
  Normal  Issuing    10m    cert-manager  Issuing certificate as Secret does not exist
  Normal  Generated  10m    cert-manager  Stored new private key in temporary Secret resource "k8s-internal-g79jq"
  Normal  Requested  10m    cert-manager  Created new CertificateRequest resource "k8s-internal-nzbnm"
  Normal  Issuing    9m33s  cert-manager  The certificate has been successfully issued

As you can see, cert-manager will automatically renew the certificate when approximately 2/3 of its lifetime has elapsed.

That's it! You now have automated, short-lived certificates for your Kubernetes cluster. There are many use cases for X.509 certificates issued through cert-manager.

ACME + Certbot

Certbot is an ACME client that can run anywhere, using a job scheduler or renewal daemon to renew certificate automatically. Let's say we have a database server db.smallstep.internal and we want to get a TLS certificate for it using an ACME http-01 challenge:

sudo REQUESTS_CA_BUNDLE=root_ca.crt certbot certonly  \
 --server https://ca.smallstep.internal/acme/acme/directory \
 --agree-tos --email carl@smallstep.com  \
 -d db.smallstep.internal

The file root_ca.crt should contain your CA certificate. The --server URL will always take the form of https://[ACME RA hostname]/acme/acme/directory.

After enrollment, certbot renew can be scheduled as a job to keep certificates fresh. The caveat here is that Certbot defaults to renewing certificates 30 days before they expire. This is great for the open web, where certificates typically have ≥ 90 day lifetimes, but for our internal infrastructure, we'll want certificates to have much shorter lifetimes.

By default, the Smallstep ACME server will issue certificates valid for 24 hours. Let's aim to renew our db.smallstep.internal certificate about 8 hours before it expires. Add renew_before_expiry = "8 hours" to each of the renewal configuration files in /etc/letsencryp/renew, and set your job scheduler to run certbot renew every 5 or 10 minutes.

Notifying Certificate-Dependent Services

Many services that depend on certificates will only read the certificate files when they start up. So when you renew a certificate, a server process that depends on it may need to be restarted or sent a SIGHUP signal to reload its certificates. To address this, Certbot can be configured to run a hook script before or after renewal to restart servers.

ACME + Certbot + Citrix ADC

You could use Certbot to deploy certificates to remote devices. For example, for an SSL virtual server on a Citrix ADC load balancer, your Certbot hook script should do the following to deploy the certificate upon renewal:

1. Copy the new certificate and key into /nsconfig/ssl/ on the Citrix ADC appliance

2. Run the following:

   add ssl certKey example_key_20201004 -cert certificate.crt -key certificate.key -passcrypt random -expiryMonitor DISABLED
   unbind ssl vserver [virtual server name] -certkeyName example_key_20201004
   bind ssl vserver [virtual server name] -certkeyName example_key_20201005
   

Sources: [1] [2]

In high availability environments, you should be aware that there is a gap in service between the unbind and bind commands.

What else runs ACME?

For enterprise environments, ACME support is already available in a wide variety of contexts, eg:

• In a F5 BIG-IP carrier-grade internal load balancer
• For IoT devices, in ESP32 SoC microcontroller firmware
• In Java applications, using acme4j

Getting started with Google CAS and ACME

Smallstep ACME Registration Authority (RA) brings ACME protocol support to GCP CAS, allowing you to automate certificate enrollment and renewal using ACME-compliant clients like certbot, Terraform, Caddy, and Kubernetes cert-manager. The Smallstep RA does not sign certificates itself. Instead, certificate requests are passed to GCP CAS to sign and catalog delivering a number of benefits including:

• Issued certificates are trusted by anything that trusts your GCP CAS root certificate.
• Issued certificates appear in your GCP CAS console and audit logs.
• Security-sensitive signing keys are managed by GCP CAS and never seen by Smallstep ACME RA.

Smallstep ACME RA is built and supported by Smallstep, the company behind the open-source step-ca certificate management toolchain. It builds on the open-source step-ca project, adding click-to-deploy integration with GCP CAS, updates, and support. Head over to the GCP marketplace and give it a try today.