SSH Emergency Access

Carl Tashian

Follow Smallstep

In this post we'll design a break glass procedure for reaching SSH hosts in an emergency, using security keys that you can store offline. This is just one approach, but you can adapt it to your circumstances. We will store an offline SSH Certificate Authority on a hardware security key, and have our hosts trust that CA. This will work on pretty much any OpenSSH setup, including our single sign-on SSH.

Why would you want this? Only as an option of last resort. A backdoor into your servers when, for whatever reason, nothing else works.

Why use certificates instead of public/private keys for emergency access?

• Passive revocation. Certificates expire; public keys don't. You can mint an SSH certificate valid for 1 minute, or even 5 seconds. Once it expires, the certificate will become unusable for new connections. This is perfect for occasional emergency access.
• You'll be able to create a certificate for any account on your hosts and send short-lived certificates to colleagues as needed.

You will need

• Hardware security keys that support resident keys. Resident keys are crypto keys that are stored completely on the security key—sometimes protected by an alphanumeric PIN. The public part of a resident key can be exported from the security key when needed, along with a private key handle. The Yubikey 5 series keys support resident keys. Preferably, you'll dedicate these keys to emergency host access only. For this post I'll just use one key, but you should have an extra for backup.
• You'll need a safe place to store these keys.
• OpenSSH 8.2 or higher on your local machine and on the servers you want emergency access to. Ubuntu 20.04 ships with OpenSSH 8.2.
• (optional) The step CLI to inspect certificates — install it with brew install step on MacOS (Linux install instructions are here)

Instructions

1. Create a certificate authority that will reside on the security key

   Insert the key and run:

   $ ssh-keygen -t ecdsa-sk -f sk-user-ca -O resident -C [security key ID]
   

   For the comment (-C), I supplied yubikey-9-512-742@smallstep.com to remind me which security key this CA refers to (on Yubikeys, the serial number is printed on the key itself).

   In addition to adding a key to the Yubikey, this will generate two files locally:

   • sk-user-ca, the key handle which references the private key stored on the security key,
   • sk-user-ca.pub, which will be the public key for your CA.

   But don't worry, there's another really private key stored on the Yubikey, that cannot be extracted.

2. Get your hosts to trust the CA

   As root on your hosts, add the following to your SSHD config (/etc/ssh/sshd_config) if you don't already have it:

   TrustedUserCAKeys /etc/ssh/ca.pub
   

   Then, append the contents of your CA public key (sk-user-ca.pub) to /etc/ssh/ca.pub on the host.

   Restart your SSHD server:

   # systemctl sshd restart
   

   Now we can try accessing the host. But we're going to need a certificate first!

   Make a key pair that will be associated with the certificate:

   $ ssh-keygen -t ecdsa -f emergency
   

   Certificates and key pairs

   It's tempting to think of an SSH certificate as a replacement for a public/private key pair. But a certificate alone is not enough to authenticate a user. Every SSH certificate also has a private key associated with it. That's why we need to generate this emergency key pair before we issue ourselves a certificate. What matters is that a signed certificate is presented to the server, pointing to a key pair for which you have the private key. We don't actually need emergency.pub to run any of these commands, but we get one from ssh-keygen anyway.

   So, public key exchange is still alive and well, even with certificates. Certificates just eliminate the need for the server to store public keys. Now create the certificate itself; it will allow logins within a 10 minute window that started 5 minutes ago (to account for clock skew), for the user ubuntu. You can change these values as needed.

   $ ssh-keygen -s sk-user-ca -I test-key -n ubuntu -V -5m:+5m emergency
   

   You'll be asked to touch your security key to sign the certificate.

   You can add additional usernames separated by commas, eg. -n ubuntu,carl,ec2-user

   You have a certificate! Now it needs the right permission:

   $ chmod 600 emergency-cert.pub
   

   You can inspect the certificate you just made, with:

   $ step ssh inspect emergency-cert.pub
   

   Here's what mine looks like:

   emergency-cert.pub
           Type: ecdsa-sha2-nistp256-cert-v01@openssh.com user certificate
           Public key: ECDSA-CERT SHA256:EJSfzfQv1UK44/LOKhBbuh5oRMqxXGBSr+UAzA7cork
           Signing CA: SK-ECDSA SHA256:kLJ7xfTTPQN0G/IF2cq5TB3EitaV4k3XczcBZcLPQ0E
           Key ID: "test-key"
           Serial: 0
           Valid: from 2020-06-24T16:53:03 to 2020-06-24T17:03:03
           Principals:
                   ubuntu
           Critical Options: (none)
           Extensions:
                   permit-X11-forwarding
                   permit-agent-forwarding
                   permit-port-forwarding
                   permit-pty
                   permit-user-rc
   

   Here, the public key is the emergencey key you created, and the signing CA is sk-user-ca.

   You should be ready to ssh:

   $ ssh -i emergency ubuntu@my-hostname
   ubuntu@my-hostname:~$
   

   That's it! 🥳 You can now create SSH certificates for any user on a host that trusts your emergency Certificate Authority.

3. Finishing up

   You can now delete emergency*. You can keep sk-user-ca* around, but you don't need to because it also resides on the security key. You may also want to remove the original PEM public key from your hosts (eg. in ~/.ssh/authorized_keys for the ubuntu user), if you have been using that for emergency access.

Emergency access runbook

The next time you need emergency access, here's how to set yourself up.

Insert the security key and run:

$ ssh-add -K

This will bring the CA's public key and key handle into the SSH agent.

Now export the public key to make a certificate:

$ ssh-add -L | tail -1 > sk-user-ca.pub

Create a certificate for any user that will last an hour—or less, if you want:

$ ssh-keygen -t ecdsa -f emergency
$ ssh-keygen -Us sk-user-ca.pub -I test-key -n [username] -V -5m:+60m emergency
$ chmod 600 emergency-cert.pub

Now, you're ready to SSH:

$ ssh -i emergency username@host

If your existing .ssh/config file causes any trouble when connecting, you can run ssh with -F none to bypass it.

If you need to send a certificate to a colleague, Magic Wormhole is a simple and secure option. The only files they'll need are emergency and emergency-cert.pub.

What I love about this approach is that it's hardware backed. You can put the security keys in a safe, and they aren't going anywhere until you need them.