SSH Agent Explained

Carl Tashian

Follow Smallstep

Introduction

The SSH agent is a central part of OpenSSH. In this post, I'll explain what the agent is, how to use it, and how it works to keep your keys safe. I'll also describe agent forwarding and how it works. I'll help you reduce your risk when using agent forwarding, and I'll share an alternative to agent forwarding that you can use when accessing your internal hosts through bastions.

What is the SSH agent?

ssh-agent is a key manager for SSH. It holds your keys and certificates in memory, unencrypted, and ready for use by ssh. It saves you from typing a passphrase every time you connect to a server. It runs in the background on your system, separately from ssh, and it usually starts up the first time you run ssh after a reboot.

The SSH agent keeps private keys safe because of what it doesn't do:

• It doesn't write any key material to disk.
• It doesn't allow your private keys to be exported.

Private keys stored in the agent can only be used for one purpose: signing a message.

But if the agent can only sign messages, how does SSH encrypt and decrypt traffic?

When first learning about public and private SSH keys, it's natural to assume that SSH uses these key pairs to encrypt and decrypt traffic. That's what I thought. But it's not the case. An SSH key pair is only used for authentication during the initial handshake.

For example, here's how a user's key is verified during the SSH handshake, from the server's perspective:

1. The client presents a public key to the server.
2. The server generates and sends a brief, random message, asking the client to sign it using the private key.
3. The client asks the SSH agent to sign the message and forwards the result back to the server.
4. The server checks the signature using the client's public key.
5. The server now has proof that the client is in possession of their private key.

Later in the handshake process, a set of new, ephemeral and symmetric keys are generated and used to encrypt the SSH session traffic. These keys may not even last the entire session; a "rekey" event happens at regular intervals.

The agent protocol

SSH uses a Unix domain socket to talk to the agent via the SSH agent protocol. Most people use the ssh-agent that comes with OpenSSH, but there's a variety of open-source alternatives.

The agent protocol is so simple that one could write a basic SSH agent in a day or two. It only has a few primary operations:

• Add a regular key pair (public and decrypted private keys)
• Add a constrained key pair (public and decrypted private keys)
• Add a key (regular or constrained) from a smart card (public key only)
• Remove a key
• List keys stored in the agent
• Sign a message with a key stored in the agent
• Lock or unlock the entire agent with a passphrase

🤔 What's a constrained key? It's usually a key that either has a limited lifetime or one that demands explicit user confirmation when it is used. The ssh-add command is your gateway to the SSH agent. It performs all of these operations except for signing. When you run ssh-add without any parameters, it will scan your home directory for some standard keys and add them to your agent. By default, it looks for:

• ~/.ssh/id_rsa
• ~/.ssh/id_ed25519
• ~/.ssh/id_dsa
• ~/.ssh/id_ecdsa

Once you add keys to the keychain, they will be used automatically by ssh.

ssh-agent and the macOS Keychain

The ssh-agent that ships with macOS can store the passphrase for keys in the macOS Keychain, which makes it even easier to re-add keys to the agent after a reboot. Depending on your Keychain settings, you still may need to unlock the keychain after a reboot. To store key passphrases in the Keychain, run ssh-add -K [key filename]. Passphrases are usually stored in the "Local Items" keychain. ssh-agent will use these stored passphrases automatically as needed.

What is agent forwarding?

SSH's agent forwarding feature allows your local SSH agent to reach through an existing SSH connection and transparently authenticate on a more distant server. For example, say you SSH into an EC2 instance, and you want to clone a private GitHub repository from there. Without agent forwarding, you'd have to store a copy of your GitHub private key on the EC2 host. With agent forwarding, the SSH client on EC2 can use the keys on your local computer to authenticate to GitHub.

How agent forwarding works

First, a little background. SSH connections can have multiple channels. Here's a common example: an interactive connection to a bastion host (jump box) runs on one channel. When agent forwarding is enabled for a connection (usually using ssh -A), a second channel is opened up in the background to forward any agent requests back to your local machine.

From ssh's perspective, there is no difference between a remote and a local ssh-agent. SSH always looks at the $SSH_AUTH_SOCK environment variable to find the Unix domain socket for the agent. When you connect to a remote host with agent forwarding enabled, SSHD will create a remote Unix domain socket linked to the agent forwarding channel, and export an $SSH_AUTH_SOCK pointing to it.

Agent forwarding comes with a risk

When you forward ssh-agent's Unix domain socket to a remote host, it creates a security risk: anyone with root access on the remote host can discreetly access your local SSH agent through the socket. They can use your keys to impersonate you on other machines on the network.

Here's an example of how that might look:

How to reduce your risk when agent forwarding

Here are a few ways to make agent forwarding safer:

• Don't turn on ForwardAgent by default.

  Many guides on agent forwarding will suggest turning on ForwardAgent using the following configuration:

  Host example.com
    ForwardAgent yes
  

  We suggest not doing that. Instead, only use agent forwarding in circumstances where you need it. ssh -A turns on agent forwarding for a single session.

• Lock your ssh agent when you use agent forwarding. ssh-add -x locks the agent with a password, and ssh-add -X unlocks it. When you're connected to a remote host with agent forwarding, no one will be able to snake their way into your agent without the password.

• Or use an alternative SSH agent that prompts you when it's being used. Sekey and Secretive are open source projects that use Touch ID on macOS to store keys in the MacBook Pro's security enclave.

• Or don't use agent forwarding at all. If you're trying to access internal hosts through a bastion, ProxyJump is a much safer alternative for this use case. (see below)

Use ProxyJump: a safer alternative

When you want to go through a bastion host (jumpbox), you really don't need agent forwarding. A better approach is to use the ProxyJump directive.

Instead of forwarding the agent through a separate channel, ProxyJump forwards the standard input and output of your local SSH client through the bastion and on to the remote host. Here's how that works:

1. Run ssh -J bastion.example.com cloud.computer.internal to connect to cloud.computer.internal via your bastion.example.com bastion host. cloud.computer.internal is a hostname that can be looked up using DNS lookup on bastion.example.com.
2. Your SSH client uses keys from your agent to connect to bastion.example.com.
3. Once connected, SSHD on the bastion connects to cloud.computer.internal and hands that connection off to your local SSH client.
4. Your local SSH client runs through the handshake again, this time with cloud.computer.internal.

You can think of it as SSHing within an SSH session; except the ssh program never runs on the bastion. Instead, sshd connects to cloud.computer.internal and gives control of that connection (standard in and out) back to your local SSH, which then performs a second handshake.

Setting up ProxyJump

Let's say my bastion host is bastion.example.com. I could set up my ~/.ssh/config file like this:

Host bastion.example.com
	User carl
Host *.computer.internal
	ProxyJump bastion.example.com
	User carl

Then I just run ssh cloud.computer.internal to connect to an internal destination through the bastion—without agent forwarding.

If ProxyJump doesn't work...

Older versions of SSH and SSHD (prior to 7.2, released in 2016) don't support ProxyJump. But you can do an equivalent operation using ProxyCommand and netcat. Here's an example:

ssh -o ProxyCommand="ssh bastion.example.com nc %h %p" cloud.computer.internal

The magic here is that SSH itself is the proxy you're using for SSH. The nc %h %p part simply opens up a raw socket connection to cloud.computer.internal on port 22. The standard I/O of the parent ssh command is piped right into the ProxyCommand so that the parent ssh can authenticate to the internal host through the proxy connection.

Questions? Comments?

We love talking about all things SSH. Hit us up on Twitter if you have questions or feedback!