Securing SSH with Vault OTP

By Riley Lloyd

System security is an important part of the increasingly prevalent issue of online security threats seen today. If an attacker gains access to your system, they can get control of most running applications, local data and connected machines. I’m sure most of us use secure shell (SSH) to connect to remote machines as it is quick and easy to use. This is all the more reason for an attacker to attempt an attack on SSH opposed to using very basic and aggravating shell connections used in other attacks.

By default, SSH servers use password authentication with optional public key authentication. This allows users that setup public key authentication if they’d like, but it’s not required to log in. If any user on the system has a fairly weak password, this allows an attacker to brute force or dictionary attack the SSH connection for the user until they gain access. However, if all user’s on the machine are required to use public key authentication, the keys (that are hopefully 4096 bits) are way too long to be brute forced or guessed.

“Great! Public key authentication sounds like the way to go right?” Well, in theory yes. In practice it may not be as great as it sounds. For instance, what happens if an attacker gets a hold of your private key? They’ll have the ability to login to any system that your public key in on with no effort. Now to mitigate this you can password protect your private key, that way the attacker needs to enter the password to use it! “But wait… didn’t we do all this just to prevent the attacker from brute forcing my password in the first place?” Yep, pretty much all we did is just add another step to the process (stealing your private key) which hopefully shouldn’t be easy for the attacker. If your private key is stolen, then you have to deal with revocation certificates, deleting the keys off the system, and other painful procedures.

Luckily, there is a better answer. Using HashiCorp’s (the makers of vagrant) Vault, we can create a one-time password (OTP) server for SSH authentication on a network. The way this works is an authenticated client will create a request to the Vault server for an OTP. The OTP is sent to the client and is used to authenticate to the remote server. The remote server will then contact the Vault server so verify that the OTP is acceptable, and allow the SSH connection (or deny it if the OTP is wrong).

Originally, I was thought it would be better to have the Vault server generate a short lived public and private key pair, which would be set to expire in a matter of minutes after authentication. That way, even if the private key is stolen, there is only a minimal amount of time that it is valid. However, there is a considerable number of drawbacks when using this method. Since the Vault server is only contacted when a user requests a key, and not when it is used, accountability becomes incredibly hard. If two users request keys for the same machine within seconds of each other, there is no way to tell what user logged in how many times with what key. Additionally, generating many dynamic keys consumes entropy. If enough requests are made, and there isn’t enough entropy to generate the keys, the request will be stalled until enough entropy is generated causing delays. Lastly, due to the interactions between the SSH server, SSH client, and Vault server, verifying host keys would not be possible.

That’s not to say that OPT doesn’t have its own problems. But there are certainly a lot less of them. Mainly, the only concern with OTP is that is the connection between the SSH server and the Vault server. If the attacker was able to man in the middle (MITM) the traffic, they could send the SSH server a successful response even if the login attempt was not valid. In order to protect against this, it is important to use transport layer security (TLS). So without further adieu, let’s build a Vault server.

To start off visit this link to get vault installed on your machines. Once that is done, we can move on to setting up SSH OTP.

First, start up a vault server with the command “vault server -dev” (you may need to use the command “export VAULT_ADDR=’’” to go any further.


Now that you have a vault server running, lets enable the SSH secrets management with the command “vault secrets enable ssh”


Now that the secrets engine is running, it is time to create a role. Do so by typing out the command below:

            vault write ssh/roles/otp_key_role \
            key_type=otp \
            default_user=username \


In this particular case, I used a test account named “username” and I set the cidr_list to “” to include all hosts. In a production environment you will want to use a much more restricted list.

Now that we have the vault server setup, we need to configure the sshd configuration on our remote host. To do this, we need to install the Vault SSH helper script on the machine which can be found here. Once the helper is installed, we need to configure PAM in /etc/pam.d/sshd to reflect the following:

#@include common-auth
auth requisite quiet expose_authtok log=/tmp/vaultssh.log /usr/local/bin/vault-ssh-helper -config=/etc/vault-ssh-helper.d/config.hcl
auth optional not_set_pass use_first_pass nodelay

And the “/etc/ssh/sshd_config” file needs to be edited to reflect the previous changes and disable password authentication:

ChallengeResponseAuthentication yes
UsePAM yes
PasswordAuthentication no

Now that the Vault server is ready and the SSH server is setup to use Vault OTP, lets create a credential with the command “vault write ssh/creds/opt_key_role ip=x.x.x.x” that we can use to login! In this case, the output from the command labeled “key” is the OTP.


Just copy and paste that key into the password prompt and BOOM, you’re in! To make this even easier for your users, you can create a bash file named “vault-ssh” that executes the following command (you’d need to pass the username and IP address to the bash script as well as install sshpass):

vault ssh -role otp_key_role -mode otp -strict-host-key-checking=no username@x.x.x.x

That way all users will be able to easily use the service without needing to know all the behind the scenes work. If it’s not easy, people won’t spend the extra time to do it and security goes out the window. Now just sit back and relax as your network’s SSH communication is the most secure it’s ever been!


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s