By Solomon Rubin
Given the high attack rate on SSH servers and the large attack surface that ssh provides, locking down SSH provides a significant challenge. Given the high deployment rate of SSH servers with default configurations the assumption can be made that many servers are left largely insecure. Plenty of tools and configurations exist to help secure these endpoints however they are many and often not entirely effective against multiple attack surfaces on their own thus providing fragmented protection.
SSH has been around for a long time, nearly two decades, and continues to be the most popular remote connection protocols to date. It’s safe to assume that most of the servers have SSH installed on them as many linux distributions, and now windows as well, comes default with SSH. This makes SSH a very obvious entrypoint for adversaries when looking at infiltration points on servers around the world. There many different kinds of attacks that can be taken against SSH servers all of which allow for shell access into the system making additional security is a necessity for any server that runs SSH.
Attacks and Vulnerabilities
There have been dozens of vulnerabilities and attack surfaces discovered within the SSH protocol and within the programs that implement it such as OpenSSH. These issues have ranged from fairly “simple” issues which lead to more significant issues such as buffer overflow vulnerabilities to much more specialized issues such as vulnerabilities in CBC implementations that allow an attacker to recover arbitrary blocks of plaintext from a connection.2 Many of these vulnerabilities are patched quickly after discovery, however there are long standing attacks that are much harder to mitigate with patches.
The most common attack on SSH servers around the world is the brute force password attack. This has been around for nearly 15 years and continues to be the downfall of many servers even today. Brute force attacks are when the attack simply tries random password and username combinations until they are able to get in. Often using simple or known common passwords. In a test from securi.net, within 7 days of putting up a server they detected 15,000 brute force attacks.3,10 They were also able to see the most common usernames, passwords, and what attackers did after they got inside their box.
Man in the middle
Another common attack is a man in the middle attack in which an attacker sits between the client and the server and pretends to be the server breaking the cryptography and snooping on then relaying all information to and from the real server as if they weren’t there. This allows the attacker to pick up information such as session activity, cryptographic keys, passwords, and anything else that is relied through the ssh connection. Once a server is determined as vulnerable to this sort of attack, methods such as Arp spoofing are used to redirect traffic to the MITM host.7
With attacks like brute force and MITM simple patches do not apply because these attack surfaces are based on what can effectively be classified as “user error” and lapses in configuration. It’s worth mentioning that even the most comprehensive of configurations can still fall trap to these attacks which is why additional tools must also be utilized to better protect a system. Additionally, these particular attack methods are pretty universal across many different client entry points whether that be an HTTP server, telnet (but why), sftp, etc. It will become clear that some of the protection methods that are covered can apply to the other services mentioned above. In the case of SSH, the primary method used will be hardening the actual sshd_config in order to stop as many issues as possible at the core. Other tools like Fail2Ban will be used to remove repeat offenders. Tools like this are extremely powerful and can apply to other services as well.
If using SSH on a server this should be the first place that one looks to determine how secure an installation is. Due to the age of ssh, there are many legacy configurations which need to be accounted for and ideally disabled unless specifically in use. Other configurations account for general good security practices and minimize risk of brute force attacks and other bad actors getting into a machine.
Forwarding windows using X11 has always been a pain point because this protocol was not built with security in mind and opens up a window that could potentially allow for malicious commands to be sent to the client. Disabling this closes a huge security hole in the system. Doing this is quite simple.
Rhosts is an old and now uncommon authentication method that was extremely vulnerable to spoofing that SSHd still implements. In short, it was a way to trust other systems by it’s IP address which is extremely easy to spoof. Thus, it should be disabled.
No Empty Passwords
PAM accounts are not required to have passwords depending on their use which is fine and understood. However one misconfigured account could potentially allow for an unauthenticated entry point into a server by mistake. SSHd will allow for no empty passwords to mitigate this issue entirely.
The root user on a server should be considered sacred and should only be accessed in the most secure way as it has all permissions on a box. The root user should only be accessed by first logging in via an unprivileged user first. Directly logging into the root user via a remote session should simply never be done and should be disallowed in the configuration.
No Password Authentication
This particular configuration relates directly to the brute force attack previously looked at. As passwords are extremely weak due to their short length and generally low variance in characters. Additionally, they often follow patterns which make them extremely easy to brute force. To mitigate this issue, disabling passwords all together is ideal. Instead, using Public/Private Key authentication is much more secure and much better practice. This configuration is a combo of password authentication being disabled and enabling public key authentication.
PubkeyAuthentication yes PasswordAuthentication no
Changing ports is another method of securing an ssh server through obscurity. While an nmap scan would easily show a running ssh server on any port, it’s an extra step an attack would need to do in order to access the machine and may deter a simple bot.
This option setups a separate child process without any privileges to handle any incoming traffic that is not authenticated. Only after the user is authenticated do they get any privileges. That way if anything interrupts or corrupts that process there will be no privilege escalation. By default this is enabled and should remain enabled.
Other tools also exist for ssh auditing such as Lynis or ssh-audit.
This configuration will allow for a significantly hardened configuration over the default configuration of sshd. Although this will help significantly, it doesn’t won’t necessarily insure the safety of a machine on its own and it’s worth reviewing the following configuration tools as well.
Netfilter is the framework within the linux that allows for network operations such as packet filtering, NAT, port translation, and other advanced functionality. This functionality can allow for advanced firewall configurations as well as many other configurations. Netfilter can be managed using userspace utilities such as iptables, nftables, or other similar programs.
Netfilter can be used to limit, or outright disallow the use of ssh at the network level.
Allow Some SSH
This configuration will allow certain ip address through to the ssh service. This can be useful if the exact ip addresses for the clients are known in order to restrict who can access the machine. For example, if a jumpbox off network or vpn service is used to access the machine or the machine is only accessed from a single static ip then this method can be useful.
iptables -A INPUT -s <addr>/32 -p tcp -m tcp --dport 22 -j ACCEPT
This can be used as many times as needed until all allowed ips are accounted for. The following configuration is then required to block all other traffic.
Although this may not seem logical if the goal is to have ssh enabled, this actions acts as a base for limiting ssh to certain ips. The following configuration will block ssh at the kernel network level while still leaving the service running. In conjunction with previous configuration this will disallow all ssh traffic. However, the previous configuration acts as a set of exceptions to the disallow rule.
iptables -A INPUT -p tcp -m tcp --dport 22 -j DROP
Netfilter is incredibly powerful and can do a significant amount of work in locking down an ssh system (along with other services). However, this configuration is specific to situations where a specific set of ips are always used. This is not always the case and thus this configuration is not always applicable. Due to how powerful it is, it was worth including.
Fail2Ban is a utility that will detect potential malicious activity from external sources on a server. Using data from logs, Fail2Ban will execute an arbitrary action, generally updating a network filter rule, to remove the threat for a specified amount of time whether that is temporarily or permanently. While Fail2Ban works very well with SSH, it’s able to look through virtually any log for malicious patterns and take action based on this. This is especially effective against brute force attacks.
Fail2Ban offers a lot of different configuration options which allows changes many different parameters such as how easily someone gets banned or how long they get banned for. There are also configurations that will send alert emails after an attacker gets caught in a filter.
The effectiveness of Fail2Ban is highly variable on the configuration supplied to a server. Relaxed configuration will not be as effective but will reduce the amount a legitimate user will be locked out. While a more restrictive may be extremely effective, it’s very possible that legitimate users will be locked out.
Fail2Ban will be especially effective on a more active IP block that have more targeted attacks by bots or other malicious individuals. In tests done on a particularly active block from a VPS service in a time span of a month an IP list of roughly 700 unique IPs were blocked. This number increased by an order of magnitude over a year. On a less active IP block from a larger enterprise datacenter only 16 unique IPs were blocked over a year’s time period. The effectiveness will remain fairly constant, however the need will range based on the inherited ip addresses and the type of traffic the services on the machine attract. A large web store will likely attract more attackers than a machine running nothing but ssh.
The jail is the core component of Fail2Ban that needs to be configured. Because SSH is a core service, the log filters and post-detection actions are pre-build. While these items can be changed and configured as required the default configuration works quite well. The banning/unbanning configuration lies within the Jail component. This portion connects the filter and action as well as defines the ban time, maxretry count, and ‘findtime’ between logins.
The ban time configuration defines how long a user is banned for after they get caught in a filter. It’s not bad practice to turn this number up fairly high to prevent them from logging in for a good amount of time. The configuration below is set to 72 hours.
bantime = 259200
Default Max Retry
The default max retry defines how many times a user can attempt to login before they get banned. This can be overridden by per-service configurations.
maxretry = 5
FInd time defines the search interval for max logins. If maxretry is set to 5 and the findtime is set to one hour, if more than 5 login attempts from one user are find in that time frame, they will be banned.
findtime = 3600
Service Configuration Group
[ssh] enabled = true port = ssh filter = sshd logpath = /var/log/auth.log maxretry = 6
This config group defines the jail for ssh. It’s fairly self explanatory. The last line defines the maxretry for ssh. This defines how many times a user can attempt to login before they get banned and will override the default max retry. Because log paths and ports can vary, this can also be set as necessary.
SSH Host Keys
Host keys are cryptographic keys that are generated by the server and used to verify and decrypt messages sent from the SSH server that is being connected too. When a host key of a server is unknown by a client a warning is shown that looks similar to
The authenticity of host 'test.com (220.127.116.11)' can't be established. ECDSA key fingerprint is SHA256:5E2FFA3B4B76900379FD4676EA32279.
When connecting to a server for this first time, this is entirely normal, after the key is verified it gets stored in ~/.ssh/known_hosts and does not need to be verified again. On the off chance that the machine gets compromised or the host key changes another error will pop up indicating as such.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that the RSA host key has just been changed. The fingerprint for the RSA key sent by the remote host is ab:cd:ef:12:34:cd:11:10:3a:cd:1b:a2:91:cd:e5:1c. Please contact your system administrator. Add correct host key in /home/user/.ssh/known_hosts to get rid of this message. Offending key in /home/user/.ssh/known_hosts:1 RSA host key for app-domain.example.com has changed and you have requested strict checking. Host key verification failed.
This is what the known_hosts mechanism is protecting against.
A host key “changing” can mean a number of things. It can be entirely innocent in that someone actually changed a host key on the server and simply didn’t alert anyone. Whether the server got a refresh or the host keys just need to be cycled out this is not an uncommon occurrence. However, a little communication goes a long way to insure that users aren’t blindsided by this.
The other scenario is a little more scary and gives reason to take a second look at these sorts of errors. Many people blindly accept key fingerprints when connecting to servers, however it’s very good practice to first verify that the key fingerprint matches what the server says it should be it. It’s possible that someone is providing a false a key in order to man in the middle the communications.
The mitigations for this is held equally on the side of the user and the side of the administrator. For the user, it is important to be aware of the dangers of blindly accepting host keys or blindly updating host keys without knowing if they have actually changed or knowing what they have changed too.
For the administrator, it is important to never reuse host keys between machines and to be sure to safeguard these keys at all costs. If a server ever gets compromised, the first thing that should be done is regenerated those keys in case they were compromised. It also doesn’t hurt to periodically (every year) regenerate those keys.
While each protection on it’s own is fairly powerful and can protect against a lot, when put together a server becomes much more resistant to attacks of many different types. SSH configurations alone are powerful and can be fairly secure. Coupled with Fail2Ban, brute force attacks and general weak security is easily mitigated. Added with other good security practices as outlined a robust system is created. As they say, it takes a village, and security is no exception.
- “15.4. Configure the /Etc/Ssh/sshd_config File.” Nested Loops, tldp.org/LDP/solrhe/Securing-Optimizing-Linux-RH-Edition-v1.3/chap15sec122.html.
- “Bug 472068 – CVE-2008-5161 OpenSSH: Plaintext Recovery Attack against CBC Ciphers.” Red Hat Bugzilla Main Page, bugzilla.redhat.com/show_bug.cgi?id=472068.
- Cid, Daniel, et al. “SSH Brute Force – The 10 Year Old Attack That Still Persists.” Sucuri Blog, 28 Sept. 2016, blog.sucuri.net/2013/07/ssh-brute-force-the-10-year-old-attack-that-still-persists.html.
- DigitalOcean. “A Deep Dive into Iptables and Netfilter Architecture | DigitalOcean.” 5 Common Server Setups For Your Web Application | DigitalOcean, DigitalOcean, 18 July 2017, http://www.digitalocean.com/community/tutorials/a-deep-dive-into-iptables-and-netfilter-architecture.
- “Hardening OpenSSH.” Hardening OpenSSH – XCCDF Security Guide, dev.gentoo.org/~swift/docs/security_benchmarks/openssh.html.
- Kelly J. Rose. “Hardening Your SSH Server (opensshd_config).” Kelly J. Rose, 6 Sept. 2013, kjrose.blog/2013/09/06/hardening-your-ssh-server-opensshd_config/.
- “Main Page.” Fail2ban, http://www.fail2ban.org/wiki/index.php/Main_Page.
- “Man-in-the-Middle Attack Using Cain.” Configure SSH Key Based Secure Authentication | SSH.COM, http://www.ssh.com/attack/man-in-the-middle.
- “Permanently Ban Repeat Offenders With fail2ban (UPDATED).” Phil Hagen’s Scratch Pad, 15 June 2016, stuffphilwrites.com/2013/03/permanently-ban-repeat-offenders-fail2ban/.
- R4stl1n. “R4stl1n/SSH-Brute-Forcer.” GitHub, 5 Mar. 2018, github.com/R4stl1n/SSH-Brute-Forcer.
- Thomson, Iain. “Evil OpenSSH Servers Can Steal Your Private Login Keys to Other Systems – Patch Now.” The Register® – Biting the Hand That Feeds IT, The Register, 14 Jan. 2016, www.theregister.co.uk/2016/01/14/openssh_is_wide_open_to_key_theft_thanks_to_roaming_flaw/.
- “The Netfilter.org Project.” Linux 2.4 Packet Filtering HOWTO, http://www.netfilter.org/.
- “Understanding Secure Shell Host Host Keys”, VanDyke, 2004, https://www.vandyke.com/solutions/host_keys/host_keys.pdf.