The problem: SSH can be brute-forced
I usually leave an SSH server on a dedicated port on every server I administer and, as you may recall, I even linked two well-written guides to properly configure and harden SSH services.
Now, Internet is a notoriously bad place: scanners and exploiters have always been there, but brute-forcers are on the rise and SSH is one of the services that is heavily targeted by them. Let’s gather some data:
quebec:/var/log # ag "Invalid user" auth.log.2 auth.log.3 | wc -l
4560
quebec:/var/log # head -n 1 auth.log.3 | cut -d " " -f 1-3
May 8 07:39:01
quebec:/var/log # tail -n 1 auth.log.2 | cut -d " " -f 1-3
May 21 07:39:01
So, even if my SSH is allowing only PubkeyAuthentication, in the timespan of two weeks there has been 4560 brute-force attemps (~325 attempts per day).
This is annoying and potentially insecure, depending on your configuration. What can we do?
The solution: using Fail2ban
I have recently read some posts about this problem, and luckily (for us) there are multiple solutions to this problem: the most popular one is Fail2ban. To cut a long story short, the idea behind Fail2ban is to monitor log files of the monitored services and keep track of which IP addresses are trying to use a brute-force attack to use the service. If the same IP address causes a number of bad events in the specified time frame, Fail2ban bans that IP (using netfilter/iptables) for a configure d time amount.
So I just need to install Fail2ban and I am ready to go:
# zypper in fail2ban
NO. NO. NO. You will not be protected against brute-force attacks if you just install it without configuring it.
You MUST configure it!
Let’s take a step back. The core of Fail2ban is the configuration:
# "bantime" is the number of seconds that a host is banned.
bantime = 600
# A host is banned if it has generated "maxretry" during the last "findtime"
# seconds.
findtime = 600
maxretry = 5
These are the default values. And brute-forcers know them, so they can time accordingly their attempts not to break these limits (or to begin again their attempts after the bantime). Again, let’s gather some data. I noticed a frequent brute-forcer IP and follow his data:
quebec:/var/log # ag "Invalid user" auth.log.2 auth.log.3 | ag <IP> | cut -d " " -f 1-3
auth.log.2:8106:May 21 03:34:22
auth.log.2:8112:May 21 03:43:41
auth.log.2:8116:May 21 03:53:00
auth.log.2:8120:May 21 04:02:18
auth.log.2:8126:May 21 04:11:47
auth.log.2:8132:May 21 04:21:08
Can you believe it? It was just on the edge of the 600 seconds between every attempt!
Key concept: outsmarting brute-forcers
The key concept here is to provision a personalized version of your specific choosing of these values, in order to outsmart the brute-forcers (which is easily done). In order to do so, do not modify Fail2ban’s default config file (/etc/fail2ban/fail2ban.conf) but rather just override the defaults in another config file that you can create with:
quebec:/etc/jail2ban # awk '{ printf "# "; print; }' /etc/fail2ban/jail.conf | sudo tee /etc/fail2ban/jail.local
Your customizations have to be defined in /etc/fail2ban/jail.local, so your selected bantime, findtime and maxretry must go there.
Further customization
Some useful findings:
- you can selectively whitelist a group of IPs, hosts, subnets, etc. in order to not being banned when accessing services from whitelisted IPs (e.g.
ignoreip = <VPN subnet>/8 <another VPN subnet>/16) - you can receive an email everytime someone is banned with their whois (with good relief from the system administrator)
- if you are using WordPress, you can extend Fail2ban to monitor failed WordPress login attempts with a WordPress plugin (did I mention that Fail2ban not only monitors sshd? It also monitors nginx, apache, vsftpd and other services)
- you can have Fail2ban automatically sends the banned IPs to a shared blacklist (I have never used this)
Disadvantages
Everything seems perfect now, but what are the disadvantages of it? Given that Fail2ban reasons in terms of banning single IPs, it cannot protect you against distributed brute-force attacks. In this case Fail2ban is pretty useless and other solutions should be implemented that depends from case to case.
