On brute force stupidity

Everyone who’s ever managed any internet-facing server is aware of the ridiculous amount of brute force SSH login attempts by all kinds of botnets. Some folks decide to move their SSHD to a non-standard port, some rely on complicated shenanigans like port knocking, and some use tools like fail2ban. I’m unfortunate enough to manage a little over a dozen servers, so I decided to have some fun with fail2ban.

[sshd]
enabled = true
banaction = %(banaction_allports)s
 
[recidive]
enabled = true

My configuration is pretty straightforward. You fuck up, you get banned. You fuck up repeatedly, you get banned for a longer time. Nothing special there. Given that I’m running a similar config on many boxes, I decided to compile some data relating to the origins of login attempts. This data was collected over a period of ~2 months on ~12 servers.

Here’s a quick plot of the number of times a certain IP address was banned. Only the top 100 abusers are included, because the chart has a very long tail indeed. I removed the IP addresses from the X-axis because there’s no way to include them without turning into a black blob.

It should be immediately obvious that a relatively small number of IP addresses is responsible for a metric fuckton of unwelcome activity. Remember that this represents the number of times an IP was banned. Left unchecked, the number of attempts increases by orders of magnitude.

The top offender (and the only one whose full IP address I’ll publish) is 116.31.116.38. It’s part of a Chinese subnet. It managed to get banned a staggering 4466 number of times. More than the next 5 abusers combined.

As the following chart illustrates, a whopping 76% of these IP addresses belong to Chinese subnets.

I daresay the internet would be a slightly better place if those 100 machines were permanently disconnected. It’s likely they’re just unsuspecting folks with compromised machines. But I for one am permanently firewalling all of them on any box I have access to.

SSH Gateway Shenanigans

I love OpenSSH. Part of its awesomeness is its ability to function as a gateway. I’m going to describe how I (ab)use SSH to connect to my virtual machines. Now, on a basic level, this is pretty easy to do, you can simply port forward different ports to different virtual machines. However, I don’t want to mess about with non-standard ports. SSH runs on port 22, and anyone who says otherwise is wrong. Or you could give each of your virtual machines a seperate IP address, but then, we’re running out of IPv4 addresses and many ISPs stubbornly refuse to use IPv6. Quite the pickle!

ProxyCommand to the rescue!

ProxyCommand in ~/.ssh/config pretty much does what it says on the tin: it proxies … commands!

Host fancyvm
        User foo
        HostName fancyvm
        ProxyCommand ssh foo@physical.box nc %h %p -w 3600 2> /dev/null 

This allows you to connect to fancyvm by first connecting to physical.box. This works like a charm, but it has a couple of very important drawbacks:

  1. If you’re using passwords, you have to enter them twice
  2. If you’re using password protected key files without an agent, you have to enter that one twice as well
  3. If you want to change passwords, you have to do it twice
  4. It requires configuration on each client you connect from

Almighty command

Another option is the “command=” option in ~/.ssh/authorized_keys on the physical box:

command="bash -c 'ssh foo@fancyvm ${SSH_ORIGINAL_COMMAND:-}'" ssh-rsa [your public key goes here]

Prefixing your key with command=”foo” will ensure that “foo” is executed whenever you connect using that key. In this case, it will automagically connect you to fancyvm when you log in to physical.box using your SSH key. This has a small amount of setup overhead on the server side but it’s generally the way I do things. The only real drawback here is that’s impossible to change your public key, which isn’t too bad, as long as you keep it secure.

The Actual Shenanigans

The command option is wonderful, but some users can’t or won’t use SSH key authentication. That’s a bit trickier, and here’s the solution I’ve come up with — but if you have a better one, please do share!

We need three things:

  1. A nasty ForceCommand script on the physical box
  2. A user on the physical box (with a passwordless ssh key pair)
  3. A user on the VM, with the above user’s public key in ~/.ssh/authorized_keys

This will grant us the magic ability to log in to the VM by logging in to the physical box. We only have to log in once (because the second part of the login is done automagically by means of the key files). A bit of trickery will also allow us to change the gateway password, which was impossible with any of our previous approaches.

Let’s start with a change in the sshd_config file:

Match User foo
        ForceCommand /usr/local/bin/vmlogin foo fancyvm "$SSH_ORIGINAL_COMMAND"

This will force the execution of our magic script whenever the user connects. And don’t worry, things like scp will work just fine.

And then there’s the magic script, /usr/local/bin/vmlogin:

#!/bin/bash

user=$1
host=$2
command="${3:-}"

if [ "$command" = "passwd" ] ; then
        bash -c "passwd"
        exit 0
fi
command="'$command'"
bash -c "ssh -e none $user@$host $command"

Update 2016

The above script no longer works with SFTP on CentOS 7 with Debian guests. Not sure why, and I’m too lazy to find out. So here’s a script that works around the problem.

#!/bin/bash

user=$1
host=$2
command="${3:-}"

if [ "$command" = "passwd" ] ; then
        bash -c "passwd"
        exit 0
fi

# SFTP has been fucking up. This ought to fix it.
if [ "$command" = "/usr/libexec/openssh/sftp-server" ] || [ "$command" = "internal-sftp" ] ; then
        bash -c "ssh -s -e none $user@$host sftp"
        exit 0
fi

command="'$command'"
bash -c "ssh -e none $user@$host $command"

And there you have it, that’s all the magic you really need. Everything works exactly as if you were connecting to just another machine. The only tricky bit is changing the gateway password: you have to explicitly provide the passwd command when connecting, like so:

ssh foo@physical.box passwd