Video showing our firewall counters increase every second.

What Happened?

Once in a while, we get a large number of totally unwanted hits. I am thinking that the ones we're getting now are not really DDoS, but rather spiders looking for forms to fill out to send us spam. That being said, the result has been many:

503 Server Unavailable

So for us it was very similar to a DDoS. That's what happens whenever a robot sends you multiple HTTP requests per second non-stop all day long. Our servers are only for some 10,000 to 50,000 hits a month at this point, so receiving that many per second is just not sustainable.

Search for the Culprit

So I was getting many hits and had to look for an IP address which would represent the result for the useless traffic. I used netstat to list the established and timed out connections and checked the IP addresses with dig.

dig -x 159.138.0.0

This tells me that the IP comes from a cloud server:

ecs-159-138-0-0.compute.hwclouds-dns.com.

In other words, it's not a person but servers and we do not really need servers to connect to us on and on.

Blocking the IP with the Firewall

To prvent further slowdowns, I added that IP to my firewall:

sudo iptables -I unwanted 1 -i eth0 -s 159.138.0.0/16 -j DROP

My unwanted chain comes from my iplock tool. Only at this time iplock is not capable of locking an IP with a mask so I used a direct command line.

You'll have to change eth0 and that name of the chain for your own firewall.

Note here I use an Insert command (-I) so the new entries appear early in my firewall. This means we can quickly drop unwanted visitors and generally waste less time.

Note:

I use -j DROP and not -j REJECT for a good reason. That way we stop wasting any bandwidth (i.e. we do not send a reply).

In most cases, you would think that the sender will wait for a timeout before trying again. In our case, no such luck. The attack continued on and on seemingly unaltered with several hits per second.

Watching Our Firewall

Here is the command line I use to check the firewall:

watch -n1 "sudo iptables -nvx -L unwanted | grep -v '0        0'"

The watch command executes the specified strings once per second (-n1).

The string says to run iptables and list the entries in the unwanted chain which have had at least 1 hit (i.e. the "0        0" means that we did not yet get a hit after we added that line to our firewall.)

Similarly, I have a netstat command to see established connections:

watch -n1 "netstat -a64n | grep 443.*ESTA | sed -e s/.*:443// -e s/ESTABL.*// | sort -u"

As we can see, I check for lines that sho ESTABLISHED and are connections on port 443 (secure HTTP protocol).

Then I clean up the line in order to be able to sort -u which means IPs will appear only once in the output.

Again I used watch -n1 in this example. If you want to have more time to grab an IP and verify it, you may want to use -n10 and no watch command at all (i.e. only run the netstat command).

Also while searching for the culprit, I did not parse out IPs that were not still connected and used less so I could go up and down easily. What I've noticed is that I was getting many hits from similar IPs (addresses that started with the same two numbers.)

One comment I have to make... this is going to be harder to manage against IPv6 addresses. These are not as easy to read.

Over 100,000 hits in 6 Hours

I went to bed and slept a short time then checked the numbers again.

First of all, the counters are still going up at a similar pace. So the robots have acknowledged nothing.

Second, the number of hits was already over 100,000. This gives us an average for the main two set of IPs of 4.6 hits per second.