Surviving a 20000+ node botnet Attack

My web server has been under attack since early this month.  This is a dedicated server that I have leased for years.  It only hosts a couple of sites for me, my family and a few select friends.  Nothing of any real importance or sensitivity exists on it.  Why this insignificant little server attracted the attention of someone who has access a 20,000+ node, worldwide bonnet is beyond me.

It started when I noticed that sites weren’t loading.  I shelled into the box and found the load hovering around 30+.  ps and top showed that apache was the culprit.  I combed through some logs and found that my wife’s site, messymissy.net, was being hammered.  Hundreds of POST requests per second to her index page.  I tcpdumped some of it and found that it was garbage or encrypted payloads destined for gryphn.com.  She has owned gryphn.com for almost 10 years and has it parked on top of messymissy.net.

We unparked the domain and removed the DNS zone file and apache started working again.

A couple of hours later we noticed that nothing on our server was resolving.  I shelled back in and found that DNS was now being hammered with queries to cached zone files for gryphn.com (which didn’t exist).  This log excerpt represents a tenth of a second worth of traffic.

Feb  3 20:35:55 host named[3235]: client 103.8.44.8#23376: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 196.43.54.190#13041: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 193.2.1.102#39491: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 195.186.4.108#59071: query (cache) ‘grYPhN.cOM/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 209.156.227.34#44924: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 89.95.242.180#56873: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 213.228.58.145#5210: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 192.221.159.76#44010: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 74.125.189.16#47278: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 194.90.2.4#63342: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 202.216.229.12#25343: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 74.208.3.18#34990: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 74.208.3.17#48741: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 61.153.81.123#30836: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 68.105.29.237#30849: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 192.221.134.4#28981: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 83.206.226.34#10582: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 110.164.252.215#39831: query (cache) ‘gryphn.com/NS/IN’ denied
Feb  3 20:35:55 host named[3235]: client 196.43.54.190#17049: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 208.69.32.21#36506: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 173.194.96.19#58355: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 209.156.227.34#38061: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 80.10.201.97#21826: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 164.124.101.49#16876: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 74.125.16.215#54383: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 209.18.35.114#2426: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 195.186.4.108#29276: query (cache) ‘grYPhN.cOM/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 74.125.178.16#54930: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 193.2.1.102#6891: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 195.186.1.173#39050: query (cache) ‘GryPhn.coM/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 80.10.201.33#27523: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 192.221.151.75#65393: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 195.20.253.11#53176: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 210.94.72.122#58224: query (cache) ‘gryphn.com/A/IN’ denied

 I logged into my DNS provider and enabled the use of their DNS servers.  We awaiting propagation of the new authoritative name servers and load returned to normal.

Immediately following that we started receiving distributed brute force login attacks to multiple email accounts (that don’t exist) associated with multiple domains that we host.  I configured my firewall scripts to monitor for this sort of thing and block them.  As the firewall block list grew, the amount of invalid login attempt notifications shrunk.  Eventually a large part of the botnet was being blocked by my firewall.

I guess they still had some nodes that weren’t blocked yet (and some fight left in them), because the most recent activity involves distributed brute force login attempts against WordPress sites.  I added a mod_security signature to catch it and modified my firewall scripts to block IPs that trigger the rule too many times.

It’s a really fun cat and mouse game of changing attack methods on a massive scale (world-wide bonnet of 20,000+ zombies).  I’m working on scripts that will mine my logs for multiple block events and send automated x-ARF notifications to abuse@contacts for the zombies.

I have no idea what it is they are after, but I’m having fun playing.

If you are responsible, use the contact form on my site to send me an idea of what it is you want.  I won’t give it you, but the suspense is killing me.  🙂