Surviving a 20000+ node botnet Attack

My web server has been under attack since early this month.  This is a dedicated server that I have leased for years.  It only hosts a couple of sites for me, my family and a few select friends.  Nothing of any real importance or sensitivity exists on it.  Why this insignificant little server attracted the attention of someone who has access a 20,000+ node, worldwide bonnet is beyond me.

It started when I noticed that sites weren’t loading.  I shelled into the box and found the load hovering around 30+.  ps and top showed that apache was the culprit.  I combed through some logs and found that my wife’s site, messymissy.net, was being hammered.  Hundreds of POST requests per second to her index page.  I tcpdumped some of it and found that it was garbage or encrypted payloads destined for gryphn.com.  She has owned gryphn.com for almost 10 years and has it parked on top of messymissy.net.

We unparked the domain and removed the DNS zone file and apache started working again.

A couple of hours later we noticed that nothing on our server was resolving.  I shelled back in and found that DNS was now being hammered with queries to cached zone files for gryphn.com (which didn’t exist).  This log excerpt represents a tenth of a second worth of traffic.

Feb  3 20:35:55 host named[3235]: client 103.8.44.8#23376: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 196.43.54.190#13041: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 193.2.1.102#39491: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 195.186.4.108#59071: query (cache) ‘grYPhN.cOM/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 209.156.227.34#44924: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 89.95.242.180#56873: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 213.228.58.145#5210: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 192.221.159.76#44010: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 74.125.189.16#47278: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 194.90.2.4#63342: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 202.216.229.12#25343: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 74.208.3.18#34990: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 74.208.3.17#48741: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 61.153.81.123#30836: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 68.105.29.237#30849: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 192.221.134.4#28981: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 83.206.226.34#10582: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 110.164.252.215#39831: query (cache) ‘gryphn.com/NS/IN’ denied
Feb  3 20:35:55 host named[3235]: client 196.43.54.190#17049: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 208.69.32.21#36506: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 173.194.96.19#58355: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 209.156.227.34#38061: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 80.10.201.97#21826: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 164.124.101.49#16876: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 74.125.16.215#54383: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 209.18.35.114#2426: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 195.186.4.108#29276: query (cache) ‘grYPhN.cOM/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 74.125.178.16#54930: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 193.2.1.102#6891: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 195.186.1.173#39050: query (cache) ‘GryPhn.coM/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 80.10.201.33#27523: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 192.221.151.75#65393: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 195.20.253.11#53176: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 210.94.72.122#58224: query (cache) ‘gryphn.com/A/IN’ denied

 I logged into my DNS provider and enabled the use of their DNS servers.  We awaiting propagation of the new authoritative name servers and load returned to normal.

Immediately following that we started receiving distributed brute force login attacks to multiple email accounts (that don’t exist) associated with multiple domains that we host.  I configured my firewall scripts to monitor for this sort of thing and block them.  As the firewall block list grew, the amount of invalid login attempt notifications shrunk.  Eventually a large part of the botnet was being blocked by my firewall.

I guess they still had some nodes that weren’t blocked yet (and some fight left in them), because the most recent activity involves distributed brute force login attempts against WordPress sites.  I added a mod_security signature to catch it and modified my firewall scripts to block IPs that trigger the rule too many times.

It’s a really fun cat and mouse game of changing attack methods on a massive scale (world-wide bonnet of 20,000+ zombies).  I’m working on scripts that will mine my logs for multiple block events and send automated x-ARF notifications to abuse@contacts for the zombies.

I have no idea what it is they are after, but I’m having fun playing.

If you are responsible, use the contact form on my site to send me an idea of what it is you want.  I won’t give it you, but the suspense is killing me.  🙂

Dark Horse Stout (Batch 7)

Dark Horse 7I have been working on ‘the perfect stout’ for a long time.  In fact one of the first batches I ever brewed was a stout.

A while back I create “The Stout Experiment” in an effort to try multiple additions in a single batch.  Oddly enough the control (without any additions) ended up being the best.

I have since done three other bathes (including this one) that has tweaked the recipe slightly each time, based on my tasting notes.

Appearance: Dark, no light gets through. Tan (khaki) head with tight bubbles. Alcohol clings to the side of the glass with lacing from the head.
Mouthfeel: silky smooth, but not thick or heavy. The rolled oats contributed to this.
Flavor: Sweet Carmel at first gives way to burnt coffee, dark chocolate and a hint of grapefruit from the cascade hops.
Aroma: Carmel and noble hops.

All in all it’s the best stout I have ever had but I can identify at least two or three places that show room for improvement.

Next batch I think I will add more roasted barley to turn up that burnt coffee flavor a little and possibly change the aroma hop.

5stars

Fall Cometh

I’m going to brew a big, warming beer for my next batch.  Something that will toast your innards with alcohol burn and warm you up.

I am thinking either a Scottish wee heavy of Russian imperial stout.

The only problem is that these types of beers require extensive aging.  a RIS would be barely drinkable by Christmas.

I saw on a brewing tv episode that a mead maker used a staggered yeast addition to give his mead a cellared flavor right from the carboy.

Has anyone tried this with beer?  I would rather not experiment on a huge expensive beer like a RIS, but will if I don’t hear from anyone… for science!

At the same time I wanted my next batch to be the one I tried the “Brew in a Bag” method.  Do the full mash in my boil kettle is appealing.  To raise the temperature I just add or adjust flame.  After the mash I just lift the bag out, rinse/sparge and start my boil.  Plus it will be less clean up without having to rinse my mash tun.

The problem with my next batch being a big beer and my first bib batch is the risk of overflowing my kettle.  15-20 lbs of grain, 7 gallons for the boil plus however much I need to figure in for absorption during the mash… than re adding for sparge.  None of my software will do the math for brew in a bag so this is all going to have to be done manually.. yup.. on my fingers.

Superior Australian Lager Yeast

I used “Superior” brand Australian Lager yeast for the first time on a black lager. I was able to find very little information on it online so I pretty much threw caution to the wind and used it.

I pitched at about 70 degrees and stuck in my lager area at about 50 degrees. After a week of not checking on it I took a gravity reading and it was still at its original gravity. No fermentation had happened at all.

I transferred it off its yeast cake into a 5gal carboy and stuck it in my ale closet to warm up so I could pitch another yeast.

After one day at 70 (even after transferring it off its yeast cake) it went crazy! Lava lamp style active fermentation. Apparently this is a lager yeast that has to ferment at ale temps? After about 10 days the gravity was at its expected terminal gravity reading. I transferred it again and stuck it in the lager closet at 50. I will let you guys know how it turns out.

Breaking the Silence

Wow has it been a long time!  I was maintaining radio silence during my security clearance background investigation.  Now that its all over expect me to post more (honest).

For those of you not close to me, I was hired by the DoD (Department of Defense).  What does this mean?  Well you will never again hear the words “today at work…” uttered again.  That does not mean I will lacking topics to post about.

My security research at home is picking up again.  Expect posts on the topics of forensics, anti-forensics, malware and possibly a new pentest tool or two.

It’s nice to be back and for those of you reading this, thank you for sticking it out and visiting again.

tcp/2550 and the Chinese

While investigating an unrelated issue and digging through firewall logs I noticed a decent amount of traffic destined for tcp/2550 on one of my work servers.

The traffic mostly (82 of the 84 events today) originates from sequential IPs out of China.  This immediately raises alarms with me.

Upon further examination I discovered even stranger patterns.

  • destination port tcp/2550
  • source port is tcp/80
  • Over the last 24 hours 82 attempts had been made (and blocked) by Chinese
  • All Chinese IPs target 1 specific host
  • 2 attempts from US data centers to two other IPs
  • Further correlated searches on source IPs returns little else outside of what I normally see on the firewall
  • Digging back 30 days indicates that today was the first time such traffic has hit me

Port 2550 is associated with a protocol called ADS (Automation Device Specification) created by Beckhoff for use in their TwinCAT system.  This information meant absolutely nothing to me.  I have never heard of the protocol, company for product so I started digging.

It’s for embed systems.  Its billed as “PLC and Motion Control on the PC” meaning that it could be used for automating just about anything try this website.

“TwinCAT consists of run-time systems that execute control programs in real-time and the development environments for programming, diagnostics and configuration. Any Windows programs, for instance visualization programs or Office programs, can access TwinCAT data via Microsoft interfaces, or can execute commands”

According to the “Applications and Solutions” section of their website it can be used for Robotic Assembly automation, Building/HVAC Automation, Water Treatment and Management, Semiconductor Manufacturing, Medical engineering, the Energy Industry and so on.  These all seem like pretty tempting targets if I was interested taking over a countries infrastructure.

Odder still… I port scanned the target server and it does not have anything running on that port.  I also have historical port scans going back months (so I can detect when new listeners are launched) and it was never open.

Am I missing any known malware that operates on that port?

I think I’m going to send some of this output to the SANs internet storm center to see if they know anything about it.

Hey Mac Users… The Honeymoon is Over.

I know, its sad.  I too am a die hard mac user.

Today alone I have received 4 copies of an email with the subject line “2 Populaar Myths About Female Orgasms –  How to Become an Irresistible Lover” containing an attachment named “Preview.app Document”.

I haven’t had a chance to analyse the .app yet, but I think its safe to assume that its malware of some sort.

The good news is that OS X is still built well.  If I double click it thinking its a document its going to tell me “Hey stupid!  This is an app that was downloaded from the Internet.  Are you sure you want to run it?”.  Maybe not in those exact words.  At that point if I say – “I thought I was opening an document, but sure, lets run this app-like-document” – then I deserve to be infected.

For all the detail oriented folks here are the headers (bold are items changed to protect my info):

Return-path: <efflrescent@aperfectmix.com>
Envelope-to: MY_ADDRESS
Delivery-date: Fri, 01 May 2009 09:39:27 -0400
Received: from [87.18.181.177] (helo=ksecb.telecomitalia.it)
by myserver.mydomain.com with smtp (MyMail Dameon)
(envelope-from <efflrescent@aperfectmix.com>)
id 1LzsxZ-0000Ib-JG
for MY_ADDRESS; Fri, 01 May 2009 09:39:27 -0400
Message-ID: <49FAF79E.9745295@aperfectmix.com>
Date: Fri, 01 May 2009 13:39:25 -0100
From: Chesner <efflrescent@aperfectmix.com>
MIME-Version: 1.0
To: MY_ADDRESS
Subject: 2 Populaar Myths About Female Orgasms –  How to Become an Irresistible Lover
Content-Type: multipart/mixed;
boundary=”————32D524EA4E2E67F07C94899F”
X-Spam-Status: No, score=3.8
X-Spam-Score: 38
X-Spam-Bar: +++
X-Spam-Flag: NO

The body contains no data.

Mining Ports for Malware

I recently wrote a script that runs croned and port scans all of our servers daily.  It saves the output and diffs it compared to the previous days and emails me as new ports open up.

I think this will be a good way to detect new services and potential malware infection, but what about machines that are already infected?

To fix that I wrote in a function that parses the output for known malware ports.  The only problem is that I cant find a definitive list of known malware ports.  Does anyone know of such a resource?

Loaded C:\WINNT\system32\KERNEL32.dll differs from file image

I have recently been updating my Windows Forensics First Responder script and have noticed a number of servers reporting the following when using Sysinternals/Microsoft’s listdlls.exe.

*** Loaded C:\WINNT\system32\KERNEL32.dll differs from file image:
*** File timestamp:         Wed Apr 18 12:25:36 2007
*** Loaded image timestamp: Wed Apr 18 12:25:37 2007
*** 0x77e40000  0x102000  5.02.3790.4062  C:\WINNT\system32\KERNEL32.dll

Now I can think of lots of malicious reasons why this would be.  In fact I recently wrote on one of these reasons.   But I cant think of any legitimate reasons.

I’m not one to jump to conclusions without having evaluated all possibilities but my research is turning up almost nothing.

Can anyone think of a legitimate reason why windows would load kernel32.dll and then something alter it as its going into memory?

Thanks guys.

Why Won’t Dell Stop Sucking?!

For some reason people keep buying Dells.

I remember a couple of years ago all the small form factor optiplex’s I had suffered from a bad cap on the motherboard.  Eventually all of them just die.

My whole team at work have the same model workstation and the PSU went on each of them, one by one.

I have a service tag – the “serial number” unique to each computer – and type it into their site looking for drivers.  You would think, being that this tag is unique, that they could look up your computer and give you your network card drivers, your video driver etc.  NO!  Instead they give you the choice to download every driver for every chipset that was ever used on that given model.  Why do I have this service tag?!  Why don’t I just type in the model?!  Its the same results!

After all that people still buy these pieces of crap.  They never even question why that is.