Surviving a 20000+ node botnet Attack

My web server has been under attack since early this month.  This is a dedicated server that I have leased for years.  It only hosts a couple of sites for me, my family and a few select friends.  Nothing of any real importance or sensitivity exists on it.  Why this insignificant little server attracted the attention of someone who has access a 20,000+ node, worldwide bonnet is beyond me.

It started when I noticed that sites weren’t loading.  I shelled into the box and found the load hovering around 30+.  ps and top showed that apache was the culprit.  I combed through some logs and found that my wife’s site, messymissy.net, was being hammered.  Hundreds of POST requests per second to her index page.  I tcpdumped some of it and found that it was garbage or encrypted payloads destined for gryphn.com.  She has owned gryphn.com for almost 10 years and has it parked on top of messymissy.net.

We unparked the domain and removed the DNS zone file and apache started working again.

A couple of hours later we noticed that nothing on our server was resolving.  I shelled back in and found that DNS was now being hammered with queries to cached zone files for gryphn.com (which didn’t exist).  This log excerpt represents a tenth of a second worth of traffic.

Feb  3 20:35:55 host named[3235]: client 103.8.44.8#23376: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 196.43.54.190#13041: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 193.2.1.102#39491: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 195.186.4.108#59071: query (cache) ‘grYPhN.cOM/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 209.156.227.34#44924: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 89.95.242.180#56873: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 213.228.58.145#5210: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 192.221.159.76#44010: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 74.125.189.16#47278: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 194.90.2.4#63342: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 202.216.229.12#25343: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 74.208.3.18#34990: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 74.208.3.17#48741: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 61.153.81.123#30836: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 68.105.29.237#30849: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 192.221.134.4#28981: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 83.206.226.34#10582: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 110.164.252.215#39831: query (cache) ‘gryphn.com/NS/IN’ denied
Feb  3 20:35:55 host named[3235]: client 196.43.54.190#17049: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 208.69.32.21#36506: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 173.194.96.19#58355: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 209.156.227.34#38061: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 80.10.201.97#21826: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 164.124.101.49#16876: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 74.125.16.215#54383: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 209.18.35.114#2426: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 195.186.4.108#29276: query (cache) ‘grYPhN.cOM/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 74.125.178.16#54930: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 193.2.1.102#6891: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 195.186.1.173#39050: query (cache) ‘GryPhn.coM/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 80.10.201.33#27523: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 192.221.151.75#65393: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 195.20.253.11#53176: query (cache) ‘gryphn.com/A/IN’ denied
Feb  3 20:35:55 host named[3235]: client 210.94.72.122#58224: query (cache) ‘gryphn.com/A/IN’ denied

 I logged into my DNS provider and enabled the use of their DNS servers.  We awaiting propagation of the new authoritative name servers and load returned to normal.

Immediately following that we started receiving distributed brute force login attacks to multiple email accounts (that don’t exist) associated with multiple domains that we host.  I configured my firewall scripts to monitor for this sort of thing and block them.  As the firewall block list grew, the amount of invalid login attempt notifications shrunk.  Eventually a large part of the botnet was being blocked by my firewall.

I guess they still had some nodes that weren’t blocked yet (and some fight left in them), because the most recent activity involves distributed brute force login attempts against WordPress sites.  I added a mod_security signature to catch it and modified my firewall scripts to block IPs that trigger the rule too many times.

It’s a really fun cat and mouse game of changing attack methods on a massive scale (world-wide bonnet of 20,000+ zombies).  I’m working on scripts that will mine my logs for multiple block events and send automated x-ARF notifications to abuse@contacts for the zombies.

I have no idea what it is they are after, but I’m having fun playing.

If you are responsible, use the contact form on my site to send me an idea of what it is you want.  I won’t give it you, but the suspense is killing me.  🙂

tcp/2550 and the Chinese

While investigating an unrelated issue and digging through firewall logs I noticed a decent amount of traffic destined for tcp/2550 on one of my work servers.

The traffic mostly (82 of the 84 events today) originates from sequential IPs out of China.  This immediately raises alarms with me.

Upon further examination I discovered even stranger patterns.

  • destination port tcp/2550
  • source port is tcp/80
  • Over the last 24 hours 82 attempts had been made (and blocked) by Chinese
  • All Chinese IPs target 1 specific host
  • 2 attempts from US data centers to two other IPs
  • Further correlated searches on source IPs returns little else outside of what I normally see on the firewall
  • Digging back 30 days indicates that today was the first time such traffic has hit me

Port 2550 is associated with a protocol called ADS (Automation Device Specification) created by Beckhoff for use in their TwinCAT system.  This information meant absolutely nothing to me.  I have never heard of the protocol, company for product so I started digging.

It’s for embed systems.  Its billed as “PLC and Motion Control on the PC” meaning that it could be used for automating just about anything try this website.

“TwinCAT consists of run-time systems that execute control programs in real-time and the development environments for programming, diagnostics and configuration. Any Windows programs, for instance visualization programs or Office programs, can access TwinCAT data via Microsoft interfaces, or can execute commands”

According to the “Applications and Solutions” section of their website it can be used for Robotic Assembly automation, Building/HVAC Automation, Water Treatment and Management, Semiconductor Manufacturing, Medical engineering, the Energy Industry and so on.  These all seem like pretty tempting targets if I was interested taking over a countries infrastructure.

Odder still… I port scanned the target server and it does not have anything running on that port.  I also have historical port scans going back months (so I can detect when new listeners are launched) and it was never open.

Am I missing any known malware that operates on that port?

I think I’m going to send some of this output to the SANs internet storm center to see if they know anything about it.

Hey Mac Users… The Honeymoon is Over.

I know, its sad.  I too am a die hard mac user.

Today alone I have received 4 copies of an email with the subject line “2 Populaar Myths About Female Orgasms –  How to Become an Irresistible Lover” containing an attachment named “Preview.app Document”.

I haven’t had a chance to analyse the .app yet, but I think its safe to assume that its malware of some sort.

The good news is that OS X is still built well.  If I double click it thinking its a document its going to tell me “Hey stupid!  This is an app that was downloaded from the Internet.  Are you sure you want to run it?”.  Maybe not in those exact words.  At that point if I say – “I thought I was opening an document, but sure, lets run this app-like-document” – then I deserve to be infected.

For all the detail oriented folks here are the headers (bold are items changed to protect my info):

Return-path: <efflrescent@aperfectmix.com>
Envelope-to: MY_ADDRESS
Delivery-date: Fri, 01 May 2009 09:39:27 -0400
Received: from [87.18.181.177] (helo=ksecb.telecomitalia.it)
by myserver.mydomain.com with smtp (MyMail Dameon)
(envelope-from <efflrescent@aperfectmix.com>)
id 1LzsxZ-0000Ib-JG
for MY_ADDRESS; Fri, 01 May 2009 09:39:27 -0400
Message-ID: <49FAF79E.9745295@aperfectmix.com>
Date: Fri, 01 May 2009 13:39:25 -0100
From: Chesner <efflrescent@aperfectmix.com>
MIME-Version: 1.0
To: MY_ADDRESS
Subject: 2 Populaar Myths About Female Orgasms –  How to Become an Irresistible Lover
Content-Type: multipart/mixed;
boundary=”————32D524EA4E2E67F07C94899F”
X-Spam-Status: No, score=3.8
X-Spam-Score: 38
X-Spam-Bar: +++
X-Spam-Flag: NO

The body contains no data.

Mining Ports for Malware

I recently wrote a script that runs croned and port scans all of our servers daily.  It saves the output and diffs it compared to the previous days and emails me as new ports open up.

I think this will be a good way to detect new services and potential malware infection, but what about machines that are already infected?

To fix that I wrote in a function that parses the output for known malware ports.  The only problem is that I cant find a definitive list of known malware ports.  Does anyone know of such a resource?

Loaded C:\WINNT\system32\KERNEL32.dll differs from file image

I have recently been updating my Windows Forensics First Responder script and have noticed a number of servers reporting the following when using Sysinternals/Microsoft’s listdlls.exe.

*** Loaded C:\WINNT\system32\KERNEL32.dll differs from file image:
*** File timestamp:         Wed Apr 18 12:25:36 2007
*** Loaded image timestamp: Wed Apr 18 12:25:37 2007
*** 0x77e40000  0x102000  5.02.3790.4062  C:\WINNT\system32\KERNEL32.dll

Now I can think of lots of malicious reasons why this would be.  In fact I recently wrote on one of these reasons.   But I cant think of any legitimate reasons.

I’m not one to jump to conclusions without having evaluated all possibilities but my research is turning up almost nothing.

Can anyone think of a legitimate reason why windows would load kernel32.dll and then something alter it as its going into memory?

Thanks guys.

A Very Righteous Hack

A roadside traffic sign in Austin, Texas was hacked into so that it displayed a message warning passing motorists of zombies ahead. Police are investigating the incident, and if they are caught, the perpetrators could face misdemeanor road sign tampering charges.  The vandals broke a lock on the sign and then managed to gain access to the computer that controls its readout because it was using a default password.  They also changed the password, so city employees had to wait for the manufacturer to reset the password before the sign could be changed.  A city spokesperson acknowledged that while “the sign’s content was humorous, … the act of changing it wasn’t.”

http://www.dallasnews.com/sharedcontent/dws/news/localnews/transportation/stories/013009dnmetzombies.1595f453.html

I have an issue of 2600 magazine from about 5 years ago that contains that default password.  I had always thought it would be funny if did something like this.  They even changed the default password.  How perfect.

DISCLAIMER: I do no endorse the “hacking” of morons who don’t change default passwords.

Take THAT IE Fan Boy

Bruce Schneier just posted an interesting article on his blog entitled “Interview with an Adware Developer“.

This article reinforces many of the things I have been telling people for a very long time, but for whatever reason never sinks in.

I should probably first speak about how adware works. Most adware targets Internet Explorer (IE) users because obviously they’re the biggest share of the market. In addition, they tend to be the less-savvy chunk of the market. If you’re using IE, then either you don’t care or you don’t know about all the vulnerabilities that IE has.

IE has a mechanism called a Browser Helper Object (BHO) which is basically a gob of executable code that gets informed of web requests as they’re going. It runs in the actual browser process, which means it can do anything the browser can do—which means basically anything.

Aside from reinforcing that Internet Explorer is a poor choice to use for web browsing (unless you enjoy collecting and cleaning malware… you know, for practice), it also outlines an interesting new technique that I recently witnessed as I was cleaning a machine.

If you also have an installer, a little executable, you can make a Registry entry and every time this thing reboots, the installer will check to make sure the BHO is there. If it is, great. If it isn’t, then it will install it. That’s fine until somebody goes and deletes the executable.

The next thing that Direct Revenue did—actually I should say what I did, because I was pretty heavily involved in this—was make a poller which continuously polls about every 10 seconds or so to see if the BHO was there and alive. If it was, great. If it wasn’t, [ the poller would ] install it.

During my live analysis of this machine I used the ms/sysinternals filemon program to watch for a bit and noticed explorer.exe doing something similar to what the author describes.

34139    6:32:11 PM    explorer.exe:2916    OPEN    C:\NTDETECT.COM:QebiesnrMkudrfcoIbamtykdDa.exe:$DATA    NOT FOUND    Options: Open  Access: Read

The article explains how they will create a seemingly random named file (a hash of the mac address) and use that as the installer.  This one appears to be a variant on the technique that takes it a step further and uses hidden data streams (or alternate data streams). These are data streams that I had previously detected and removed.

The article also has an interesting point about evasion.

Next we made a function shuffler, which would go into an executable, take the functions and randomly shuffle them. Once you do that, then of course the signature’s all messed up. [ We also shuffled ] a lot of the pointers within each actual function. It completely changed the shape of the executable.

In the virology and malware world this is known as polymorphism, and is a very effective technique for evading most anti-virus/spyware programs.

Now the truly frightening part mentions using interrupt handlers instead of executables and states that they decided not to do it.  Because the concept is written, someone will run with it.

There was one further step that we were going to take but didn’t end up doing, and that is we were going to get rid of threads entirely, and just use interrupt handlers. It turns out that in Windows, you can get access to the interrupt handler pretty easily. In fact, you can register with the OS a chunk of code to handle a given interrupt. Then all you have to do is arrange for an interrupt to happen, and every time that interrupt happens, you wake up, do your stuff and go away. We never got to actually do that, but it was something we were thinking we’d do.

What this all boils down to is that the malware authors once again have leap frogged the anti-virus industry.  Microsoft also needs to take a more proactive role in securing IE and Windows against these sorts of threats.

The days of recycling the old code as variants is over and its time that we prepare ourselves for a whole new world of malware threats.

More on Heartland

Dark Reading posted an article entitle “Report: Law Enforcement Closing In On Heartland Breach Perpetrator

“Many experts continue to speculate on why it took so long for Heartland to identify and disclose the breach. According to the Storefront Backtalk report, the payment processor revealed the breach was first discovered in late October or early November, whereas previous statements indicated that it was only in the fall. The company has had two outside forensics teams and the Secret Service working on the problem for more than two months, and yet the “sniffer” software used to collect the data was located only last week.”

If this turns out to be true, heads should roll.

Heartland Breach

Heartland Payment Systems acts as a payment gateway for credit card transactions for over 250,000 businesses.  At some point a sniffer was installed in their data center intercepting all transactions.  Some media outlets are calling this the “largest data breach ever”.  They process “100 million credit card payments a month and more than 4 billion transactions per year” but currently have no idea when the malicious software was installed.

Most states (and federal and industry regulations) strictly mandate how breaches are reported to consumers and how quickly.  Unfortunately this incident falls into a bit of a gray area in that consumers are 2 steps removed from the breach.  As best I can tell, heartland simply has to notify their customers (mostly restaurants and other businesses) and then its the responsibility of these 250,000 or so businesses to inform their customers.  I assure you that some will slip through the cracks or intentionally not be notified by small businesses fearing bad PR.

Heartland just launched a site to provide some positive PR and is sending it to their customers (not end consumers).  They did not distribute this URL to the general public.  The reason this entirely new domain (that does not contain “heartland” at all) was launched is because Heartland’s main site makes no mention of the breach at all.

After reading the heartland statement by Robert O. Carr, CEO it becomes abundantly clear where their loyalty and concerns lie.  With statements like:

“In fact, since our disclosure of the breach on Tuesday, January 20, 2009, more than 400 new merchants, new payroll clients and new check management clients have demonstrated their continued trust in our services by joining as new customers.”

This is clearly damage control.  It’s in poor taste to mix marketing with breach notifications.

“As a cardholder, you will not be held financially responsible for any unauthorized transactions. You should regularly monitor your card and bank statements and report all suspicious activity to your issuing bank (the bank that issued the card, not the card brand).”

That last statement is the only thing that even makes reference to the end consumers whose data was compromised.  Most breach laws require that the responsible party (Heartland) purchase credit card monitoring services for a year for each effected person.   This statement indicates that “you will not be held financially responsible” but does not provide the why or how.  This statement does not indicate that Heartland will reimburse you as a consumer nor does it say they will purchase credit monitoring services for you.

This is just another example of how we, as modern consumers, need to take responsibility for our own safety and proactively monitor our own accounts.  We are obviously in this alone.

Conn. Teacher Cleared of Felony Endangerment in Pop-Up Case

The case against Connecticut substitute teacher Julie Amero has finally
come to a close.  Prosecutors dropped the felony charges against her,
but the agreement called for a guilty plea to a misdemeanor charge of
disorderly conduct and surrender of her state teaching credential.
Amero had previously been convicted of endangering minors and faced 40
years in prison.  Prosecutors alleged that in 2004 she had surfed to
dubious websites that displayed pornographic pop-ups on a computer in
the classroom; when security specialists caught wind of the case, they
pushed to examine the computer in question and found that the school
district had inadequate anti-malware protection on that computer and the
pop-ups were not Amero’s fault.

This is easily one of the most frustrating InfoSec stories of recent years.  In case you are unaware, some poor substitute teacher in Conn was using a computer in a classroom when a flood of pornographic pop-ups (induced by malware) came on the screen.  She found herself in court facing Child Endangerment charges and up to 40 years in prison.

This highlights how scary our legal system can get.  If you have no idea what a case is about do not try to render a verdict.  Defer it to another judge, a jury or call in some experts.  For gods sake, don’t sentence someone for not doing anything wrong.