I leave my Dual 2.5ghz G5 PowerMac on all the time. So I never really had a chance to notice that the cpu liquid cooling system was leaking.

I had never heard of this happening before and thought I would be one of the first to encounter such a problem until I saw the little screen print next to the large corrosive pile of dried yuck that reads “If you see liquid unplug the computer and consult the manual”.

The CPU area is completely covered by a giant radiator looking heat sink and a bezel that is only supposed to be opened by an apple certified tech. I was able to snap some photos of the fore mentioned ‘corrosive dried yuck’ for your viewing pleasure.

The liquid it uses is NOT water. The MSDS for the chemical used can be viewed here (in case you care).

Now to get apple on the horn and get my baby back up and running.

Permalink

It’s such an amazing time of year in this state. We have so many great things to look forward to.

The sky and the roads are the same washed out white color. The sky; because it would be criminal to see the sun. The roads; because they (as is the case with everything else this time of year) are absolutely caked with salt.

The snow that has gathered on the side of the road takes the most attractive black dirt color.

Some of the snow has melted to reveal all of the garbage that the lazy Michiganders toss out the window. Garbage that has been hidden under snow for most of the winter.

Between the constant freezing and thawing, and the ungodly amount of weight that Michigan law allows for trucks, the roads are at the peak of their disrepair. Anyone who lives in Michigan or has visited knows how poor our roads are.

Ya, its just downright lovely.

Permalink

MSU just got a site license agreement with Eset for Nod32. This was at a time when the collective frustrations with Norton/Symantec Antivirus where at all time high. I have noticed over the years a few very prevalent problems with Symantec’s antivirus solution.

1. The updates don’t come as quickly and often as I would like.
2. Norton is slow to release fixes for already infected machines. In some cases I find myself writing an in house fix to mitigate the damage.
3. It’s a resource hog. It’s just heavy. It drastically affects performance when real time scan is enabled (which it should be to be effective) because it’s running all disk writes and reads through its filters.
4. Anyone who has had to use their server component knows that I don’t even need to continue this sentence.

Given these sins I decided to buy a few licenses for Nod32 and keep it on my key chain flash drive ready to install on the next machine I see with a virus related issue.

On Jan 27 the Brepibot.L took a couple of my users by surprise. It was too early in its life to be detected by the campuses clamav and a few users ran the exe before I could send out my warning to the distro lists.

Norton didn’t have a def that would fix it for a couple of days. In that time Nod32 got it with no problem, and even cleaned it on a guinea pig machine.

The next day a faculty member was having issues with random word doc corruption and suspected it to be virus related. I removed Norton and installed nod32 and then updated its defs. I ran a complete system scan and oddly enough it found four infected files that Norton had not previously detected. Two of these files where OLD viruses (one was my doom and another was sober). The problem ended up being a failing usb flash drive that he had the documents on.

Now to develop a deployment strategy.

Permalink

At work it is fairly common for hard drives to die, machines to become infected with a virus, trojan or worm and the occasional compromise. It just happens when you have so many machines; the odds are against you.

For this reason I wanted to build a machine specifically for forensics purposes, similar to the Forensic Recovery of Evidence Device (FRED) systems put out by Digital Intelligence.

The first time I saw the FRED it just made sense.

• Empty removable drive bays for the drive you want to image, analyze or restore.
• Conveniently accessible ports for every imaginable type of peripheral.
• A large storage disk for storing images of drives, case files and VMWare images, etc.
• And of course, load it with every imaginable forensic and security tool you could ever need.

Here are the parts and reason for their selection:

Multi-function Panel
I used the SunBeamTech 20-in-1 5.25 Multi-function Panel. This little unit is awesome! It occupies one 5.25″ drive slot and contains USB, Firewire, SATA, Composite Video, Audio Jacks, a -TON- of card readers, two internal thermometers that display their respective temperatures on an LCD display on the front and two fan speeds controllers. This unit has a LOT of cables that really clutter up the interior of the computer. If your totally anal you may be able to zip and twist tie them up, but man; it is a LOT of cables.

5.25″ Storage Drawer
I use this to store adapters screws for the drive trays. As well as a couple of my commonly used adapters; the 2.5″ Laptop hard drive to 3.5″ IDE hard drive converter and the dual PS2 to USB.

Hardware Write Blocker
If you want your evidence to be admissible in court a write blocker is going to be a necessity. This ensures that the data isn’t altered in anyway (inadvertently or otherwise) during your investigation. I went with the MyKey NoWrite FPU. It comes recommended from the forensic’s community and is accredited for forensic investigation. It also supports both IDE and SATA and is relative inexpensive compared to other write blockers.

Removable Drive Bays
I have 3 SATA and 2 IDE. These particular removable drive bays have built in silent fans, with a nice digital temperature readout.

10 Bay ATX Case
I went with the Aerocool Masstige, mainly because it had all of the 5.25″ bays I needed in a mid-tower-ish size. Anything else with that many bays was a beast of a full-tower. It also turned out to be a strange coincidence that this case looks strikingly similar to the one used on the FRED.

Power Supply
You’re going to need a great deal of juice to power all of these drives and devices. I went with a 500w PSU.

SATA Controller Card
I used a generic PCI IDE/SATA combo. Anything will do.

FireWire/USB Controller Card
Once again, I purchased a generic PCI card for this purpose. The one I got has one firewire and one usb port on the inside. On the outside it has 2 firewire and 4 usb ports. In my case, most of those ports were taken up by my multifunction panel.

Motherboard
Be sure to use a modern motherboard with about 4+ PCI slots. These will fill quick as you add device capabilities.

RAM
The more RAM the better. If you plan on doing viral research or virtual honey nets you will want to run VMWare. The more virtual machines you have running the more RAM you need.

Hard Drive(s)
Drive images, case files, evidence collection bin; you will be using the storage on this machine for many many purposes. It would be best to run a large RAID5 array in addition to your boot drive. Your array will be fairly static as far as physical drive additions and subtractions so its not necessary for it to be in removable drive bays, however keeping your OS drive in the bays is a good idea so you can switch OS by swapping in a new drive.

Once you put all of these pieces together you will have a very useful multipurpose machine. You will find that it will come in handy for so much more than forensic analysis.

The software I chose to install and why, will be outlined in a later article.

Permalink

One of the more common questions my support team at work receives is in regards to Windows based computers starting to run slowly. This will eventually happen to all windows machines and is simply in the nature of windows.

The long-term problem is that the windows registry (the database that underlies windows and controls everything from passwords to last window locations) simply gets clogged from installing and uninstalling software.

Most Windows professionals recommend that if you want a machine to remain ‘speedy’ that you should reformat it every 6-12 months (new Windows install). Obviously not all of us have the time to do this so I will outline a few things that can be done short of formatting that will still significantly increase system performance.

Permalink

This is a rather cool illustration representing all of the undersea network cables as of year-end 2004. The thought of cables that span the length of the oceans just boggles my mind. This is a lightbox image (as are all images on my site) so you can click it to enlarge.

Permalink

I frequently get asked what I do to secure the operating systems I use. Specifically from a nuts and bolts, configuration stand point.

When I tell them I use the NSA Security Configuration Guides they are a generally surprised to learn that such guides exist. Not only do they exist; they are an incredibly helpful resource and combines a huge amount of no-nonsense tweaks that NSA uses to secure their own machines.

NSA Security Configuration Guide for OS X

NSA Security Configuration Guide for Windows XP

NSA Security Configuration Guide for Windows 2003 Server

NSA’s The 60 Minute Network Security Guide

Permalink

I used to be ‘with it’.
Then they changed what ‘it’ was.
now what I’m with isn’t ‘it’.
and what’s ‘it’ seems weird and scary to me.
It’ll happen to you!

- Abe “Grandpa” Simpson

Permalink

I recently wrote a cluster article that made it to the front page of digg.com. This was by far the most traffic my site has ever seen and I was very happy to see it happen.

Shortly after being digg’ed (dugg, digg’d, eh never mind) I started thinking of what I could do to best spend my time while all the traffic was coming in, so I SSH’ed into my server. Here are fun things I came up with to do while being digg’ed

‘tail -f‘ your apache dom logs. Watch all the different IPs flow by, all looking at the same page. Very rarely did I see anyone poke around. They always just checked out the one article and left. I’ll have to tweak the site a bit to make it more sticky.

‘iptraf’. Its a hoot. Watching all the connections come in (80 new connections in about 10 seconds) I couldn’t help but flash back to working the night shift as a sysadmin at Liquidweb. I would always run this utility on a machine that was on the receiving end of a DDoS attack. The effect is very similar.

Knowing “the greater your exposure the greater your risk”, I started to get paranoid. I checked my /tmp and /var/tmp (and all other world writable dirs) quickly to look for any odd files. I ran ‘netstat -a‘ a few times to make sure no one was poking around on ports they didn’t belong on. Then I ran ‘tail -f /var/log/messages‘ for a while. In doing so I found someone start trying to brute force my ftp daemon. I grep‘ed my domlogs for the ip and find he was referred to my webpage from digg. Ha! Being paranoid paid off! I added the IP to my firewall’s black list and kept watching for a while.

I was then informed by my friend Shelby that I should really be running bsuite for wordpress and google analytics. Google wasn’t accepting any new users so I scrambled and installed bsuite. I must admit that it is pretty damn cool for generating blog specific stats.

Throughout this entire process I was amazed to see that the large amounts of traffic barely made my little ole P4 (webmaster series from Liquidweb) break a sweat. The load hung around 0.5 at max.

After all was said and done, I went from an average of 20 unique visitors per day to 5,000 for the last 2 days. It brought me a total of 11,000 unique visitors over the past 4 days and is still growing as tons of other sites linked to it.

So to recapp,

• ‘netstat -a‘ to ensure no one is poking around
• regularly check all 777 directories
• ‘iptraf‘ to watch it all go down
• ‘tail -f /var/log/messages‘
• ‘tail -f‘ your apache domain logs for your site.
• ‘uptime‘ or ‘top‘ to monitor your load. ‘watch -n 30 uptime‘ will refresh it every 30 seconds.

Thanks digg’ers. I’ll keep writing them, if you keep coming and reading them. Remember that I have a bunch of other useful posts in my various categories, stay a wile and poke around

Permalink

This cluster was created for Dr. Shin Han Shiu for use decoding plant genomes and other bioinformatics related tasks.

Software:
It runs the Rocks clustering distribution that is based on CentOS (which is based on RedHat Enterprise Linux) and is pretty much a cluster on a disk. It contains ganglia for monitoring, MPI and Sun Grid Engine (SGE) for task queuing, pre-configured kickstart server for painlessly building computing nodes, pre-configured dhcp, nfs and other network services, apache, mysql and just about anything else you will need for a cluster.

Architecture:
One head and many nodes is the basic idea. The head is the machine that contains all the private and public daemons to run the cluster. It consists of two gigabit network cards. One with a public ip plugged into one of my public switches and the other plugged into a dedicated cluster switch with no uplink to the internet or public network. All of the nodes contain one network that is plugged directly into this ‘private’ switch. You can think of it as a NAT network with the head acting as the firewall/router.

Hardware:
This particular cluster was pieced together out of pieces to fit specific needs. We wanted to maximize the CPU qty per rack units used so we went with the Dual AMD Opteron machines with Dual Core CPUs. That gave us a total of 4 CPUs for what was supposed to be 2 rack units of space. Some where along the line between the university’s purchasing department and the vendor we ended up with 3U machines.

Each machine has a DVD rom so I don’t need to swap CDs during OS installs.
Each node uses a Tyan Thunder K8SD Pro motherboard because it supports our CPU choice, it has integrated video and dual gigabit ethernet, PCIX slots to support raid controllers and a can accept up to 32gigs of PC3200 ram.

The head contains 2 3Ware SATA RAID controllers. One is an 8 port that has 1.5 terabytes of storage running RAID5 for the storage array. The other is a 4 port that has 3x120gig drives running RAID1 with 1 hot spare for the OS drive.

We then have Type 1 and Type 2 nodes, the only difference being the Type 1 has 2 gigs of RAM and the Type 2 has 16. The qty break down is as follows.

Head: 1
Type1: 3
Type2: 1

setup:
The setup should have been far easier than it was. Lets just say that I have had a heck of a time making 3Ware sata raid controllers work on x86_64 RedHat based linux based OSes. It has made me decide that I will be using LSI MegaRaid cards from now on. This isn’t the first server I have had these problems on.

head setup:
I stuck the boot DVD in the head and at the boot prompt typed ‘frontend’ to indicate that it shouldn’t try and boot a kickstart install. That feature is very useful as you will be installing far more computing nodes then you will heads (one to many).

The OS install is fairly similar to any other text based RedHat install. I think it said it could do a graphical install but apparently it doesn’t support the integrated ATI XL on the Tyan mother board.

I won’t go through step by step, but I will spew out a few useful tidbits of wisdom that I discovered along the way.

Let it automatically set up your partitions. It was smart enough to turn my raid5 array into storage and break up the OS specific partitions on the smaller raid1 array. The partitions it set up for the OS where very sane.

Watch closely as you configure your networking. One is for your public interface and the other is the private. It will annoy you if you are trying to whiz through and assign them in the wrong order.

I installed every Roll (Rocks packages) available on the dvd as we plan on using both MPI and SGE.

Step by step head setup.

Node setup:
On the head you will need to log in as root (either via ssh or locally) and run ‘insert-ethers’. This is a sort of wait-for-call screen like we had in the bbs days, only its used for initiating a kickstart with a node. Those familiar with redhat’s kickstart may wonder why they have to do this instead of just leaving the kickstart server running. The answer (as best as I can determine) is security. This way a machine can’t be introduced into your network without you (root) initiating it.

Next we stick the boot DVD or CD into the first node. Being that this is the first rack (referred to as cabinet in the ROCKS docs) and the first node it is compute-0-0. So the if you have multiple racks, the 3rd node in the second rack would be compute-1-2. That is if you choose to stick with their naming scheme. Honestly, I see no reason not to. Its descriptive and well thought out, besides you can assign the public interfaces host name to anything you like.

After it boots from the cd/dvd and connects to the kickstart server you will see it on ‘insert-ethers’ on the head. You can identify it by the MAC address that it displays. At this point the head sends the kickstart information to the new node and registers it with all necessary services including dhcp, its mysql database and so forth.

Step by step node install

“Rinse and repeat” for each of the other nodes and you have a fully functional HPC (high performance computing) cluster.

Permalink