Arpwatch is an amazingly useful tool that promiscuously listens on a specified interface for arp broadcasts.  It takes what it learns and saves the the output in a database for later reference in the following format.

mac_address ip unix_date/time hostname

It will take any changes/additions and log them to /var/log/messages as well as optionally emailing them.

This functionality is useful for detecting

• Man-in-the-middle attacks
• Arp spoofing/poisoning
• Session hijacking attacks
• New hosts introduced onto your network

Set up and configuration is easy.  Just download and compile arpwatch from lbnl’s site, create an arpwatch user (unless you want it to run as root… which you don’t), create an empty arpwatch database (touch/home/arpwatch/arp.dat) and run it.

The command line arguments you run will differ depending on how your network is set up, so check out the man page to be safe. The following should work for most situations.

/usr/sbin/arpwatch -i eth0 -u arpwatch -f /home/arpwatch/arp.dat -n x.x.x.x/21 -e -

-i eth0 tells it to listen on /dev/eth0 only.  You can run multiple instances of arpwatch for each nic/network if you are multihomed.

-u arpwatch tell it to run as the user ‘arpwatch’ instead of root.

-f /home/arpwatch/arp.dat tells it to save the arp database in that file instead of the default location

-n x.x.x.x/21 tells it that an additional address range is in use on this interface.  If you have IPs outside of those defined on your monitor nic it will report them as bogon.

-e – tells it not to email you with every thing it discovers.  You will want to run it this way the first time to avoid flooding your mail box.

I am a firm believer in David Allen’s GTD (or Getting Things Done) and have been searching for a nice and easy to use task tracking system. I am also a huge fan of all things gui-less so naturally I started coding series of scripts for the purpose and GeekTool to display todo’s on my desktop.

Initialy it was nothing more the following line added to my .bash_profile

alias todo=’echo $1 >> ~/todo.txt’

But before long I found myself wanting to write a ‘done.sh’ script to remove items. At that point I realized I had to implement a numbering system, use copious amount of awk and sed and spend far more time than I cared to on the project.

At some point I recall briefly reading about something similar on digg. A quick google search lead me to todotxt.com.

What a system this is! Combined with GeekTool I have an excellent way of staying organized.

Rather than explaining how it works I embedded an example video for you.

Now all I have to do is implement a system that uses DUE:MM/DD and a cron’ed script to alert me via growl when something is due. Ahhh, if only I had some free time.

After giving notice at my last job I found myself whipping together a lot of documentation for the person who would be taking over for me.

He really enjoyed the “Linux Basics” one I put together and said it would be a useful thing to stick on my blog… so here it is.

Note: Please forgive any odd formating, it is taken from a wiki.

File System

/ : root of the file system contains all devices and directory’s

/root : the root users home directory

/home : all other users home dirs reside in here

/boot : All the kernels and boot specific info

/tmp : temporary files are stored here, is commonly world writable so keep an eye on it

/dev : on linux even hardware devices are part of the file system, they are stored here.

/bin : executables that should be safe for normal users to run

/var : the system writes data here during its operation, commonly contains /var/lib/mysql and /var/www

/opt : optional software, 3rd parties stick stuff here

/sbin : system executables that only root should need

/proc : the OS uses this to keep track of everything on the system in real time. No need to muck around in here

/mnt or /media: this ware new file systems get mounted (cds, floppys, flash drives)

/etc : all config files

FS NOTE: when tweaking configs do ‘cp something.conf something.conf.bk’ and tweak away. If you flub something up just ‘rm -f something.conf; mv something.conf.bk something.conf; service restart something’ and your back up and running with your original config.

Basic commands

• whoami : displays current user
• top : displays the top cpu/memory eaters and system load.. like task manager on windows
• ps : displays all processes running.. ps aux is the most useful way to run it
• wall “some text” : sends a broadcast message to all logged on users
• man program : displays the ‘man page’ or manual for a given program. Uber useful. Use space bar to page down and q to exit
• program -h : displays the help for a given program, briefer than man
• du -sh dirName : Displays the total size of a directory recursively
• df -kh : displays total and available storage on all partitions for the system
• locate filename : finds ware a program or file is located on the system
• w : displays who is ssh’ed or logged in.
• watch -n seconds filename : will execute a file every n seconds. Useful to watch who is online, watch -n 3 w
• wget http://somesite.com/somefile : gets a file via ftp, rsync, http, etc from a remote host.
• netstat : displays all listening ports and active connections
• ifconfig : used for listing network interface info and setting it
• clear : clears the terminal
• md5sum filename : displays the md5 checksum of the given file

additional command operators

|
the pipe is used to send one command through another.
ps | more -- pauses ps
ps | grep ssh -- only display lines that contain ssh

;
used to "stack commands" or issue multiple commands on 1 line.
cd ..; ls

&
puts a command in the background.  Will let you know when the command is finished

>
write what is displayed on the screen from a given command to a text file
ls -alh /root > /root/myRoot.txt

>>
appends screen output to an existing file

File Permissions

 

Listing Permissions

ls -al will display all files in a list with their owners and permissions

-rw-r--r--   1 irq13 irq13 1006 Jan 24 10:16 .bashrc

Now to break down the above example…

-rw-r--r-- is the permissions area.  The first - would be d if the item is directory, otherwise it will be -.  The second 3 dashes indicate read/write/execute for the owner, the second is r/w/x for the group and third is r/w/x for everyone else.

The next number is the inodes associated with the file. This isn’t important for you to know the basics

Next when it says irq13 irq13 that indicates the owner of the files name group

Changing ownership of a file

chown username:groupname file

Changing permissions of a file

chmod XXX filename

chmod uses a numeric system for assigning ownership. XXX represents 3 numbers. The first is the permissions applied to the owning user, 2nd is group, 3rd is everyone else.

1: execute
2: write
3: write & execute
4: read
5: read & execute
6: read & write
7: read, write & execute

Remember that 777 is only to be used as a trouble shooting step to rule fs permissions out. NEVER leave a dir as 777. Its useful to do ‘ls -alh * > perm_capture.txt’ before messing with a file. That way you can restore its original permissions.

Attributes

Files also have attributes, similar to the ones found in the windows world.

lsattr filename : Lists the attributes of a file or directory

chattr +-=[ASacDdIijsTtu] filename

to add an attribute use + to remove use –

File Attributes

append only (a)
compressed (c)
no  dump  (d)
immutable  (i)
data journaling (j)
secure deletion (s)
no tail-merging (t)
undeletable (u)
no atime updates (A)
synchronous  directory  updates  (D)
syn-chronous updates (S)
top of directory hierarchy (T)

Use man chattr for an explanation of each attribute

launching scripts and bins

• If an executable file is in your path you may simply type its name from anywhere on the system and it will execute.
• To see what your path is type ‘path’
• To execute a file in the current directory type “./filename
• To execute a file it must have execute permissions for either your username or a group you belong to.

User Management

useradd

useradd userName

then run “passwd userName” to set the new users pw

passwd

passwd username

will ask for the new pw twice

Service/Daemon Management

restarting/stopping/starting a service

On any init.d based linux distro you can restart a service with the following…

/etc/init.d/serviceName restart

You may replace ‘restart’ with ’stop’ or ’start’ (and in some cases ’status’).

Forcefully stopping a service

killall processName

Killing on instance of a service

kill pid

The pid can be gathered by either top or ps

Disabling/adding/listing services

chkconfig –list

displays all the services and if they are set to run in different runlevels

use the –del daemonName to remove a service or –add daemonName to add one

setting a program to run at startup

Add a line executing the command at the end of /etc/rc.local

File Manipulation

Editing Text Files

vi is by far the best text editor but has a learning curve to it. If you want simplicity use nano

display a text file from the command line

cat filename

or

more filename

Display the last few lines of a text file

tail filename

or you can display the last 50 lines of a file with…

tail -50 filename

or you can display lines as they are written to a file (or follow) with the following: (UBER useful for log files)

tail -f filename

copy a file

cp filename destination

move a file

mv filename destination

 

delete a file

rm -f filename : removes the file. -f makes it so it doesn’t ask you if you are sure

Displaying the differences between two files

diff file1 file2

Installing crap

On redhat derived systems (RedHat, Fedora, CentOS, Rocks, Mandrake, etc) yum is your package manager.

yum install appname : installs the application from the remote yum repository

yum search appname : does a search on the repository for a given program

yum remove appname : uninstalls an app

use ‘man yum’ for a complete list

archives

tar.gz or .tgz is the most common compression found in the linux world. that is tared (Tape ARchive) and gziped. Sometimes called “tar balls”.

tar -xzf file.tgz : will X’trackt a tar/gzip file.

tar -czf myfile.tgz someDir : will create a tar and gziped archive of the given directory

gunzip : un gzips a file

unzip : unzips a .zip file

Linux Security

Read these this SANs Checklist (www) (pdf) and install Bastille Linux.

‘I give up’ is not a phrase you will hear from me all that often. But I just can’t take any more. Novell has me at my wits end. I can’t believe people use this with any sort reliability.

Throughout my months of toying with it I have issues and stopping blocks with each and every component. Some servers require many, many components to effectively work.

Here’s a brief run down of just a couple of the annoyances:

Updates and patches come rapid fire (about two per day) and often leave the system broken. I have had them cause dependency issues each time I have applied them. This will do crazy stuff from switching the physical network card that eth0-2 are assigned or out right breaking NSS. In fact, every update I have run broke NSS. You just can’t have that in a production environment. Technically you could script an auto-updater, however, per Novell support “Automating the updates might have its own risks [...] because of that, rug doesn’t have a –force option the way RPM does.”

Things that should be done by installers must be done manually. A great example of this is having to manually enable remote administration of a GroupWise server. For example, you need to share out /usr/local/gw using samba. But first you have to install and configure samba. That’s essentially all the docs say on the subject is to ‘install samba’. Not ‘Download package X, install it using command Y, tweak this directive in X.conf, and so on’. So I installed Samba from source. After struggling to get it integrated into the eDirectory I discover Novell-Samba. Who knew, they just said ‘Install samba’.
The install process for the OS and packages drives me insane! The OES cd set consists of 10 CDs. During the initial install you are asked to supply almost all 10 CDs in varying order and you have to re-insert a number of them multiple times. It also asks for the Suse Core 2 CD2 and 3. Which end up being the Suse Linux Enterprise Server disk 3 and 4. I figured that out just out of desperation and feeding it random CDs.

The documentation is lacking. It assumes that all Novell customers are intimately familiar with Novell terminology and technology (see previously mentioned GroupWise/samba example).

GroupWise acts as an open relay by default and no settings changes will help that. Users hate the GroupWise client, the outlook plug-in makes Outlook buggy and slow. The cross platform GroupWise client (Linux and Mac) is really bad. The only way to remedy this is to purchase an expensive third party app

I purchased the only (at the time) official Novell Press book for Open Enterprise entitled “Open Enterprise Server, Administrators Handbook, Suse Linux Edition”. Being the only official book I assumed it would be comprehensive and cover anything and everything relating to OES. What I found was that it is entirely based on a pre-release version of OES and a large number of important things have changed since it was published. In fact, a couple of things the book tells you to do regarding updates will break an otherwise happy server.

Overall I would just Novell to hammer all these things out, test thoroughly and make the docs useful. Don’t assume everyone using the product is a 15 year Novell-Netware veteran.

I recently wrote a cluster article that made it to the front page of digg.com. This was by far the most traffic my site has ever seen and I was very happy to see it happen.

Shortly after being digg’ed (dugg, digg’d, eh never mind) I started thinking of what I could do to best spend my time while all the traffic was coming in, so I SSH’ed into my server. Here are fun things I came up with to do while being digg’ed

‘tail -f‘ your apache dom logs. Watch all the different IPs flow by, all looking at the same page. Very rarely did I see anyone poke around. They always just checked out the one article and left. I’ll have to tweak the site a bit to make it more sticky.

‘iptraf’. Its a hoot. Watching all the connections come in (80 new connections in about 10 seconds) I couldn’t help but flash back to working the night shift as a sysadmin at Liquidweb. I would always run this utility on a machine that was on the receiving end of a DDoS attack. The effect is very similar.

Knowing “the greater your exposure the greater your risk”, I started to get paranoid. I checked my /tmp and /var/tmp (and all other world writable dirs) quickly to look for any odd files. I ran ‘netstat -a‘ a few times to make sure no one was poking around on ports they didn’t belong on. Then I ran ‘tail -f /var/log/messages‘ for a while. In doing so I found someone start trying to brute force my ftp daemon. I grep‘ed my domlogs for the ip and find he was referred to my webpage from digg. Ha! Being paranoid paid off! I added the IP to my firewall’s black list and kept watching for a while.

I was then informed by my friend Shelby that I should really be running bsuite for wordpress and google analytics. Google wasn’t accepting any new users so I scrambled and installed bsuite. I must admit that it is pretty damn cool for generating blog specific stats.

Throughout this entire process I was amazed to see that the large amounts of traffic barely made my little ole P4 (webmaster series from Liquidweb) break a sweat. The load hung around 0.5 at max.

After all was said and done, I went from an average of 20 unique visitors per day to 5,000 for the last 2 days. It brought me a total of 11,000 unique visitors over the past 4 days and is still growing as tons of other sites linked to it.

So to recapp,

• ‘netstat -a‘ to ensure no one is poking around
• regularly check all 777 directories
• ‘iptraf‘ to watch it all go down
• ‘tail -f /var/log/messages‘
• ‘tail -f‘ your apache domain logs for your site.
• ‘uptime‘ or ‘top‘ to monitor your load. ‘watch -n 30 uptime‘ will refresh it every 30 seconds.

Thanks digg’ers. I’ll keep writing them, if you keep coming and reading them. Remember that I have a bunch of other useful posts in my various categories, stay a wile and poke around

This cluster was created for Dr. Shin Han Shiu for use decoding plant genomes and other bioinformatics related tasks.

Software:
It runs the Rocks clustering distribution that is based on CentOS (which is based on RedHat Enterprise Linux) and is pretty much a cluster on a disk. It contains ganglia for monitoring, MPI and Sun Grid Engine (SGE) for task queuing, pre-configured kickstart server for painlessly building computing nodes, pre-configured dhcp, nfs and other network services, apache, mysql and just about anything else you will need for a cluster.

Architecture:
One head and many nodes is the basic idea. The head is the machine that contains all the private and public daemons to run the cluster. It consists of two gigabit network cards. One with a public ip plugged into one of my public switches and the other plugged into a dedicated cluster switch with no uplink to the internet or public network. All of the nodes contain one network that is plugged directly into this ‘private’ switch. You can think of it as a NAT network with the head acting as the firewall/router.

Hardware:
This particular cluster was pieced together out of pieces to fit specific needs. We wanted to maximize the CPU qty per rack units used so we went with the Dual AMD Opteron machines with Dual Core CPUs. That gave us a total of 4 CPUs for what was supposed to be 2 rack units of space. Some where along the line between the university’s purchasing department and the vendor we ended up with 3U machines.

Each machine has a DVD rom so I don’t need to swap CDs during OS installs.
Each node uses a Tyan Thunder K8SD Pro motherboard because it supports our CPU choice, it has integrated video and dual gigabit ethernet, PCIX slots to support raid controllers and a can accept up to 32gigs of PC3200 ram.

The head contains 2 3Ware SATA RAID controllers. One is an 8 port that has 1.5 terabytes of storage running RAID5 for the storage array. The other is a 4 port that has 3×120gig drives running RAID1 with 1 hot spare for the OS drive.

We then have Type 1 and Type 2 nodes, the only difference being the Type 1 has 2 gigs of RAM and the Type 2 has 16. The qty break down is as follows.

Head: 1
Type1: 3
Type2: 1

setup:
The setup should have been far easier than it was. Lets just say that I have had a heck of a time making 3Ware sata raid controllers work on x86_64 RedHat based linux based OSes. It has made me decide that I will be using LSI MegaRaid cards from now on. This isn’t the first server I have had these problems on.

head setup:
I stuck the boot DVD in the head and at the boot prompt typed ‘frontend’ to indicate that it shouldn’t try and boot a kickstart install. That feature is very useful as you will be installing far more computing nodes then you will heads (one to many).

The OS install is fairly similar to any other text based RedHat install. I think it said it could do a graphical install but apparently it doesn’t support the integrated ATI XL on the Tyan mother board.

I won’t go through step by step, but I will spew out a few useful tidbits of wisdom that I discovered along the way.

Let it automatically set up your partitions. It was smart enough to turn my raid5 array into storage and break up the OS specific partitions on the smaller raid1 array. The partitions it set up for the OS where very sane.

Watch closely as you configure your networking. One is for your public interface and the other is the private. It will annoy you if you are trying to whiz through and assign them in the wrong order.

I installed every Roll (Rocks packages) available on the dvd as we plan on using both MPI and SGE.

Step by step head setup.

Node setup:
On the head you will need to log in as root (either via ssh or locally) and run ‘insert-ethers’. This is a sort of wait-for-call screen like we had in the bbs days, only its used for initiating a kickstart with a node. Those familiar with redhat’s kickstart may wonder why they have to do this instead of just leaving the kickstart server running. The answer (as best as I can determine) is security. This way a machine can’t be introduced into your network without you (root) initiating it.

Next we stick the boot DVD or CD into the first node. Being that this is the first rack (referred to as cabinet in the ROCKS docs) and the first node it is compute-0-0. So the if you have multiple racks, the 3rd node in the second rack would be compute-1-2. That is if you choose to stick with their naming scheme. Honestly, I see no reason not to. Its descriptive and well thought out, besides you can assign the public interfaces host name to anything you like.

After it boots from the cd/dvd and connects to the kickstart server you will see it on ‘insert-ethers’ on the head. You can identify it by the MAC address that it displays. At this point the head sends the kickstart information to the new node and registers it with all necessary services including dhcp, its mysql database and so forth.

Step by step node install

“Rinse and repeat” for each of the other nodes and you have a fully functional HPC (high performance computing) cluster.

I am in the process of setting up my first Rocks cluster and have
encountered a problem that I can ‘t figure out.

I have my front end set up and am attempting to kickstart my first computing
node. I run install-ethers on my head, boot the first node from the rocks
dvd and wait. The head shows the correct MAC attaching and then displays
404 as the status. Looking at the node I see the error stating…

https://10.1.1.1///install/sbin/public/kickstart.cgi?arch=x86_64&np=4&projec

t=rocks

Ftp IE Error

I checked /var/www/html and see no install or sbin. I found the files it is
looking for in /states/partition1/home/install/sbin so I symlinked it
adjusted the httpd.conf accordingly and still no love.

Even if it worked as I expected I still can ‘t account for the extra two
front slashes after the ip.

I have inherited an Active Directory Domain and Exchange Server when I started this job. I want to be rid of both of them and use only linux stuff. I ‘m pretty sure I can just remove the AD server and replace it with smb/radius or smb/kerberos but I want to be able to keep some sort of exchange server compatible mail daemon. They have a bunch of projects that require calandar sharing. Does anyone know of such a beast? And if I remove the AD server and replace it with smb will it kill the exchange server? My guess is yes.

I am having far too much fun for a Sunday night. I ‘m at work tending to the datacenter when I receive a notice of an outgoing DoS attack. I scramble to find an ircbot/DoS perl script on the servers /tmp dir. It was most likely placed there through a crappiply coded web app.

I decide to view the source of the file and figure out what it does. Its coded in Spanish. I dont speek spanish but this isnt a huge deal. Its almost like reverse engineering a virus or worm. You have to assume that the variables are named to be misleading or vauge to make it harder on the person reverse engineering it.

So I figured out the irc server, port, name sceme and chans that it connects to and I configured iRCN to connect to them disguised as a part of the botnet.

Once connected I kill the pid for the bot on the infected server and watch it drop off in the bot net channel. How neat.

Now Im waiting for commands to be issued to me to start DoSing people.

Welp, I installed Mandrake 10 on my desktop and am much happier with the results. Evolutions works, GAIM works, my display is better than before but still not perfect.

I figure with a little tweaking I will get all the monitors working properly and not mirroring each other.

Ahhhh, its nice to not crash or count on windows.

On another note… I took my “Designed for WindowsXP” sticker off my computer and put on my toilet right above the flusher handle. Its better suited for there.