[cobalt-users] Server Hacked! - Play by Play

[spamcatcher <spamcatcher@xxxxxxxxx>]

OK, here's the play by play, I am sure we've missed something, so feel 
free to make comments and suggestions as to what we did.

Wednesday
Got email from the CPU stating CPU overloaded. Got another email that 
webserver was down. Checked the error log and it indicated that too many 
files were open and that the web server restarted.
We figured that the script we just added had caused the problem. We 
figure we'll see if it happens again before taking the script offline.

Late Saturday, early Sunday
Got not one, but six emails from different people regarding their 
machines (desktop and servers) being portscanned by one of our servers.
Also noted that the CPU had the same problems on Friday night, Saturday 
morning as Wednesday. Same problems listed in the log.
Looked in the secure log (didn't know about "last" at this point) and saw 
a number of logins that did not originate from our ISP. Panic sets in.
Anger, embrassment, rage all ran thru our minds.
Once the room stop spinning, we looked at the xferlog log and discovered 
pretty much the same thing. We noted that there were no download of files 
out of the server except for one case - from a fourth party that didn't 
have the intuder's IP address, but that was a file that the intruder had 
uploaded. We noted that the intruder had uploaded three files. One was a 
gz file and it was in a ".adpro" directory under the "home" directory.
Turns out the intruder had installed another ftp server on the server in 
the .adpro directory. This directory is removed.
We then checked the passwd file. The intruder had created an account, 
"adpro" (id 500) in it.
The account was removed.
Rebooted server.
We then changed all the passwords on all our accounts.
Doing some poking around, we also found a ".chief" directory under the 
"etc" directory. It contained the source and binary of what appears to be 
some port scanning software. This directory is removed.
Reloaded all scripts on the server.
Rebooted the server.
Did a "last" command and noted the earliest intrusion was the Saturday 
before.

OK, before we go ahead and wipe this puppy, any suggestions as to 
anything else we may want to look at to see if we can dig up anything 
more evidence?

Regards,
Kar Mui