Re: [cobalt-developers] restore original binaries 'ps', 'last' etc.

[Michael Stauber <devel@xxxxxxxxxxxxxx>]

Hi Jeff,

> You
> can have a kernel hack; some of them are almost invisible.  You can also
> have replacement files on your computer so the checksum and directories
> will look right even though they aren't.  Both of these hacks are
> somewhat common in Linux.

Yeah, some of 'em are almost undetectable. See 
http://la-samhna.de/library/lkm.html for a good summary on that issue.

I've run into a few boxes which had SuckIT installed, which is (if properly 
installed) almost undetectable unless you sift through the memory in realtime 
looking for the right giveaways.

Just this week I checked a customers box which had been missing a few critical 
patches (/usr/lib/authenticate, ModSSL and to boot it also did sport 
OpenSSH-2.9.2 <sigh>).

There were no visible signs of intrusions. Even "rpm -Va" showed nothing 
unusual and you could md5sum and compare binaries without getting any odd 
results. Comparing /sbin/initd bit by bit with one from a good box also 
returned no differences - at all. 

CHkrootkit didn't find anything either, except for some deletions in wtmp. A 
portscan from the outside showed no unusual ports being open. So except for 
the deletions in wtmp everything looked perfect.

However, the box pulls 6 Gigs of traffic per day, which is a dead giveaway in 
this case as the normal logs report next to no activity.

Solution? Well, OS restore. What else? <sigh>

Recommending to people to stay up to date on patches gets kinda pointless with 
such crappy patches from Sun (XTR Kernel, anyone?) which are often worse than 
the issues which they fix. :o(

Speaking of the XTR Kernel update: Just got off the hook with a good customer 
whose XTR was also wrecked by the Kernel update. Two hours after the update 
his RAID5 array went titsup.com up and his /home is now empty.

-- 

With best regards,

Michael Stauber
mstauber@xxxxxxxxxxxxxx
Unix/Linux Support Engineer