Re: [cobalt-users] Strange RaQ3 Crash...PHP???

["Steve Werby" <steve-lists@xxxxxxxxxxxx>]

"Bradley Caricofe" <caricofe@xxxxxxxxxxx> wrote:
> This morning I'm surfing around the site checking things out and all of a
> sudden it's inaccessible.  Cannot bring up any sites on the server nor can
> it be pinged.  I was playing with the php bulletin board when this
happened
> and had just logged in as admin.  My isp apparently ran into some problems
> bringing the server back up but two hours later it is up and running,
> however all traces of my php installation are GONE.

A service or program crashing would not delete PHP.  Since you installed PHP
from source PHP is in the form of a loadable module called libphp4.so,
probably located in /usr/lib/apache unless you installed it to a
non-standard location.  If that file is gone I suspect that your ISP
reloaded your system from backups they had.  Please check whether the file
is there and try restarting Apache from the commandline.

> I haven't been able to determine the exact cause of the crash yet, I just
> know I was messing with a php based application when it happened.

High load could have caused your server to crash.  But I doubt it was the
result of a flaw or security vulnerability in PHP.

> Has
> anyone had any similar issues with php or with a RaQ3?

Between my own boxes and clients', there have been times when Apache went
down.  But restarting it from the commandline always worked.

> I remember thinking
> to myself that I needed to order an additional IP for the bulletin board,
as
> it was the only login on the site that was not ssl secured and all my
other
> ip's were being used.  Is it possible that someone sniffed my bulletin
board
> password

Yes.

> and exploited something within php

Something within insecure (poorly written) PHP code with insecure passwords
on a box that has a vulnerability, yes.  An exploit within PHP itself (v
4.1.2), not likely, but there's no way to rule that out.

> which resulted in it
> uninstalling or corrupting itself?

No.

> The server is probably in need of rebuilding, many, many files located
> throughout the system have strings of
> UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU
> all throughout them.

Can you list the names of some of these files?  Are they config files in
/etc, webpages, etc.?  Were they files that existed beforehand?  It would
probably be helpful if you posted a list with file names, paths, ownership
and permissions and possibly links to a few on the web so we can take a
look.  What you found is *very* suspicious.

> I'm told by
> the isp that these are artifacts of a disk restore they had to do?

That needs further explanation.  I've never done a restore from backup files
or an OS restore CD and had anything remotely similar happen.

> My
> system was never backed up by these folks that I know of, so I'm not sure
> what the disk restore process they are referring to entails.

Ask if they restored from the OS restore CD.  If they did and your sites and
users are still there, they must have backed them up first or already had
them backed up somewhere.  If they're not backing the server's files up
regularly I hope you are.  Be proactive.  Don't create and test a backup and
recovery system after you need it.  Have one in place before you need it.

> I don't want
> to rebuild this system until I know what happened.  Logs on the server
don't
> show much at all, they just stop recording when it crashed and start again
> when it came up 2 hours later.

Any possibility that the server's connection to the internet went down for 2
hours?  That someone pulled the connection out of the back of the server
accidentally?  Was cron still running?  See /var/log/cron.  If you're just
referring to the Apache logs, that may just mean that Apache was down.  If
the ISP reloaded the OS from the OS restore CD none of the logs should be
there.  I suggest you find out exactly what this restore they did entailed.

> I did run the latest version of chkrootkit,
> it says all good.  Can anyone tell me where else to look for info on what
> caused this?

I'd start by getting better explanations from your ISP.  You may want
someone with troubleshooting and security experience to take a peek in the
box too.  I do this kind of thing all the time.  If I were in your shoes and
thought there may have been a security vulnerability I'd strongly consider
re-installing the OS.  But at this point you really don't have enough good
information to know what happened.

--
Steve Werby
President, Befriend Internet Services LLC
http://www.befriend.com/