Re: [cobalt-users] Mail Bomb ... I'm stumped

["Steve Werby" <steve-lists@xxxxxxxxxxxx>]

"Ursula" <ursulasays@xxxxxxxxxxxx> wrote:
> They weren't bounces, all were delivered to
> httpd@localhost. They all had reply-tos @hotmail (real
> accounts with storage exceeded bounces, or fake
> accounts, doubling the problem).

Now I'm confused.  Do you really mean the To header, not the Reply-To
header?  Otherwise there should be no bounce message problem on your end.

> The only vaguely suspicious output from chkrootkit was
> this:
> Searching for suspicious files and dirs, it may take a
> while...
> /usr/lib/perl5/site_perl/5.005/i386-linux/auto/mod_perl/.packlist
> /usr/lib/perl5/site_perl/5.005/i386-linux/auto/MD5/.packlist
> /usr/lib/perl5/site_perl/5.005/i386-linux/auto/Quota/.packlist
> /usr/lib/perl5/site_perl/5.005/i386-linux/auto/XML/Parser/.packlist
> /usr/lib/perl5/site_perl/5.005/i386-linux/auto/Devel/Symdump/.packlist
> /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/.packlist
>
/usr/lib/perl5/site_perl/5.005/i386-linux/auto/Msql-Mysql-modules/.packlist
> /usr/lib/perl5/5.00503/i386-linux/.packlist
> I believe the .packlist files being found are false
> positives, So I'm reasonably sure the server is safe.

Correct, those files are of no concern.  But I wouldn't conclude the server
hasn't been compromised or there isn't a hole allowing a script to be
uploaded and run or an existing script to be taken advantage of.

> There have been a few instances of this bombing, of
> various intensities, over the past couple of weeks -
> It seems odd that someone with root access would draw
> attention to their entry in such a manner (though I
> suppose nothing would surprise me about how such
> people get their jollies)

I don't know that it's a hacker and I don't know that they have root
privileges, but I wouldn't be surprised.  It's so easy to locate, download
and run port scanners and other programs and install rootkits that require
simple cut/paste knowledge and there are so many vulnerable machines that
some hackers don't really care if they draw attention to their intrusion or
that you detect it.  They simply move on.  But there are plenty who do not
want to get detected and want to utilize your server undected for extended
periods.

> It's more likely that the
> > emails were generated by
> > a script running as user httpd.  And in order for
> > the script to be executed,
> > it must have either had user, group or world
> > executable permission.  That
> > probably goes without saying, but armed with that
> > information it should be
> > easy to narrow down the candidates.  The program
> > "find" is your friend.  If
> > we thought the offending script was in a site we
> > could do:
> >
> > find /home/sites/ \
> > \( -perm -0100 -o -perm -0010 -o -perm -0001 \) \
> > -type f -user httpd -exec ls -al {} \;
> >
>
> This did show up something curious, all the stats
> files and folders generated by webalizer.

It showed folders?  The find command above uses the "-type f" parameter
which should restrict the search to files only, no directories.  If you hand
typed the command perhaps you had a typo.  I don't think it's related, but
there's no reason that Webalizer reports should be executable.  If they are
it's probably a good idea to remove the executable permissions since they're
not needed.

> Nothing
> unfamiliar stood out though. There are 160-odd sites
> on here though, rather tricky to search the content of
> each individual file.

That's why you need to use tools like find, rpl, etc. to locate the best
candidates for investigating.  Unfortunately there's no magic answer.  If
you'd like, you can run something like tcpdump to sniff all the traffic on
the server and save to disk and analyze when this happens again.  Warning:
it will eat up disk space fast so keep an eye on it if you do.

> The other httpd suspects are php and Frontpage, with
> so many files involved I'm not sure what way to check
> them all for something that may be the culprit -
> particularly as the access/error logs revealed nothing
> at all out of the ordinary correlating with the time
> the first email came through.

I might not have been clear before, but just because the script is running
as httpd does not mean it's being run by the webserver.  If it's not being
run by the webserver it won't appear in the Apache logs.  Like I said in my
previous email, I've seen executable scripts owned by httpd installed in
/home/tmp either directly or in a hidden subdirectory.  Again, these will
not appear in Apache logs because they don't run over the web.  Those I've
seen are installed and run as a continuous process and open a connection to
another server on a non-standard port to communicate.  You may want to
install nmap http://www.insecure.org/nmap/ on the server, your PC or another
server and scan the server for open ports to see if anything's reported.  Of
course, if you have PortSentry or another program which listens on unused
ports (and blocks port scans) then disable it first.

> > But I actually suspect that the file is in
> > /home/tmp/ so I'd also run the
> > script to check that path.  Or check the entire
> > server by doing this:
> >
> > find / \
> > \( -perm -0100 -o -perm -0010 -o -perm -0001 \) \
> > -type f -user httpd -exec ls -al {} \;
> >
> > Why do I suspect /home/tmp/?
> Nothing at all out of the ordinary in /home/tmp/

Go to /home/tmp and do this:

find . -name ". *"

and:

find . -name "* *" -exec ls -1 {} \;

and let us know if anything is found.  Change "." to "/" to report for whole
server if you'd like.  These are two commands I run to check servers for a
couple of types of hidden files (one type which is simply obscured - appears
as . or .. in a directory list and another which is completely invisible to
a directory list).

> Thanks for your helpful reply. My main suspects are
> still a disguised php file

By that I assume you mean one using an AddType directive within .htaccess to
have files with an unusual file extension parsed by PHP.  Easy enough to
find.  Install rpl http://www.laffeycomputer.com/rpl.html and run it like
this:

rpl -sR "AddType" "AddType" /home/sites/

The -s is simulate (reports matching files, doesn't change them) and -R is
recursive.  There are other tools to find/replace text, but IMO this is the
simplest.  If you wanted to speed it up you could probably change
/home/sites/ to /home/sites/*/web/.htaccess.

>, or maybe a Frontpage
> exploit, but digging up the culprit is proving to be
> an enormous headahce.

I'm going to repeat this again - there is no guarantee that it's the result
of a file served by Apache.  In fact, that is likely something that can be
ruled out based on the fact that you checked the logs and found nothing
suspicious.  If it was served by Apache that means either it's in the logs
and you just didn't notice it (doubtful), you didn't recognize a file as
being a potential culprit (possible) or a modified version of syslogd is
installed that doesn't log accesses of this particular file (feasible, but I
haven't seen this done in the wild for Apache logs, have for other logs
though).  So please do not rule out a file owned by httpd not running over
the web.  Please.  :-)  Also, please update us on your progress, especially
if you've located the cause, so that we can learn from what you found.

--
Steve Werby
President, Befriend Internet Services LLC
http://www.befriend.com/