Re: [cobalt-users] CPU heavily loaded, low on memory, smtp server not responding

["E.B. Dreger" <eddy+public+spam@xxxxxxxxxxxxxxxxx>]

> Date: Tue, 22 Jan 2002 17:50:33 -0500
> From: Brian Rahill <cobalt@xxxxxxxxxxxxxxxxxxxxxxx>

> Thanks for the info. Regardless my box was evidently under some
> serious strain.

Correct.  I just hate to see someone add RAM, upgrade CPU, etc.
when disk performance is what they need. :-)

> It's hard to say on any of these as I didn't catch it "in the
> act" The load average is fine now back to around 0.17.

Well, this is good if it lasts.

> I got some many "bad referral" message it was crazy.  I'm used
> to about 10 lame server messages and about 1 or 2 bad referral
> messages every hour or so.  But hundreds within an hour is
> unprecedented.
> 
> >That's odd.  Maybe coincidence, maybe not.  I've never tried
> >running BIND with such a high loadavg;
> 
> Do you think it was related to the load average then?

Here's where I must 1) take a SWAG or 2) look through BIND
source.  I'll do #1 for now.

It's plausible.  With 100 processes all waiting, it'll take BIND
a looong while to get anything from disk.

That said, my gut feel is that I doubt it.  I don't know that
BIND has any realtime intelligence.  i.e., I would expect it to
time out, not declare itself unauthoritative.

I guess that one could run 100 "bonnie" sessions on a nameserver,
and see what happens...

> >Start "top" at root.  Press "M" --
> >Also, if you
> >ps ax | grep http | wc -l
> 
> Thanks for all the great commands.  Again, this box currently
> isn't acting up, but next time I'll use these.

The first one shows running processes sorted from greatest memory
usage to smallest.  RSS is "resident segment size".

The second command was an example for checking number of commands
containing "http" running, which should be mostly or all Apache.
You might try checking for other commands, too.

Note that a mailbomb could not produce loadavg that high.  If
Sendmail quits accepting at a loadavg of 12, there had to be
_something_ else continuing to sap your system.

What services are you running?  Apache is my first suspect, but
by no means the only possible culprit.

> I was thinking this looked like a DOS or something.  Still
> don't know.

It could be.  Or it could be heavy traffic / something ran away.

	ps ax | grep -v http | wc -l

now to see how many non-Apache processes are running.  Do the
same when it acts up.  Compare the numbers, and see what you
find.


HTH,
Eddy

---------------------------------------------------------------------------
Brotsman & Dreger, Inc. - EverQuick Internet Division
Phone: +1 (316) 794-8922 Wichita/(Inter)national
Phone: +1 (785) 865-5885 Lawrence
---------------------------------------------------------------------------

Date: Mon, 21 May 2001 11:23:58 +0000 (GMT)
From: A Trap <blacklist@xxxxxxxxx>
To: blacklist@xxxxxxxxx
Subject: Please ignore this portion of my mail signature.

These last few lines are a trap for address-harvesting spambots.  Do NOT
send mail to <blacklist@xxxxxxxxx>, or you are likely to be blocked.