[cobalt-users] HTTPD dies and won't restart. semaphore problem.

[Mike Jeffers <mjeffers@xxxxxxxxxxxxxxxxxxxxxxxxx>]

Hey everyone. I'm reposting this as I got no response before. I'm hoping
someone has an idea of where I can go from here.

Thanks!

---------- Forwarded message ----------
Date: Fri, 1 Aug 2003 08:47:22 -0500 (CDT)
From: Mike Jeffers <mjeffers@xxxxxxxxxxxxxxxxxxxxxxxxx>
To: cobalt-users@xxxxxxxxxxxxxxx
Subject: HTTPD dies and won't restart. semaphore problem.


We have been plagued for several months now by a mysterious problem on our
RAQ4r. Apache dies unexpectedly and won't restart with the same command
line dictated by the rc.d/init.d service file.

Whenever we try to restart apache the following error message appears in
the error.log file:

[Mon Jul 28 15:01:09 2003] [warn] pid file /var/run/httpd.pid overwritten
-- Unclean shutdown of previous Apache run?
semget: No space left on device
semget: No space left on device

After poking around a little more we found that there were an excessive
amount of semaphores belonging to HTTPD hanging around. It wasn't until we
used the ipcrm function to "kill the httpd semaphores" that we would see
apache successfully restart.

Only for a short time now have we been graphing via MRTG the httpd
semaphore count which can be seen here:

http://www.adaptivedataworks.com/mrtg/gerald/sem.html

This graph shows that apache crashed around 128 semaphores and it was
restarted almost immediately (with our manual intervention). The
importance of this graph shows the slow growth rate to at least give us an
idea of when it'll happen again.

Can ANYONE please point us in the right direction as to how to figure out
WHAT or WHY this is happening?

Many many many thanks in advance,

Mike Jeffers
Adaptive Data Works, LLC
http://www.adaptivedataworks.com