RE: [cobalt-users] RaQ4 low on memory

[Graeme Fowler <graeme.fowler@xxxxxxxxxxxxxx>]

James wrote:

> Total memory is:	648720 KB
> Used memory is:	627840 KB
> Free memory is:	20880 KB
> Percent used is:	96
<snip> 
> Aug 28 00:07:51 www kernel: VM: do_try_to_free_pages failed 
> for kswapd...
> Aug 28 00:07:58 www last message repeated 6 times
> Aug 28 01:37:31 www kernel: Unable to load interpreter 
> /lib/ld-linux.so.2
> Aug 28 01:37:32 www last message repeated 2 times

Whoa, not good at all... your server got so memory-starved just prior to the
first of these messages that it hit a problem in the kernel's virtual memory
handler code. Looks like that killed (or at the very least zombied) kswapd,
which now means it's unlikely you're using any swap memory.

The death of kswapd leads to a horrid slow death for the machine as it
becomes incapable of either pushing to or reading back from the swapspace.
Bad.

I'd suggest you have a cron job (or several) which run in the small hours;
between them and the rest of the services on the machine they've consumed
all available RAM on the machine and just about killed it. Luckily it had
enough grunt left to recover, albeit flakily.

The way to recover? A reboot, I'm sad to say. Boom goes your uptime ;-)

After that you'll need to spend some time making sure billions of things
don't happen at once. And it might be worth looking through your webserver
access logs and maillogs to see what else was happening at the time, too.

HTH

Graeme
-- 
Graeme Fowler
System Administrator
Host Europe Group PLC