Re: [cobalt-developers] RaQ3 memory leak problems?

[Michael Stauber <devel@xxxxxxxxxxxxxx>]

Hi Jeff,

> We installed RaQ3-All-Security-4.0.1-13453.pkg on a system a while back,
> and afterwards we found out about memory programs, possibly with GLIBC.
>
> The system is now losing most of it's available memory every time
> cron.daily runs, and sendmail stops accepting incoming messages, usually
> shortly after 6 am.
>
> Yes, of course we can reboot the RaQ every morning at 6 am.  But we'd
> rather not.

Back then when the patch came out I had the VP of Sun Professional  Services 
crawling through one of my RaQ3's which was bleeding memory like hell, too. 
It was just one of many RaQs with the same problem. He agreed that it most 
likely must have something to do with the GLIBC patch and said they were 
looking into it. 

The SUN/Cobalt RaQ forum also has a lengthy and older topic on that issue with 
+40 replies, but a resolution in form of a patch does not appear to be in 
sight.

My resolution was a migration of all sites to a RaQ4R with 512MB and *without* 
that faulty GLIBC patch but even that didn't help. That box is as well 
bleeding memory and needs a reboot every two or three days. The longest time 
it stayed up in a row was two weeks, but then it crashed so badly that only a 
hard reboot brought it back.

Having many dynamic processes which spawn and end childs to handle requests 
(Apache, MySQL and ASP) seems to worsen the problem dramatically. When a 
child dies apparently not all previously used memory of it is returned to the 
system for re-usage. A few bytes remain unusable and the number of unusable 
bytes increases with each dying process.

I have MRTG set up to monitor memory usage on my boxes and it has generated a 
set of nice graphics which tell the story in all its gory details:

RaQ4R with 512 MB and without the Glibc patch:
http://ns2.solarspeed.net/~mrtg/freemem.html

RaQ3 with 384 MB and the Glibc patch:

http://admin.smd.net/~mrtg/freemem.html

The RaQ is now running just 4 very lightly used domains, MySQL is still 
active, but no longer used. That rusty old box now runs like a charm in that 
lightly loaded state. But as soon as I put some traffic on it the memory is 
gone in less than 24 hours.

I'd bet money on MySQL being the most memory degrading application here, but 
it doesn't appear to matter if it's the PKG from PKGmaster.com or the 
3.23.49a RPMs which I packaged a while ago. 

-- 

Mit freundlichen Grüßen / With best regards

Michael Stauber
mstauber@xxxxxxxxxxxxxx
Unix/Linux Support Engineer