Re: [cobalt-users] [RAQ3] Runaway process control?

[Grant Stern <grant@xxxxxxxxxxxxxx>]

On Tuesday, October 29, 2002, at 05:20 AM, 
cobalt-users-request@xxxxxxxxxxxxxxx wrote:

> We have a client whose site runs an e-commerce solution which will 
> remain nameless - anyway, their particular store has about 10,000 
> product lines - this in itself is NOTHING too problematic - however, 
> the problems occur with one of their store sections, namely 
> Semiconductors.  This category has quite a few thousand items in it - 
> which results in an HTML file of around 400kb in size.

why not make this a CSV flat text file.  that should cut the size way 
way down and help with your pattern match.

> Again - this file is not the problem - but when someone searches for a 
> particular product, and the result is on this HUGE file, the browsing 
> public causes a CGI script to read and parse the offending HTML file, 
> in order to highlight the search phrases.

or maybe you should heed the call of the evil empire and use XML, it's 
supposed to be THE THING for flat file pattern matching

> This process can take quite a long time - our research has shown that 
> their viewers are impatient enough to click the back button, click the 
> result again, and repeat the process, and repeat, repeat, repeat - 
> before getting bored.

this calls for a transition to be written into your script or page.  at 
the very least, display something while they are waiting that says, 
don't go up and back.  u could write this into the perl script, or use 
dhtml (if you're using dreamweaver,  they have whole extensions for 
this).  maybe use a cookie to limit calls based on ip to one a minute.  
or use javascript to destroy the back button.  lots of suggestions will 
work here, just sit down with the dumbest users you know (the ones that 
don't know what a web browser are, but think AOL IS THE INTERNET!!!!) 
and watch them interact with your page with one or all of these design 
fixes to stop them from ditching and starting over.

> How is this a problem?   Well... .each time some impatient browser 
> does this, the server fires up cgi to serve the page - each of these 
> cgi script has been known to consume about 60Mb of RAM!

have you thought of using PHP/Mysql for the search functions or the 
Perl DBI?  it can't use 60 MB to parse something so small.  heck, just 
using a flat file or xml file (in an xml way) should make this a 
painless procedure.  lastly, if u want to pass the burden to the 
consumer (the way of american business!), make an xml file with a flash 
front end, and let all the parsing happen on the client side.  then 
when someone complains, you can tell them to upgrade their equipment, 
while u smugly serve up all your sites on 160 MB of ram, like my raq! 
but i don't have this issue :) i know it won't be fun with a few 
thousand items to put into a new format, but it will fix everything in 
the most simple and continuing way.

> Now - I found no less than 24 of these processes today - each using 
> 60Mb of RAM - or rather - 60Mb of SWAP in most cases.  These processes 
> invariably end up sitting around for ages trying to send out data to a 
> port that is no longer waiting for it - this uses CPU and HUGE amount 
> of RAM - plus it puts the server into SWAP mode pretty darned quickly.

ouch.  i couldn't afford to design that way.

> Apart from adding more physical RAM - does anyone have a solution for 
> throttling these processes, or at least killing them off if they sit 
> for a while spinning their heels?

you posted this as a server issue today, but i see this as a purely 
design related problem.  you will be well served by reviewing your 
technique and streamlining overhead requirements, and informing users 
of what's happening behind the scenes when they have an opportunity to 
gum the works.  tho, u should seriously consider killing perl as well 
and replacing with PHP/MySQL or PHP/Flat File.  html is not made to be 
searchable, and you're using alot of resources to do it.  changing it 
into a flat file should be as simple as opening the thing in a browser 
and copy/pasting.

best of all the above solutions are guaranteed NOT TO VIOLATE your 
warranty! <snicker>

my $.024985598747372834837283,
grant