Re: [cobalt-developers] handling extreme load

["Steve Werby" <steve-lists@xxxxxxxxxxxx>]

"Matthew Nuzum" <cobalt@xxxxxxxxxxxxx> wrote:
> I was just appraised that a customer is making a presentation of a new
> application.  They expect the presentation will yield about 3000
> simultaneous users into the website directly after.
>
> I'm a little concerned about my server's ability to handle this load.
>
> I have to say that I'm a little skeptical that there will be this high
> of a turnout, but at the same time, whenever you say something like
> that, Murphy's law is more often right than wrong.

Before you jump through hoops tuning the server for one event, you may want
to clarify what your customer means by "simultaneous users" and get details
on how that number was arrived at.  It may turn out there is no valid reason
for concern.  In my experience, when all but the most technically-savvy
clients talk about simultaneous users, they really mean the number of
simultaneous users who will be viewing a webpage on the site.  What one
should probably be concerned about is "simultaneous processes".  Depending
on how long a user stays on a page and how much CPU time is required to
generate a page simultaneous users and simultaneous processes can be very,
very different.  Case in point - a few months ago I developed a management
interface for a company that has salespeople who sell AFLAC insurance.  For
now, it's a tool for managing the salesforce, enrolling companies and
employees and tracking and reporting results.  Though the business is quite
new it's growing quickly and the customer expects tens of thousands of
employees to be enrolled and accessing their accounts later this year.  By
then an interface for employees to manage their accounts, learn about
different insurance options and interact with the sales staff will be
implemented.  I was asked whether the system could handle 500 simultaneous
users.  After a short discussion it was clear that it could.  The customer
made the assumption that during peak usage 500 users may be using the system
at the same time.  Because of the nature of the pages the employees will be
accessing and the optimized nature of the database and queries that generate
the respective webpages, page generation will require very little CPU time
and on all, but the navigation-related pages, employees will likely spend an
average of 15-45 seconds on each webpage.  Because of this, 500 simultaneous
users should result in very few simultaneous processes.  Obviously, I do not
know enough information about your client's site to state that the same will
be true for it.  Perhaps your client's site takes longer to generate
webpages and perhaps the users will spend less time on them.  I'm not trying
to stear you away from optimizing the server to handle a high number of
simultaneous users, but I thought I'd recommend this as a first course of
action.  There is certainly plenty that can be done in terms of optimizing
PostgreSQL (and MySQL), Apache and other areas of the server.  And
statistical projections of expected load and benchmarking of expected peak
scenarios can be used to test the server under the expected peak load to see
if further optimization or controls are needed.

> I have a Raq4i with 256MB of Ram.  I run MySQL set to "medium" mode,
> meaning I used the my-medium.cnf default config file which is optimized
> for systems willing to dedicate about 64MB of RAM to the db.  I use the
> MySQL databases for many of the sites on this server.

You can configure MySQL to use more of your RAM.  Depending on the types of
queries (ratio of reads to inserts, ability to delay inserts) there are
configuration options that can be adjusted to improve reads or writes.  Use
of and types of table indexes (indexes slow down inserts, unnecessary
indexes slow everything down), table types (for example MyISAM vs HEAP),
etc. can improve performance.  You can also suspend logging for MySQL if
it's currently active.  Obviously, you also can upgrade to 512 MB of RAM.

> Unfortunately, this customer's application uses PostgreSQL, so I
> upgraded the Postgres to 7.1.2 a while back.  I set the connections to
> 45, but I use all persistent connections.

There are plenty of run-time config options for PostgreSQL as well.  And all
of the same types of database and query design issues apply.  I suggest
turning to a MySQL or PostgreSQL mailing list, manual or other online help
for these areas.  They're covered all the time on lists for both.

> I do not use ASP on this server, therefore I've turned it off.  However,
> it still runs in the background.  Whenever I turn it off, the cobalt
> cron job turns it back on within 45 minutes.  ps aux shows:
> /home/chiliasp/asp-apache-3000/caspd
> running a number of times.  I'd like to make this stop running.

I don't recall offhand where you need to make a modification, but I'm pretty
sure this has been covered on-list before (at least on cobalt-users).

> I have set the max clients in the httpd.conf file to 50.  I found that
> the typical httpd child process uses about 12-14 MB of Ram, of which 9.5
> is shared.  Therefore taking the remaining 4.5 MB of RAM, multiplying
> that by 50 children process yields an alarming 225 MB of RAM usage.
> (can you believe the server shipped with a default of 300?)

Perhaps you can recompile a streamlined Apache that has the bare minimum
compiled in for this one-time event.  There are probably plenty of Apache
modules and PHP modules (assuming the site runs PHP) that aren't needed.
And compiling PHP statically into Apache will get you probably a 5% boost in
speed over a DSO version.  You can always re-install the existing Apache
binary and related files after the one-time event.  There are other things
you can consider doing temporarily.  For example, turning off sendmail or
reducing the # of processes it can use, preventing cron jobs from running
during the event and doing whatever you can to limit load on the box for
other sites and services.

> Can anyone make a suggestion on what I can do to help the server hold up
> to this load?  Any advise is appreciated.

There are plenty more recommendations that can be made, but I'm reluctant to
spend more time suggesting them unless you are confident that they are
needed.

--
Steve Werby
President, Befriend Internet Services LLC
http://www.befriend.com/