[cobalt-users] case study

[Yohans Mendoza <yohans@xxxxxxxxxxxxxxxxxxxxxxxxxx>]

hi all, 

I'd like to share our experience from this past weekend, we sure
honored labor day. Please feel free to comment on things you think we
could've done better.

Sunday evening I started getting email from one of our raq's saying that
the monitor had died, and I needed to reboot the server, so I did, and
while it was restarting a I read yet another mail, saying the filesystem
was full. Well, I should've read first all my email before taking any
action. Needless to say, some files came up corrupted, among those,
httpd.conf, so all our sites were gone (web-wise I mean). My next step was
to clean the tmp directory, since cobalt doesn't rotate some logs in that
directory. Next I recovered an httpd.copnf file from my thursday backup.
Once I had the web reconfigured, I figured I'd restart the server, since I
was getting an email about the sites being out of sync.
Well, that was the killing bullet, the server came up alright, but with
the network configuration all messed up. The server was not reachable,
then I tried to reconfigure the network throught the front panel, but it
wouldn't take it. Then my second biggest mistake was to try to reset the
server through the reset button, but it only resets the admin  password,
and the root password gets all screwd up.

Well, that made matters worse, because when I tried accessing the server
through the serail port, I couldn't do I thing, since I had lost root's
password.

The Recovery
The decision was to migrate all the sites 50+ to another raq. We had to
create the sites manually, then put the backed up files in the correct
place. The users were created manually, because we didn't know if only
appending the passwd files would do it, and we were afraid of messing
another server.

We had the whole thing up and running in about 6 or 7 hours.
Now we're waiting for the recovery cd to recover the other server.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Yohans Mendoza					System Analyst
yohans@xxxxxxxxxxxxxxxxx			Sirius Images Inc.	
http://www.sirius-images.net/users/yohans    	http://www.sirius-images.com 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~