[cobalt-users] GUI not working (postgres won't start!?)

[cobalt@xxxxxxxxxxxxxxxx]

Hello All,
I had to reboot one of our raq3 servers (according to knowledgebase) which
I later discovered I didn't even have to do, when I was installing a secondary
singed CA for ssl/https. So I rebooted the server and when it came back up,
the GUI was not working. After some digging I found out that postgres was not
started and would not start from the init.d/postgrest start / restart scripts
and would not start manually via the postgres user either.

I have some clues, though I don't know what they mean. Starting with /var/log/postgres
this is what I see when I try to start the service:

root@web-raq05:/var/log# cat postgresql
011116.06:45:01.132 [12575] DEBUG:  Data Base System is starting up at Fri Nov 16 06:45:01 2001
011116.06:45:01.148 [12575] FATAL 2:  Control file context is broken
011116.06:45:01.148 [12575] FATAL 2:  Control file context is broken
Startup failed - abort

When I try to start it by user postgres. (I've tried the -D switch with /var/lib/pgsql and many others and it
seems like it can't find my data directory)

I'm pretty baffled at this point:

No data directory -- can't proceed.
[postgres pgsql]$ /usr/bin/postmaster -S -D /usr/lib/pgsql
/usr/bin/postmaster does not find the database system.  Expected to find it in the PGDATA directory "/usr/lib/pgsql", but unable to open file with pathname "/usr/lib/pgsql/base/template1/pg_class".

[postgres pgsql]$ locate pg_class

/var/lib/pgsql/data/base/cobalt/pg_class
/home/chiliasp/odbc/internaldb/data/base/template1/pg_class_relname_index
/usr/include/pgsql/catalog/pg_class.h
[postgres pgsql]$ /usr/bin/postmaster -S -D /var/lib/pgsql/data/base/cobalt/
/usr/bin/postmaster does not find the database system.  Expected to find it in the PGDATA directory "/var/lib/pgsql/data/base/cobalt/", but unable to open file with pathname "/var/lib/pgsql/data/base/cobalt//base/template1/pg_class".

No data directory -- can't proceed.
[postgres pgsql]$ /usr/bin/postmaster -S -D /var/lib/pgsql/data/base/cobalt
/usr/bin/postmaster does not find the database system.  Expected to find it in the PGDATA directory "/var/lib/pgsql/data/base/cobalt", but unable to open file with pathname "/var/lib/pgsql/data/base/cobalt/base/template1/pg_class".

No data directory -- can't proceed.
[postgres pgsql]$ file /var/lib/pgsql/data/base/cobalt/base/template1/pg_class
/var/lib/pgsql/data/base/cobalt/base/template1/pg_class: can't stat `/var/lib/pgsql/data/base/cobalt/base/template1/pg_class' (No such file or directory).
[postgres pgsql]$ exit

I also checked the permissions of /tmp to make sure the .s.pos* could start and the /tmp dir is fine. Just like my other raqs that postgres is running fine on.

One more thing, when I try to start postgres by the user postgres it asks for a passowrd? :

root@web-raq05:/var/log# su - postgres
[postgres pgsql]$ /etc/rc.d/init.d/postgresql start
Setting up PostgreSQL: Password:
su: incorrect password
failed!

Any hints appriciated as I have about 150 sites on this thing that customer care will need GUI access to help when they call in.

Thanks a bunch!

Cobalt Man