[cobalt-users] bizarre RaQ4 problem

["E.B. Dreger" <eddy+public+spam@xxxxxxxxxxxxxxxxx>]

Greetings all,


I'm wrangling with a RaQ4.  After a seemingly-random amount of
uptime (anywhere from a few minutes to several hours), anything
utilizing MySQL ceases to work:

* MySQL apparently isn't running; see below.

* The MySQL unix domain socket is gone.

* The MySQL error log thinks mysqld _is_ running ("Number of
  processes running now: <some number>").

* "netstat -anpt" shows tcp 0.0.0.0:3306 in state LISTEN, but
  with a hyphen instead of the pid/cmdline.

* Observed from the client end, 3306/TCP server connection
  attempts appear successful... then hang.  IMHO, this supports
  the theory that nothing is listening to the socket.

The md5 checksums for ps and netstat appear clean.  None of the
entries in /proc show a commandline of mysqld.  chkrootkit finds
naught.  IOW, I'm hesitant to suspect a cracked box; the
possibility can't be discarded, but I doubt any rootkits are
present on the box.

I've increased fs.file-max and fs.inode-max sysctl values, but
with no apparent effect.  The box is not running out of memory,
and never eats into swap.

As a final tidbit, inetd seems to suffer the same "ghost socket"
fate as mysqld _after_ mysqld has been dead (between worlds?) for
some amount of time.  Oddly, inetd continues to show up in "ps"
output.

Anyone ever encountered anything similar?


Eddy
--
Brotsman & Dreger, Inc. - EverQuick Internet Division
Bandwidth, consulting, e-commerce, hosting, and network building
Phone: +1 (785) 865-5885 Lawrence and [inter]national
Phone: +1 (316) 794-8922 Wichita

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Date: Mon, 21 May 2001 11:23:58 +0000 (GMT)
From: A Trap <blacklist@xxxxxxxxx>
To: blacklist@xxxxxxxxx
Subject: Please ignore this portion of my mail signature.

These last few lines are a trap for address-harvesting spambots.
Do NOT send mail to <blacklist@xxxxxxxxx>, or you are likely to
be blocked.