[cobalt-users] RaQ4 with latest kernel just installed reports /home partition full when not

["Steve Werby" <steve-lists@xxxxxxxxxxxx>]

Earlier today I was preparing to install a newer version of Python and the
latest version of Mailman on a client's RaQ4 (RaQ4 hardware and RaQ4
software).  Part way through unpacking the source tarball for Python onto
the /home partition, tar starting issuing error messages like the following:

tar: Python-2.2.2/Tools/unicode: Cannot mkdir: No such file or directory
Python-2.2.2/Tools/unicode/makeunicodedata.py

So I checked partition usage.

Filesystem            Size  Used Avail Use% Mounted on
/dev/hda1             726M  617M   71M  90% /
/dev/hda3             194M   19M  165M  10% /var
/dev/hda4             8.2G  2.6G  5.1G  34% /home

Not only did the /home partition have plenty of space, but the other
partitions did as well.  I tried creating a directory and a file within the
/home partition to see if that was possible (both failed).

mkdir Python-2.2.2/Tools
mkdir: cannot create directory `Python-2.2.2/Tools': No space left on device

I also tried unpacking the tarball to /root (which is on the / partition)
and it worked without incident.  It's definitely not an issue with the
tarball as I tried another tarball, mkdir and touch with the same results.

I did some research and didn't find anything conclusive so I removed some
unnecessary files from the partition in case it truely was full and df was
incorrect (I removed ~500 MB of files).  I received the same error.  I
checked the Apache logs to see if they were being written to and they were.
The client tried creating a user through the GUI and it wouldn't allow it,
reporting that the directory couldn't be created.

Next the client rebooted and that didn't help.  I proceeded to disable all
services accessing the /home partition, unmount /home using umount and run
fsck in interactive repair mode.  It outputted:

Parallelizing fsck version 1.18 (11-Nov-1999)
[/sbin/fsck.ext2 -- /home] fsck.ext2 -r /dev/hda4
e2fsck 1.18, 11-Nov-1999 for EXT2 FS 0.5b, 95/08/09
/dev/hda4: clean, 67448/67456 files, 2752127/8632512 blocks

I tried restarting all services, but Apache wouldn't restart (the admin
Apache server would though).

# From /var/log/httpd/error
Thu Jan 16 14:07:41 2003] [error] mod_ssl: Parent process could not create
SSLMutex lockfile /var/log/httpd/ssl_mutex.3142 (System error follows)
[Thu Jan 16 14:07:41 2003] [error] System: No space left on device (errno:
28)

I then killed the processes, unmounted the partition and ran fsck again and
it outputted:

Parallelizing fsck version 1.18 (11-Nov-1999)
[/sbin/fsck.ext2 -- /home] fsck.ext2 -r /dev/hda4
e2fsck 1.18, 11-Nov-1999 for EXT2 FS 0.5b, 95/08/09
/dev/hda4: clean, 67456/67456 files, 2752145/8632512 blocks

I checked /var/log/kernel and didn't see anything that jumped out at me.
Right now the box is rebooting with -F to force fsck.  I did some other
diagnostics, but nothing that gave me any useful info.  The box has not  had
any RAM or disk installed recently.  I had noticed that it was rebooted 26
hours ago and asked the client about that and he said that was because he
had installed the latest Kernel package.

Has anyone encountered this?  Any chance it's related to the new Kernel?  If
so, can I revert to the previous Kernel or does anyone have other advice?

--
Steve Werby
President, Befriend Internet Services LLC
http://www.befriend.com/