[cobalt-users] Raq 3 Disk space disappearing in /

[Render-Vue <sales@xxxxxxxxxxxxxx>]

Hi Yah Everyone,

Unusual one tonight...

Email received from GUI:-

Subject: The filesystem [/] is getting very full
Content: is getting very close to full. This is very dangerous for the server
and can cause unexpected errors to occur. You either need to move some
files to another storage device and delete them from the Cobalt server
or delete them altogether. Consult the documentation for help adding
storage to your Cobalt server.
Total disk space: 726.04 MB
Free disk space: 12.98 MB
Percent Used: 98 %

Now I jumped into the server and everything looked okay as it should or as 
it was after I checked this morning.
Then checked the logcheck reports prior and after warning and nothing 
obvious sticking out , though Fcheck keeps showing this...

Apr 14 05:01:38 ns fcheck: "WARNING: [ns3.xxxxxxxxxxxxxxxxxxxx] /usr/tmp 
[Times: Apr 14 04:01 2002 - Apr 14 05:01 2002]"

Went to the usual log files to see if they had suddenly grown in size - 
nope all okay, then jumped into the snort logs to see what they had, it was 
at 11Mb so I knew something was wrong right away, the snort logs are 
usually only about 1-2Mb. Processed the snort binary to send an email 
report and got this error...

Message exceeds maximum fixed size (10485760)
/root/dead.letter... Saved message in /root/dead.letter
Shutting down snort: snort ERROR!ok Starting snort:

The program still sent through an email of what it could process and I got 
this...

The number of attacks from same host to same destination using same method
=========================================================================
# of attacks from to method
=========================================================================
2049 132.170.145.96 208.155.67.191 MISC Large UDP Packet
1984 129.120.245.153 208.155.67.191 MISC Large UDP Packet
1969 170.140.75.144 208.155.67.191 MISC Large UDP Packet
1944 132.170.94.57 208.155.67.191 MISC Large UDP Packet
1897 129.49.166.154 208.155.67.191 MISC Large UDP Packet
1865 129.108.5.245 208.155.67.191 MISC Large UDP Packet
1863 66.28.204.253 208.155.67.191 MISC Large UDP Packet
1801 134.48.100.250 208.155.67.191 MISC Large UDP Packet
1733 152.16.224.194 208.155.67.191 MISC Large UDP Packet
1710 137.189.38.89 208.155.67.191 MISC Large UDP Packet
1684 131.215.64.166 208.155.67.191 MISC Large UDP Packet
1679 129.108.32.89 208.155.67.191 MISC Large UDP Packet
1660 136.142.136.217 208.155.67.191 MISC Large UDP Packet
1654 131.156.24.106 208.155.67.191 MISC Large UDP Packet
1597 216.132.19.164 208.155.67.191 MISC Large UDP Packet
1562 157.182.196.11 208.155.67.191 MISC Large UDP Packet
1505 128.12.101.148 208.155.67.191 MISC Large UDP Packet
1479 129.7.70.26 208.155.67.191 MISC Large UDP Packet
1465 128.173.76.39 208.155.67.191 MISC Large UDP Packet
1450 128.125.99.14 208.155.67.191 MISC Large UDP Packet
1393 170.140.120.85 208.155.67.191 MISC Large UDP Packet
1314 130.91.233.186 208.155.67.191 MISC Large UDP Packet
1273 209.10.139.19 208.155.67.191 MISC Large UDP Packet
798 129.21.138.64 208.155.67.191 MISC Large UDP Packet
655 170.140.120.206 208.155.67.191 MISC Large UDP Packet
626 169.237.58.71 208.155.67.191 MISC Large UDP Packet
422 195.77.201.100 208.155.69.181 MISC Large ICMP Packet
210 169.237.22.54 208.155.67.191 MISC Large UDP Packet
210 206.155.192.251 208.155.69.181 MISC Large ICMP Packet
97 164.58.139.70 208.155.67.191 MISC Large UDP Packet
84 163.18.186.1 208.155.69.181 MISC Large ICMP Packet

Percentage and number of attacks to one certain host
==============================================
# of % attacks to method
==============================================
97.28 40924 208.155.67.191 MISC Large UDP Packet

Percentage and number of attacks from a host to a destination
============================================================
# of % attacks from to
============================================================
4.87 2049 132.170.145.96 208.155.67.191
4.72 1984 129.120.245.153 208.155.67.191
4.68 1969 170.140.75.144 208.155.67.191
4.62 1944 132.170.94.57 208.155.67.191
4.51 1897 129.49.166.154 208.155.67.191
4.43 1865 129.108.5.245 208.155.67.191
4.43 1863 66.28.204.253 208.155.67.191
4.28 1801 134.48.100.250 208.155.67.191
4.12 1733 152.16.224.194 208.155.67.191
4.06 1710 137.189.38.89 208.155.67.191
4.00 1684 131.215.64.166 208.155.67.191
3.99 1679 129.108.32.89 208.155.67.191
3.95 1660 136.142.136.217 208.155.67.191
3.93 1654 131.156.24.106 208.155.67.191
3.80 1597 216.132.19.164 208.155.67.191
3.71 1562 157.182.196.11 208.155.67.191
3.58 1505 128.12.101.148 208.155.67.191
3.52 1479 129.7.70.26 208.155.67.191
3.48 1465 128.173.76.39 208.155.67.191
3.45 1450 128.125.99.14 208.155.67.191
3.31 1393 170.140.120.85 208.155.67.191
3.12 1314 130.91.233.186 208.155.67.191
3.03 1273 209.10.139.19 208.155.67.191
1.90 798 129.21.138.64 208.155.67.191
1.56 655 170.140.120.206 208.155.67.191
1.49 626 169.237.58.71 208.155.67.191
1.00 422 195.77.201.100 208.155.69.181
0.50 210 206.155.192.251 208.155.69.181
0.50 210 169.237.22.54 208.155.67.191
0.23 97 164.58.139.70 208.155.67.191

208.155.67.191 is not one of our IP's but one on the Colo's network. I've 
contacted them so we'll see what they come back with (Snorts in sniffer 
mode), but anyway this is not the reason I'm contacting the usergroup. I 
usually have about 350-360Mb free on the Operating/File system ("/" and 
other folders) but the servers stating I've only 150Mb free now, I've been 
through every folder on the server looking for the lost 150Mb, there are no 
"dead letter" files and nothing exceptionally big in files size or out of 
the ordinary that I can see. I've even checked Portsentry logs, Host.Deny 
and flushed IP Chains but the 150Mb is still missing.

Anyone have any clue as to where to start looking or where it could have 
disappeared too?

Also this reoccuring log report that keeps showing up no since the original 
disk space short error...

Apr 14 05:01:38 ns fcheck: "WARNING: [ns3.xxxxxxxxxxxxxxxxx] /usr/tmp 
[Times: Apr 14 04:01 2002 - Apr 14 05:01 2002]"

usr/tmp is a sym link to home/tmp and in their I can see...

.s.PGSQL.5583 0b 15th March <--- does this stay
libphp4.so 5mb  2nd March <----- can this go php was updated to 4.1.2
mysql.sock 22b  15 March  <----- This stays I know that much
resend.debug 0b with todays date ???????

Had a quick search for resend.debug and it seems to be related to Majordomo 
- maybe this has nothing to do with anything above as I noticed one and one 
only Cobalt post mentioning resend.debug and someone else had it in their 
tmp folder.

I'll check the GUI once logrotate does its thing and see if the 150Mb's 
comes back.

Regards

Chae