Re: [cobalt-users] Help! Raq 3 hangs regularly (SYN flood involved?)

[flash22@xxxxxxx]

On Fri, 1 Feb 2002, Cobalter wrote:
> 
> One of our RaQ 3's has developed a "choking" habit that seems to lock it up 
> within a 1.5-hour window just about every day (actually, it skipped 
> yesterday, and I thought I had it knocked). It's back today, though.
> 
> Curiously, the logs show a possible SYN Flood that occurs every time just 
> before the machine goes quiet. A reboot brings everything back to (seeming) 

I'm going to hazard a guess, the SYN flood is on port 80, i'm betting it's
really a connection flood, eg, they are actual connect attempts, and the
server is buckeling, and the watchdog timer is resetting it, why?
Because it's not crashing, it's stopping gracefully...


> 	Feb  1 12:15:23 www kernel: possible SYN flooding on port 80. Sending cookies.
> 	Feb  1 12:16:25 www kernel: possible SYN flooding on port 80. Sending cookies.
> 	Feb  1 12:25:55 www init: Switching to runlevel: 6
> 	Feb  1 12:25:55 www getty[772]: exiting on TERM signal
> 	Feb  1 12:26:00 www sshd[558]: Received signal 15; terminating.
> 	Feb  1 12:26:01 www named[553]: named shutting down

Those are requested shutdown's via init, which was told to shut them down,
so either you have a script in cron doing it, or a crashing trojan you
missed, of the watchdog is doing it...


> 	Feb  1 12:27:48 www syslogd 1.3-3: restart.
> 	Feb  1 12:27:48 www modprobe: can't locate module bandwidth_mgr

Figure out why you are missing the bandwidth managment module

> 	Feb  1 12:27:50 www kernel: Serial driver version 4.27 with<4>keyboard: 
> Timeout - AT keyboard not present?
> 	Feb  1 12:27:50 www kernel: keyboard: Timeout - AT keyboard not present?
lol

> 	Feb  1 12:27:50 www kernel: serialnumber: Version 1.9 initialized. Serial 

You just gave the world your serial number ;P

Nothing else looks particularly odd

> I KNOW I need to scrape the box and start over, but because of 
> geography/conditions/cashflow, etc., it's going to be a couple weeks before 

You need to suspect either trojan cgi's , or a stash of warez somewhere
that a bazillion people are trying to download, or the possability someone
is trying a bruteforce password attack on a site, you need to be looking
at http logs at the time when the SYN floods happen, reduce the number if
processes apache can spawn may keep the machine up , if that's the case

> I can. Meanwhile, I've been pouring over logs, etc., comparing files, 
> replacing anything and everything I can, running lsof, nmap, portscans, 
> etc... So far I've removed linSniffer and a few other uglies, closed a few 
> ports and removed several nasties from the startup files. And am I just 
> paranoid, or has mgetty always been running as a process?? I wasn't here 

I think mgetty is on that machine,not sure , it' spart of earlier redhat
by default, assuming it's really mgetty, it will only listen to the serial
port unless someone is being really clever, but do you trust anything? ;P

Did you check for kernel modules hiding in the modules directory?
> 
> And this SYN flood -- is it real? Is it the culprit? What the #@$% can I do 
> about it????

Yes, it's real, but SYN is part of connection handshaking, you get a SYN
flood from either a lot of real onnections, of a machine pretending to
make a lot of connections (eg a port scanner)

You know, isp's buy $$$ routers to rpevent that sort of thing ;P

tcpdump on the interface logging to a file will probably tell you what you
are getting at least, tho it will make a REALLY BIG file ;P

ps: I'm not kidding about it making a REALLY BIG file ;P

Happy hunting ;P

gsh