[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[cobalt-users] Help! Raq 3 hangs regularly (SYN flood involved?)
- Subject: [cobalt-users] Help! Raq 3 hangs regularly (SYN flood involved?)
- From: Cobalter <nospam@xxxxxxxxxxxx>
- Date: Fri Feb 1 20:12:14 2002
- List-id: Mailing list for users to share thoughts on Cobalt products. <cobalt-users.list.cobalt.com>
One of our RaQ 3's has developed a "choking" habit that seems to lock it up
within a 1.5-hour window just about every day (actually, it skipped
yesterday, and I thought I had it knocked). It's back today, though.
Curiously, the logs show a possible SYN Flood that occurs every time just
before the machine goes quiet. A reboot brings everything back to (seeming)
normal until the next SYN flood roughly 24 (or 48) hours later. Here you'll
see the flood warning start at 12:13:56, then go on until 12:16:25, when
suddenly the logs get very quiet. At 12:25, the reboot begins (this is done
by hand at the NOC -- the machine seems to want to stay down for the count
until we do this), and everything comes back to life:
Feb 1 12:11:20 www named[553]: Cleaned cache of 285 RRsets
Feb 1 12:11:20 www named[553]: USAGE 1012583480 1012446682
CPU=46.93u/18.86s CHILDCPU=0u/0s
Feb 1 12:11:20 www named[553]: NSTATS 1012583480 1012446682 A=13825
CNAME=54 PTR=45356 MX=962 AAAA=56 38=1 ANY=5149
Feb 1 12:11:20 www named[553]: XSTATS 1012583480 1012446682 RR=39627
RNXD=6908 RFwdR=27154 RDupR=49 RFail=490 RFErr=0 RErr=38 RAXFR=0 RLame=2623
ROpts=0 SSysQ=7265 SAns=60506 SFwdQ=23674 SDupQ=8350 SErr=0 RQ=65556 RIQ=0
RFwdQ=23674 RDupQ=42 RTCP=11 SFwdR=27154 SFail=4 SFErr=0 SNaAns=38238
SNXD=8771 RUQ=0 RURQ=0 RUXFR=0 RUUpd=0
Feb 1 12:13:56 www kernel: possible SYN flooding on port 80. Sending cookies.
Feb 1 12:15:01 www proftpd[9553]: www.xxxxx.xxx (localhost[127.0.0.1]) -
FTP session opened.
Feb 1 12:15:01 www proftpd[9553]: www.xxxxx.xxx (localhost[127.0.0.1]) -
no such user 'anonymous'
Feb 1 12:15:01 www proftpd[9553]: www.xxxxx.xxx (localhost[127.0.0.1]) -
no such user 'anonymous'
Feb 1 12:15:01 www proftpd[9553]: www.xxxxx.xxx (localhost[127.0.0.1]) -
FTP session closed.
Feb 1 12:15:23 www kernel: possible SYN flooding on port 80. Sending cookies.
Feb 1 12:16:25 www kernel: possible SYN flooding on port 80. Sending cookies.
Feb 1 12:25:55 www init: Switching to runlevel: 6
Feb 1 12:25:55 www getty[772]: exiting on TERM signal
Feb 1 12:26:00 www sshd[558]: Received signal 15; terminating.
Feb 1 12:26:01 www named[553]: named shutting down
Feb 1 12:26:01 www named[553]: USAGE 1012584361 1012446682
CPU=47.16u/19.05s CHILDCPU=0u/0s
Feb 1 12:26:01 www named[553]: NSTATS 1012584361 1012446682 A=14603
CNAME=54 PTR=45359 MX=1095 AAAA=59 38=5 ANY=5151
Feb 1 12:26:01 www named[553]: XSTATS 1012584361 1012446682 RR=39627
RNXD=6908 RFwdR=27154 RDupR=49 RFail=490 RFErr=0 RErr=38 RAXFR=0 RLame=2623
ROpts=0 SSysQ=7265 SAns=61458 SFwdQ=23676 SDupQ=8360 SErr=0 RQ=66512 RIQ=0
RFwdQ=23676 RDupQ=44 RTCP=11 SFwdR=27154 SFail=4 SFErr=0 SNaAns=38301
SNXD=8913 RUQ=0 RURQ=0 RUXFR=0 RUUpd=0
Feb 1 12:26:06 www kernel: Kernel logging (proc) stopped.
Feb 1 12:26:06 www kernel: Kernel log daemon terminating.
Feb 1 12:26:08 www exiting on signal 15
Feb 1 12:27:48 www syslogd 1.3-3: restart.
Feb 1 12:27:48 www modprobe: can't locate module bandwidth_mgr
Feb 1 12:27:50 www kernel: klogd 1.3-3, log source = /proc/kmsg started.
Feb 1 12:27:50 www kernel: Inspecting /boot/System.map
Feb 1 12:27:50 www kernel: Loaded 7617 symbols from /boot/System.map.
Feb 1 12:27:50 www kernel: Symbols match kernel version 2.2.16.
Feb 1 12:27:50 www kernel: Loaded 7 symbols from 1 module.
Feb 1 12:27:50 www kernel: Linux version 2.2.16C32_III
(root@xxxxxxxxxxxxxx) (gcc version egcs-2.91.66 19990314/Linux (egcs-1.1.2
release)) #1 Fri Nov 9 21:54:54 PST 2001
Feb 1 12:27:50 www kernel: Ignoring bogus EBDA pointer 3FDF000
Feb 1 12:27:50 www kernel: Detected 298812 kHz processor.
Feb 1 12:27:50 www kernel: Pending 0x10
Feb 1 12:27:50 www kernel: Calibrating delay loop... 596.38 BogoMIPS
Feb 1 12:27:50 www kernel: Memory: 127720k/131072k available (1244k
kernel code, 416k reserved, 1628k data, 64k init)
Feb 1 12:27:50 www kernel: Dentry hash table entries: 16384 (order 5, 128k)
Feb 1 12:27:50 www kernel: Buffer cache hash table entries: 131072 (order
7, 512k)
Feb 1 12:27:50 www kernel: Page cache hash table entries: 32768 (order 5,
128k)
Feb 1 12:27:50 www kernel: VFS: Diskquotas version dquot_6.4.0 initialized
Feb 1 12:27:50 www kernel: CPU: L1 I Cache: 32K L1 D Cache: 32K
Feb 1 12:27:50 www kernel: CPU: AMD AMD-K6(tm) 3D processor stepping 0c
Feb 1 12:27:50 www kernel: Checking 386/387 coupling... OK, FPU using
exception 16 error reporting.
Feb 1 12:27:50 www kernel: Checking 'hlt' instruction... OK.
Feb 1 12:27:50 www kernel: POSIX conformance testing by UNIFIX
Feb 1 12:27:50 www kernel: mtrr: v1.35a (19990819) Richard Gooch
(rgooch@xxxxxxxxxxxxx)
Feb 1 12:27:50 www kernel: PCI: Using configuration type 1
Feb 1 12:27:50 www kernel: PCI: Probing PCI hardware
Feb 1 12:27:50 www kernel: Linux NET4.0 for Linux 2.2
Feb 1 12:27:50 www kernel: Based upon Swansea University Computer Society
NET3.039
Feb 1 12:27:50 www kernel: NET4: Unix domain sockets 1.0 for Linux NET4.0.
Feb 1 12:27:50 www kernel: NET4: Linux TCP/IP 1.0 for NET4.0
Feb 1 12:27:50 www kernel: IP Protocols: ICMP, UDP, TCP, IGMP
Feb 1 12:27:50 www kernel: TCP: Hash tables configured (ehash 131072
bhash 65536)
Feb 1 12:27:50 www kernel: Initializing RT netlink socket
Feb 1 12:27:50 www kernel: Starting kswapd v 1.5
Feb 1 12:27:50 www kernel: Cobalt watchdog v1.4 enabled
Feb 1 12:27:50 www kernel: Cobalt I2C bus initialized
Feb 1 12:27:50 www kernel: Cobalt temperature sensor v1.3 enabled
Feb 1 12:27:50 www kernel: Serial driver version 4.27 with<4>keyboard:
Timeout - AT keyboard not present?
Feb 1 12:27:50 www kernel: keyboard: Timeout - AT keyboard not present?
Feb 1 12:27:50 www kernel: no serial options enabled
Feb 1 12:27:50 www kernel: ttyS00 at 0x03f8 (irq = 4) is a 16550A
Feb 1 12:27:50 www kernel: ttyS01 at 0x02f8 (irq = 3) is a 16550A
Feb 1 12:27:50 www kernel: pty: 256 Unix98 ptys configured
Feb 1 12:27:50 www kernel: Real Time Clock Driver v1.09
Feb 1 12:27:50 www kernel: lcd: Cobalt LCD Driver v3.12^M
Feb 1 12:27:50 www kernel: serialnumber: Version 1.9 initialized. Serial
number=630000074bec5701.
Feb 1 12:27:50 www kernel: Copyright (c)1994-2000 Axent Technologies, Inc.
Feb 1 12:27:50 www kernel: Uniform Multi-Platform E-IDE driver Revision: 6.30
Feb 1 12:27:50 www kernel: ide: Assuming 33MHz system bus speed for PIO
modes; override with idebus=xx
Feb 1 12:27:50 www kernel: ALI15X3: IDE controller on PCI bus 00 dev 78
Feb 1 12:27:50 www kernel: ALI15X3: chipset revision 193
Feb 1 12:27:50 www kernel: ALI15X3: 100%% native mode on irq 14
Feb 1 12:27:50 www kernel: ide0: BM-DMA at 0xf000-0xf007, BIOS
settings: hda:DMA, hdb:DMA
Feb 1 12:27:50 www kernel: ide1: BM-DMA at 0xf008-0xf00f, BIOS
settings: hdc:DMA, hdd:DMA
Feb 1 12:27:50 www kernel: hda: ST315323A, SN=7EH0BJ0T, FWREV=3.02, ATA
DISK drive
Feb 1 12:27:50 www kernel: ide: Assuming 33MHz system bus speed for PIO
modes; override with idebus=xx
Feb 1 12:27:50 www kernel: ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Feb 1 12:27:50 www kernel: hda: ST315323A, 14652MB w/512kB Cache,
CHS=29770/16/63, UDMA(33)
Feb 1 12:27:50 www kernel: md driver 0.90.0 MAX_MD_DEVS=256, MAX_REAL=12
Feb 1 12:27:50 www kernel: translucent personality registered
Feb 1 12:27:50 www kernel: linear personality registered
Feb 1 12:27:50 www kernel: raid0 personality registered
Feb 1 12:27:50 www kernel: raid1 personality registered
Feb 1 12:27:50 www kernel: raid5 personality registered
Feb 1 12:27:50 www kernel: raid5: measuring checksumming speed
Feb 1 12:27:50 www kernel: raid5: MMX detected, trying high-speed MMX
checksum routines
Feb 1 12:27:50 www kernel: pII_mmx : 758.190 MB/sec
Feb 1 12:27:50 www kernel: p5_mmx : 700.659 MB/sec
Feb 1 12:27:50 www kernel: 8regs : 321.183 MB/sec
Feb 1 12:27:50 www kernel: 32regs : 199.263 MB/sec
Feb 1 12:27:50 www kernel: using fastest function: pII_mmx (758.190 MB/sec)
Feb 1 12:27:50 www kernel: sym53c8xx: at PCI bus 0, device 14, function 0
Feb 1 12:27:50 www kernel: sym53c8xx: 53c875 detected
Feb 1 12:27:50 www kernel: sym53c875-0: rev 0x4 on pci bus 0 device 14
function 0 irq 12
Feb 1 12:27:50 www kernel: sym53c875-0: ID 7, Fast-20, Parity Checking
Feb 1 12:27:50 www kernel: scsi0 : sym53c8xx-1.7.3a-20010304
Feb 1 12:27:50 www kernel: scsi : 1 host.
Feb 1 12:27:50 www kernel: scsi : detected total.
Feb 1 12:27:50 www kernel: md.c: sizeof(mdp_super_t) = 4096
Feb 1 12:27:50 www kernel: Partition check:
Feb 1 12:27:50 www kernel: hda: hda1 hda2 hda3 hda4
Feb 1 12:27:50 www kernel: autodetecting RAID arrays
Feb 1 12:27:50 www kernel: autorun ...
Feb 1 12:27:50 www kernel: ... autorun DONE.
Feb 1 12:27:50 www kernel: VFS: Mounted root (ext2 filesystem) readonly.
Feb 1 12:27:50 www kernel: Freeing unused kernel memory: 64k freed
Feb 1 12:27:50 www kernel: Adding Swap: 131536k swap-space (priority -1)
Feb 1 12:27:50 www kernel: eth0: Invalid EEPROM checksum 0xaa99, check
settings before activating this device!
Feb 1 12:27:50 www kernel: eth0: Intel PCI EtherExpress Pro100 82559ER,
00:10:E0:01:B9:98, I/O at 0x1100, IRQ 11.
Feb 1 12:27:50 www kernel: Board assembly 000000-000, Physical
connectors present:
Feb 1 12:27:50 www kernel: Primary interface chip None PHY #0.
Feb 1 12:27:50 www kernel: General self-test: passed.
Feb 1 12:27:50 www kernel: Serial sub-system self-test: passed.
Feb 1 12:27:50 www kernel: Internal registers self-test: passed.
Feb 1 12:27:50 www kernel: ROM checksum self-test: passed (0xdbd8681d).
Feb 1 12:27:50 www kernel: Receiver lock-up workaround activated.
Feb 1 12:27:50 www kernel: eth1: Invalid EEPROM checksum 0xa999, check
settings before activating this device!
Feb 1 12:27:50 www kernel: eth1: Intel PCI EtherExpress Pro100 82559ER,
00:10:E0:01:B9:97, I/O at 0x1200, IRQ 10.
Feb 1 12:27:50 www kernel: Board assembly 000000-000, Physical
connectors present:
Feb 1 12:27:50 www kernel: Primary interface chip None PHY #0.
Feb 1 12:27:50 www kernel: General self-test: passed.
Feb 1 12:27:50 www kernel: Serial sub-system self-test: passed.
Feb 1 12:27:50 www kernel: Internal registers self-test: passed.
Feb 1 12:27:50 www kernel: ROM checksum self-test: passed (0xdbd8681d).
Feb 1 12:27:50 www kernel: Receiver lock-up workaround activated.
Feb 1 12:27:58 www kernel: portmap: RPC call returned error 111
Feb 1 12:27:58 www kernel: RPC: task of released request still queued!
Feb 1 12:27:58 www kernel: RPC: (task is on xprt_pending)
Feb 1 12:28:03 www kernel: portmap: RPC call returned error 111
Feb 1 12:28:03 www kernel: RPC: task of released request still queued!
Feb 1 12:28:03 www kernel: RPC: (task is on xprt_pending)
Feb 1 12:28:03 www kernel: lockd_up: makesock failed, error=-111
Feb 1 12:28:08 www kernel: portmap: RPC call returned error 111
Feb 1 12:28:08 www kernel: RPC: task of released request still queued!
Feb 1 12:28:08 www kernel: RPC: (task is on xprt_pending)
Feb 1 12:28:09 www rpc.statd[452]: unable to register (SM_PROG, SM_VERS,
udp).
Feb 1 12:28:10 www modprobe: can't locate module block-major-22
Feb 1 12:28:10 www modprobe: can't locate module block-major-22
Feb 1 12:28:10 www modprobe: can't locate module block-major-33
Feb 1 12:28:10 www modprobe: can't locate module block-major-33
Feb 1 12:28:10 www modprobe: can't locate module block-major-34
Feb 1 12:28:10 www modprobe: can't locate module block-major-34
Feb 1 12:28:10 www modprobe: can't locate module block-major-8
Feb 1 12:28:10 www last message repeated 4 times
Feb 1 12:28:10 www modprobe: can't locate module block-major-13
Feb 1 12:28:10 www modprobe: can't locate module block-major-13
Feb 1 12:28:11 www named[552]: starting (/etc/named.conf). named
8.2.3-REL Tue Jan 30 16:56:25 PST 2001
^Iadmin@xxxxxxxxxxxxxxxxxx:/home/redhat/BUILD/bind-8.2.3/src/bin/named
Feb 1 12:28:11 www named[552]: hint zone "" (IN) loaded (serial 0)
Feb 1 12:28:13 www named[552]: listening on [127.0.0.1].53 (lo)
Feb 1 12:28:13 www named[552]: listening on [nnn.nnn.nnn.1].53 (eth0)
Feb 1 12:28:13 www named[552]: listening on [nnn.nnn.nnn.2].53 (eth0:0)
Feb 1 12:28:13 www named[552]: listening on [nnn.nnn.nnn.3].53 (eth0:1)
Feb 1 12:28:13 www named[552]: listening on [nnn.nnn.nnn.4].53 (eth0:2)
Feb 1 12:28:13 www named[552]: listening on [nnn.nnn.nnn.5].53 (eth0:3)
Feb 1 12:28:13 www named[552]: listening on [nnn.nnn.nnn.6].53 (eth0:4)
Feb 1 12:28:13 www named[552]: Forwarding source address is [0.0.0.0].zzzz
Feb 1 12:28:13 www named[553]: group = named
Feb 1 12:28:13 www named[553]: user = named
Feb 1 12:28:13 www named[553]: Ready to answer queries.
Feb 1 12:28:14 www sshd[558]: Server listening on 0.0.0.0 port 22.
Ok, now for a bit of background that will probably help: The thinkable
happened last week, and this box was entered through (slapping myself on
the forehead now) a known SSH 2.xx exploit (Somehow, that machine never got
the upgrade to the newer version...) All other recommended Cobalt system
upgrades HAVE been preformed, however, and the box is now running the
latest OpenSSH.
I KNOW I need to scrape the box and start over, but because of
geography/conditions/cashflow, etc., it's going to be a couple weeks before
I can. Meanwhile, I've been pouring over logs, etc., comparing files,
replacing anything and everything I can, running lsof, nmap, portscans,
etc... So far I've removed linSniffer and a few other uglies, closed a few
ports and removed several nasties from the startup files. And am I just
paranoid, or has mgetty always been running as a process?? I wasn't here
when this box was set up, so I'm not sure if that's default or not?
And this SYN flood -- is it real? Is it the culprit? What the #@$% can I do
about it????
Any help would be greatly appreciated! Thanks!
Desperately seeking sleep,
Vito