[cobalt-users] [RaQ3] SSH Not Responding - Postmortem

[Bradley Caricofe <bcaricofe@xxxxxxxxx>]

Howdy list, for the benefit of anyone who didn't read any of my earlier
posts, I have a RaQ3, well cared for with all patches, minimal ports and
custom software running, that went down early on the morning of July 4th. 
While the server was down I found that it was pingable however with
practically all services failing (httpd, ahttpd, ssh).  It was rebooted
several times and now I'm able to once again gain shell access.

The server seems as stable as it's ever been and I've begun earnestly
digging through the logs looking for info.  I've found some interesting
entries in /var/log/messages and the admserv and httpd access and error
log files.  Searching through the archives I've found some similar
postings from people who were likely having the same problem as us.  In
most of those cases the truly enlightened gurus on this list seemed to
think crack attempts or DOS attacks were the cause.

I just thought I'd show you all some of my log entries and get your expert
opinions.

-- from the messages log --

Jul  3 21:59:52 www kernel: VM: do_try_to_free_pages failed for syslogd...
Jul  3 21:59:53 www kernel: VM: do_try_to_free_pages failed for
poprelayd...
Jul  3 21:59:53 www kernel: VM: do_try_to_free_pages failed for
interchange...
Jul  3 21:59:55 www kernel: VM: do_try_to_free_pages failed for httpd...

(I'm not sure if these are related to yesterday's issue, however they are
weird and I haven't seen em before.  Archives suggest a virtual mem
problem...)

Jul  4 04:12:19 www kernel: Unable to load interpreter /lib/ld-linux.so.2

(Yeh, there's a ton of these in there, memory again I know, but why..?)

-- from the adm_access log --
(This is where it gets interesting, is this an obvious attack on the
cobalt gui?  Maybe just someone who was trying to get in from various
computers?)

80.8.112.51 - - [04/Jul/2002:04:12:27 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
213.17.37.15 - - [04/Jul/2002:04:43:40 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
194.65.5.231 - - [04/Jul/2002:07:36:02 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
193.216.245.84 - - [04/Jul/2002:07:37:56 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
80.9.180.115 - - [04/Jul/2002:07:39:30 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
80.203.23.120 - - [04/Jul/2002:07:42:08 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
129.96.138.71 - - [04/Jul/2002:07:50:31 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
195.194.138.161 - - [04/Jul/2002:07:51:28 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
212.195.20.252 - - [04/Jul/2002:07:53:57 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.0" 200 1119
68.128.195.22 - - [04/Jul/2002:07:54:43 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
193.217.241.109 - - [04/Jul/2002:08:09:27 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
213.139.162.197 - - [04/Jul/2002:08:16:05 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
213.139.162.197 - - [04/Jul/2002:08:16:31 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
213.84.26.175 - - [04/Jul/2002:08:21:54 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
132.180.77.189 - - [04/Jul/2002:08:25:39 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
151.37.82.251 - - [04/Jul/2002:08:29:03 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
194.210.4.136 - - [04/Jul/2002:08:31:28 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
66.161.192.157 - - [04/Jul/2002:08:34:58 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
194.122.118.7 - - [04/Jul/2002:08:35:30 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
147.162.157.132 - - [04/Jul/2002:08:44:53 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
195.93.49.171 - - [04/Jul/2002:08:46:33 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
194.158.127.36 - - [04/Jul/2002:08:55:38 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.0" 200 1119
213.172.193.30 - - [04/Jul/2002:08:59:24 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
216.209.121.163 - - [04/Jul/2002:09:05:38 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
172.157.54.205 - - [04/Jul/2002:09:10:10 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
217.128.114.130 - - [04/Jul/2002:09:10:26 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
80.60.91.45 - - [04/Jul/2002:09:24:04 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
62.45.106.224 - - [04/Jul/2002:09:37:39 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
4.47.217.6 - - [04/Jul/2002:09:39:19 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119
192.167.218.113 - - [04/Jul/2002:10:31:25 -0400] "GET
/.cobalt/images/computer_warning.gif HTTP/1.1" 200 1119


-- from the adm_error log --

[Thu Jul  4 07:55:03 2002] [error] System: Cannot allocate memory (errno:
12)
[Thu Jul  4 07:55:03 2002] [error] OpenSSL: error:07064021:memory buffer
routines:BUF_MEM_grow:Malloc failure
[Thu Jul  4 07:55:03 2002] [error] mod_ssl: SSL handshake interrupted by
system (System error follows)

(Ton of these, only during the hours all the other oddities started
showing up.  Of course, I imagine a bunch of em are us trying to access
the admin.)

-- from httpd error log --

Ouch!  malloc failed in malloc_block()

(Too many of these to count...  I guess I don't really need it spelled out
for me, this was an attack of some sort wasn't it?)

If I find more info I'll pass it along, I appreciate your thought's and
comments...

thanks,
Brad

__________________________________________________
Do You Yahoo!?
Sign up for SBC Yahoo! Dial - First Month Free
http://sbc.yahoo.com