[cobalt-users] Raq4r - Unable to handle kernel NULL pointer dereference at virtual address

["Jon" <jjma100@xxxxxxxxxxx>]

Hello,

Excuse the long post. Last night our raq4r's services stopped at
4.30am, around the time the daily cronjob works forcing the
server to be manually rebooted at 8am.  Looking at what caused
the problem I found this in /varlog/kernel:

Dec 18 04:23:40 l4i kernel: Unable to handle kernel NULL pointer
dereference at virtual address 00000011
Dec 18 04:23:40 l4i kernel: current->tss.cr3 = 0c049000, %%cr3 =
0c049000
Dec 18 04:23:40 l4i kernel: *pde = 00000000
Dec 18 04:23:40 l4i kernel: Oops: 0000
Dec 18 04:23:40 l4i kernel: CPU:    0
Dec 18 04:23:40 l4i kernel: EIP:
0010:[ext2_free_blocks+40/820]
Dec 18 04:23:40 l4i kernel: EFLAGS: 00010286
Dec 18 04:23:40 l4i kernel: eax: 0109ab25   ebx: f5448ae0   ecx:
0109ab25   edx: ea83d870
Dec 18 04:23:40 l4i kernel: esi: fbfe0400   edi: 00000000   ebp:
0109ab25   esp: c2749eb0
Dec 18 04:23:40 l4i kernel: ds: 0018   es: 0018   ss: 0018
Dec 18 04:23:40 l4i kernel: Process webalizer (pid: 31866,
process nr: 23, stackpage=c2749000)
Dec 18 04:23:40 l4i kernel: Stack: 00000000 ea83d870 00000004
0109ab28 c0126086 f5448ae0 f5448ae0 00000003
Dec 18 04:23:40 l4i kernel:        c01427a8 ea83d870 0109ab25
00000004 c2748000 ea83d870 ea83d870 ea83d940
Dec 18 04:23:40 l4i kernel:        ea83d93c 00000002 0109ab25
00000000 0000000c c0142da6 ea83d870 c2748000
Dec 18 04:23:40 l4i kernel: Call Trace: [put_last_free+50/124]
[trunc_direct+292/308] [ext2_truncate+98/496]
[do_truncate+102/156] $
Dec 18 04:23:40 l4i kernel:        [sys_open+54/148]
[system_call+52/56]
Dec 18 04:23:40 l4i kernel: Code: 80 7f 11 00 74 09 56 e8 c0 d9
fe ff 83 c4 04 c6 46 11 01 ba
Dec 18 04:23:40 l4i kernel: Unable to handle kernel NULL pointer
dereference at virtual address 00000011
Dec 18 04:23:40 l4i kernel: current->tss.cr3 = 31de2000, %%cr3 =
31de2000
Dec 18 04:23:40 l4i kernel: *pde = 00000000
Dec 18 04:23:40 l4i kernel: Oops: 0000
Dec 18 04:23:40 l4i kernel: CPU:    0
Dec 18 04:23:40 l4i kernel: EIP:
0010:[ext2_free_blocks+40/820]
Dec 18 04:23:40 l4i kernel: EFLAGS: 00010286
Dec 18 04:23:40 l4i kernel: eax: ea3c3b00   ebx: f1ba50c0   ecx:
010b3115   edx: ea004220
Dec 18 04:23:40 l4i kernel: esi: fbfe0400   edi: 00000000   ebp:
010b3115   esp: cb649eb0
Dec 18 04:23:40 l4i kernel: ds: 0018   es: 0018   ss: 0018
Dec 18 04:23:40 l4i kernel: Process webalizer2.pl (pid: 31787,
process nr: 73, stackpage=cb649000)
Dec 18 04:23:40 l4i kernel: Stack: 00000000 ea004220 00000004
010b3115 c0126086 f1ba50c0 f1ba50c0 00000000
Dec 18 04:23:40 l4i kernel:        c01427a8 ea004220 010b3115
00000001 cb648000 ea004220 ea004220 ea0042f0
Dec 18 04:23:40 l4i kernel:        ea0042ec 00000002 010b3115
00000000 0000000c c0142da6 ea004220 cb648000
Dec 18 04:23:40 l4i kernel: Call Trace: [put_last_free+50/124]
[trunc_direct+292/308] [ext2_truncate+98/496]
[do_truncate+102/156] $
Dec 18 04:23:40 l4i kernel:        [sys_open+54/148]
[system_call+52/56]
Dec 18 04:23:40 l4i kernel: Code: 80 7f 11 00 74 09 56 e8 c0 d9
fe ff 83 c4 04 c6 46 11 01 ba
Dec 18 04:23:40 l4i kernel: Unable to handle kernel paging
request at virtual address 001cd47e
Dec 18 04:23:40 l4i kernel: current->tss.cr3 = 32a61000, %%cr3 =
32a61000
Dec 18 04:23:40 l4i kernel: *pde = 00000000
Dec 18 04:23:40 l4i kernel: Oops: 0000
Dec 18 04:23:40 l4i kernel: CPU:    0
Dec 18 04:23:40 l4i kernel: EIP:
0010:[ext2_free_blocks+40/820]
Dec 18 04:23:40 l4i kernel: EFLAGS: 00010286
Dec 18 04:23:40 l4i kernel: eax: 001cd464   ebx: f9687c80   ecx:
001c1776   edx: ea886330
Dec 18 04:23:40 l4i kernel: esi: fbfe0400   edi: 001cd46d   ebp:
001c1776   esp: d8e1bec8
Dec 18 04:23:40 l4i kernel: ds: 0018   es: 0018   ss: 0018
Dec 18 04:23:40 l4i kernel: Process httpd (pid: 14219, process
nr: 40, stackpage=d8e1b000)
Dec 18 04:23:40 l4i kernel: Stack: 001cd46d ea886330 00000004
001cd46d c0126086 f9687c80 f9687c80 00000009
Dec 18 04:23:40 l4i kernel:        c0142778 ea886330 001c1776
00000009 ea886330 ea886330 00000000 ea886400
Dec 18 04:23:40 l4i kernel:        ea8863f0 00000002 001c1776
00000000 00000009 c0142da6 ea886330 ea886330
Dec 18 04:23:40 l4i kernel: Call Trace: [put_last_free+50/124]
[trunc_direct+244/308] [ext2_truncate+98/496]
[ext2_delete_inode+102$
Dec 18 04:23:40 l4i kernel:        [ext2_unlink+383/416]
[vfs_unlink+225/232] [sys_unlink+106/164] [system_call+52/56]
Dec 18 04:23:40 l4i kernel: Code: 80 7f 11 00 74 09 56 e8 c0 d9
fe ff 83 c4 04 c6 46 11 01 ba

The last two days I have been getting cron errors that our access
log has not been retotated due to an error running prerotate
script although running it after the server was rebooted manually
showed no error.

I had installed 2 512mb chips from crucial a while back (4 months
ago) and the server is running 20 sites which are not very busy
so obvioulsy any pointers would be appreciated.

regards

Jon