cauldron is oom-killing itself

| | Comments (0)
very irritating;  this may necessitate a reboot.

Mem-info:
DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: high 186, batch 31 used:30
cpu 0 cold: high 62, batch 15 used:60
Normal per-cpu: empty
HighMem per-cpu: empty
Free pages:        5412kB (0kB HighMem)
Active:152866 inactive:63067 dirty:51 writeback:0 unstable:0 free:1353 slab:17833 mapped:3657 pagetables:1058
DMA free:4012kB min:4kB low:4kB high:4kB active:3840kB inactive:0kB present:1956kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 1002 1002 1002
DMA32 free:1400kB min:4044kB low:5052kB high:6064kB active:607624kB inactive:252268kB present:1026160kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4012kB
DMA32: 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1400kB
Normal: empty
HighMem: empty
Swap cache: add 26, delete 26, find 0/0, race 0+0
Free swap  = 1048472kB
Total swap = 1048568kB
Free swap:       1048472kB
264192 pages of RAM
22575 reserved pages
145636 pages shared
0 pages swap cached
peth0: too many iterations (6) in nv_nic_irq_rx.
yum-updatesd-he: page allocation failure. order:0, mode:0x20

Call Trace:
 <IRQ> [<ffffffff8025f43f>] __alloc_pages+0x299/0x2b2
 [<ffffffff80279c31>] cache_alloc_refill+0x2b2/0x535
 [<ffffffff80279f64>] __kmalloc+0xb0/0xf0
 [<ffffffff8039ae62>] __alloc_skb+0x5c/0x123
 [<ffffffff88122e27>] :forcedeth:nv_nic_irq_rx+0x3ba/0x5bd
 [<ffffffff80257874>] handle_IRQ_event+0x4e/0x96
 [<ffffffff80257960>] __do_IRQ+0xa4/0x105
 [<ffffffff8020bd5c>] do_IRQ+0x44/0x4d
 [<ffffffff8034c980>] evtchn_do_upcall+0x19e/0x250
 [<ffffffff80209d8e>] do_hypervisor_callback+0x1e/0x2c
 [<ffffffff8020522a>] hypercall_page+0x22a/0x1000
 [<ffffffff8020522a>] hypercall_page+0x22a/0x1000
 [<ffffffff8034b922>] force_evtchn_callback+0xa/0xb
 [<ffffffff80279f93>] __kmalloc+0xdf/0xf0
 [<ffffffff8039a79b>] pskb_expand_head+0x51/0x137
 [<ffffffff8822be00>] :bridge:br_dev_queue_push_xmit+0x13e/0x1ad
 [<ffffffff882300e6>] :bridge:br_nf_post_routing+0x17a/0x195
 [<ffffffff882124d4>] :ip6_tables:ipv6_find_hdr+0x4e/0x1a1
 [<ffffffff803b676d>] nf_iterate+0x41/0x7d
 [<ffffffff8822bcc2>] :bridge:br_dev_queue_push_xmit+0x0/0x1ad
 [<ffffffff803b68f2>] nf_hook_slow+0x58/0xbc
 [<ffffffff8822bcc2>] :bridge:br_dev_queue_push_xmit+0x0/0x1ad
 [<ffffffff8822beae>] :bridge:br_forward_finish+0x3f/0x51
 [<ffffffff8822ff64>] :bridge:br_nf_forward_finish+0xf7/0xff
 [<ffffffff8823080f>] :bridge:br_nf_forward_ip+0x14d/0x15d
 [<ffffffff803b676d>] nf_iterate+0x41/0x7d
 [<ffffffff8822be6f>] :bridge:br_forward_finish+0x0/0x51
 [<ffffffff803b68f2>] nf_hook_slow+0x58/0xbc
 [<ffffffff8822be6f>] :bridge:br_forward_finish+0x0/0x51
 [<ffffffff8822bec0>] :bridge:__br_forward+0x0/0x6d
 [<ffffffff8822bf19>] :bridge:__br_forward+0x59/0x6d
 [<ffffffff8822bc61>] :bridge:br_flood+0x7d/0xc6
 [<ffffffff8822c9d9>] :bridge:br_handle_frame_finish+0x9b/0xf8
 [<ffffffff882305d7>] :bridge:br_nf_pre_routing_finish+0x2ed/0x2fc
 [<ffffffff882302ea>] :bridge:br_nf_pre_routing_finish+0x0/0x2fc
 [<ffffffff803b68f2>] nf_hook_slow+0x58/0xbc
 [<ffffffff882302ea>] :bridge:br_nf_pre_routing_finish+0x0/0x2fc
 [<ffffffff8822c93c>] :bridge:br_pass_frame_up+0x67/0x69
 [<ffffffff882311ee>] :bridge:br_nf_pre_routing+0x611/0x62f
 [<ffffffff803b676d>] nf_iterate+0x41/0x7d
 [<ffffffff8822c93e>] :bridge:br_handle_frame_finish+0x0/0xf8
 [<ffffffff803b68f2>] nf_hook_slow+0x58/0xbc
 [<ffffffff8822c93e>] :bridge:br_handle_frame_finish+0x0/0xf8
 [<ffffffff8822cba4>] :bridge:br_handle_frame+0x16e/0x1a2
 [<ffffffff8039edb6>] netif_receive_skb+0x1ca/0x2ea
 [<ffffffff803a0d6e>] process_backlog+0xd0/0x182
 [<ffffffff803a0fe8>] net_rx_action+0xe3/0x24b
 [<ffffffff802339f8>] __do_softirq+0x83/0x117
 [<ffffffff8034c980>] evtchn_do_upcall+0x19e/0x250
 [<ffffffff803fc644>] call_softirq+0x1c/0x26
 [<ffffffff8020c055>] do_softirq+0x6a/0xed
 [<ffffffff80209d8e>] do_hypervisor_callback+0x1e/0x2c
 <EOI>
Mem-info:
DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: high 186, batch 31 used:30
cpu 0 cold: high 62, batch 15 used:60
Normal per-cpu: empty
HighMem per-cpu: empty
Free pages:        5412kB (0kB HighMem)
Active:152866 inactive:63067 dirty:51 writeback:0 unstable:0 free:1353 slab:17833 mapped:3657 pagetables:1058
DMA free:4012kB min:4kB low:4kB high:4kB active:3840kB inactive:0kB present:1956kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 1002 1002 1002
DMA32 free:1400kB min:4044kB low:5052kB high:6064kB active:607624kB inactive:252268kB present:1026160kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4012kB
DMA32: 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1400kB
Normal: empty
HighMem: empty
Swap cache: add 26, delete 26, find 0/0, race 0+0
Free swap  = 1048472kB
Total swap = 1048568kB
Free swap:       1048472kB
264192 pages of RAM
22575 reserved pages
145636 pages shared
0 pages swap cached
yum-updatesd-he: page allocation failure. order:0, mode:0x20

Call Trace:
 <IRQ> [<ffffffff8025f43f>] __alloc_pages+0x299/0x2b2
 [<ffffffff80279c31>] cache_alloc_refill+0x2b2/0x535
 [<ffffffff80279f64>] __kmalloc+0xb0/0xf0
 [<ffffffff8039a79b>] pskb_expand_head+0x51/0x137
 [<ffffffff8822be00>] :bridge:br_dev_queue_push_xmit+0x13e/0x1ad
 [<ffffffff882300e6>] :bridge:br_nf_post_routing+0x17a/0x195
 [<ffffffff882124d4>] :ip6_tables:ipv6_find_hdr+0x4e/0x1a1
 [<ffffffff803b676d>] nf_iterate+0x41/0x7d
 [<ffffffff8822bcc2>] :bridge:br_dev_queue_push_xmit+0x0/0x1ad
 [<ffffffff803b68f2>] nf_hook_slow+0x58/0xbc
 [<ffffffff8822bcc2>] :bridge:br_dev_queue_push_xmit+0x0/0x1ad
 [<ffffffff8822beae>] :bridge:br_forward_finish+0x3f/0x51
 [<ffffffff8822ff64>] :bridge:br_nf_forward_finish+0xf7/0xff
 [<ffffffff8823080f>] :bridge:br_nf_forward_ip+0x14d/0x15d
 [<ffffffff803b676d>] nf_iterate+0x41/0x7d
 [<ffffffff8822be6f>] :bridge:br_forward_finish+0x0/0x51
 [<ffffffff803b68f2>] nf_hook_slow+0x58/0xbc
 [<ffffffff8822be6f>] :bridge:br_forward_finish+0x0/0x51
 [<ffffffff8822bec0>] :bridge:__br_forward+0x0/0x6d
 [<ffffffff8822bf19>] :bridge:__br_forward+0x59/0x6d
 [<ffffffff8822bc61>] :bridge:br_flood+0x7d/0xc6
 [<ffffffff8822c9d9>] :bridge:br_handle_frame_finish+0x9b/0xf8
 [<ffffffff882305d7>] :bridge:br_nf_pre_routing_finish+0x2ed/0x2fc
 [<ffffffff882302ea>] :bridge:br_nf_pre_routing_finish+0x0/0x2fc
 [<ffffffff803b68f2>] nf_hook_slow+0x58/0xbc
 [<ffffffff882302ea>] :bridge:br_nf_pre_routing_finish+0x0/0x2fc
 [<ffffffff8822c93c>] :bridge:br_pass_frame_up+0x67/0x69
 [<ffffffff882311ee>] :bridge:br_nf_pre_routing+0x611/0x62f
 [<ffffffff803b676d>] nf_iterate+0x41/0x7d
 [<ffffffff8822c93e>] :bridge:br_handle_frame_finish+0x0/0xf8
 [<ffffffff803b68f2>] nf_hook_slow+0x58/0xbc
 [<ffffffff8822c93e>] :bridge:br_handle_frame_finish+0x0/0xf8
 [<ffffffff8822cba4>] :bridge:br_handle_frame+0x16e/0x1a2
 [<ffffffff8039edb6>] netif_receive_skb+0x1ca/0x2ea
 [<ffffffff803a0d6e>] process_backlog+0xd0/0x182
 [<ffffffff803a0fe8>] net_rx_action+0xe3/0x24b
 [<ffffffff802339f8>] __do_softirq+0x83/0x117
 [<ffffffff8034c980>] evtchn_do_upcall+0x19e/0x250
 [<ffffffff803fc644>] call_softirq+0x1c/0x26
 [<ffffffff8020c055>] do_softirq+0x6a/0xed
 [<ffffffff80209d8e>] do_hypervisor_callback+0x1e/0x2c
 <EOI>
Mem-info:
DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: high 186, batch 31 used:30
cpu 0 cold: high 62, batch 15 used:60
Normal per-cpu: empty
HighMem per-cpu: empty
Free pages:        5412kB (0kB HighMem)
Active:152866 inactive:63067 dirty:51 writeback:0 unstable:0 free:1353 slab:17833 mapped:3657 pagetables:1058
DMA free:4012kB min:4kB low:4kB high:4kB active:3840kB inactive:0kB present:1956kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 1002 1002 1002
DMA32 free:1400kB min:4044kB low:5052kB high:6064kB active:607624kB inactive:252268kB present:1026160kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4012kB
DMA32: 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1400kB
Normal: empty
HighMem: empty
Swap cache: add 26, delete 26, find 0/0, race 0+0
Free swap  = 1048472kB
Total swap = 1048568kB
Free swap:       1048472kB
264192 pages of RAM
22575 reserved pages
145636 pages shared
0 pages swap cached


yup.  rebooted. 

Leave a comment

About this Entry

This page contains a single entry by luke published on September 10, 2012 3:35 AM.

another IPv6/multicast related crash; hughes. was the previous entry in this blog.

network maintenance affecting he.net connection is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.