| View previous topic :: View next topic |
| Author |
Message |
tnt
Veteran
Joined: 27 Feb 2004
Posts: 1227
|
 Posted: Thu Oct 08, 2009 12:45 pm Post subject: strange kernel messages under network load |
 |
|
as soon as I start heavy outgoing traffic to my server (~104MB/s via NFS), strange messages start to appear in my server's log:
| Code: | Oct 8 14:33:08 titan swapper: page allocation failure. order:0, mode:0x4020
Oct 8 14:33:08 titan Pid: 0, comm: swapper Not tainted 2.6.29-gentoo-r5 #1
Oct 8 14:33:08 titan Call Trace:
Oct 8 14:33:08 titan <IRQ> [<ffffffff8025d888>] __alloc_pages_internal+0x3e1/0x401
Oct 8 14:33:08 titan [<ffffffff8027d217>] __slab_alloc+0x155/0x433
Oct 8 14:33:08 titan [<ffffffff8048d840>] __netdev_alloc_skb+0x15/0x2f
Oct 8 14:33:08 titan [<ffffffff8048d840>] __netdev_alloc_skb+0x15/0x2f
Oct 8 14:33:08 titan [<ffffffff8027e3e1>] __kmalloc_track_caller+0x90/0xb7
Oct 8 14:33:08 titan [<ffffffff8048cc88>] __alloc_skb+0x61/0x124
Oct 8 14:33:08 titan [<ffffffff8048d840>] __netdev_alloc_skb+0x15/0x2f
Oct 8 14:33:08 titan [<ffffffff80406044>] rtl8169_rx_fill+0xb0/0x160
Oct 8 14:33:08 titan [<ffffffff8040645c>] rtl8169_rx_interrupt+0x368/0x3cb
Oct 8 14:33:08 titan [<ffffffff80408ce8>] rtl8169_poll+0x3b/0x19e
Oct 8 14:33:08 titan [<ffffffff80490b11>] net_rx_action+0x69/0x127
Oct 8 14:33:08 titan [<ffffffff80235eeb>] __do_softirq+0x7c/0x115
Oct 8 14:33:08 titan [<ffffffff8020c57c>] call_softirq+0x1c/0x28
Oct 8 14:33:08 titan [<ffffffff8020db70>] do_softirq+0x2c/0x6c
Oct 8 14:33:08 titan [<ffffffff80235ca0>] irq_exit+0x3f/0x7b
Oct 8 14:33:08 titan [<ffffffff8020dc66>] do_IRQ+0xb6/0xd4
Oct 8 14:33:08 titan [<ffffffff8020be53>] ret_from_intr+0x0/0xa
Oct 8 14:33:08 titan <EOI> [<ffffffff80211713>] mwait_idle+0x6a/0x6d
Oct 8 14:33:08 titan [<ffffffff8020a6b9>] cpu_idle+0x4a/0x8b
Oct 8 14:33:08 titan Mem-Info:
Oct 8 14:33:08 titan DMA per-cpu:
Oct 8 14:33:08 titan CPU 0: hi: 0, btch: 1 usd: 0
Oct 8 14:33:08 titan CPU 1: hi: 0, btch: 1 usd: 0
Oct 8 14:33:08 titan DMA32 per-cpu:
Oct 8 14:33:08 titan CPU 0: hi: 186, btch: 31 usd: 137
Oct 8 14:33:08 titan CPU 1: hi: 186, btch: 31 usd: 185
Oct 8 14:33:08 titan Normal per-cpu:
Oct 8 14:33:08 titan CPU 0: hi: 186, btch: 31 usd: 78
Oct 8 14:33:08 titan CPU 1: hi: 186, btch: 31 usd: 74
Oct 8 14:33:08 titan Active_anon:183604 active_file:32237 inactive_anon:28455
Oct 8 14:33:08 titan inactive_file:1473410 unevictable:0 dirty:92815 writeback:26 unstable:0
Oct 8 14:33:08 titan free:7912 slab:125696 mapped:6360 pagetables:1777 bounce:0
Oct 8 14:33:08 titan DMA free:10592kB min:12kB low:12kB high:16kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:9604kB pages_scanned:0 all_unreclaimable? yes
Oct 8 14:33:08 titan lowmem_reserve[]: 0 3831 8060 8060
Oct 8 14:33:08 titan DMA32 free:18908kB min:5456kB low:6820kB high:8184kB active_anon:26352kB inactive_anon:5308kB active_file:10104kB inactive_file:3602928kB unevictable:0kB present:3923040kB pages_scanned:0 all_unreclaimable? no
Oct 8 14:33:08 titan lowmem_reserve[]: 0 0 4229 4229
Oct 8 14:33:08 titan Normal free:2148kB min:6024kB low:7528kB high:9036kB active_anon:708064kB inactive_anon:108512kB active_file:118844kB inactive_file:2290712kB unevictable:0kB present:4330880kB pages_scanned:0 all_unreclaimable? no
Oct 8 14:33:08 titan lowmem_reserve[]: 0 0 0 0
Oct 8 14:33:08 titan DMA: 4*4kB 6*8kB 4*16kB 3*32kB 2*64kB 2*128kB 3*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10592kB
Oct 8 14:33:08 titan DMA32: 9*4kB 53*8kB 61*16kB 2*32kB 37*64kB 21*128kB 2*256kB 1*512kB 1*1024kB 1*2048kB 2*4096kB = 18844kB
Oct 8 14:33:08 titan Normal: 257*4kB 22*8kB 12*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2068kB
Oct 8 14:33:08 titan 1514341 total pagecache pages
Oct 8 14:33:08 titan 0 pages in swap cache
Oct 8 14:33:08 titan Swap cache stats: add 0, delete 0, find 0/0
Oct 8 14:33:08 titan Free swap = 6297408kB
Oct 8 14:33:08 titan Total swap = 6297408kB
Oct 8 14:33:08 titan 2146304 pages RAM
Oct 8 14:33:08 titan 100234 pages reserved
Oct 8 14:33:08 titan 1433446 pages shared
Oct 8 14:33:08 titan 633401 pages non-shared
Oct 8 14:33:08 titan swapper: page allocation failure. order:0, mode:0x20
Oct 8 14:33:08 titan Pid: 0, comm: swapper Not tainted 2.6.29-gentoo-r5 #1
Oct 8 14:33:08 titan Call Trace:
Oct 8 14:33:08 titan <IRQ> [<ffffffff8025d888>] __alloc_pages_internal+0x3e1/0x401
Oct 8 14:33:08 titan [<ffffffff8027d217>] __slab_alloc+0x155/0x433
Oct 8 14:33:08 titan [<ffffffff8048bd71>] skb_clone+0x4e/0x6a
Oct 8 14:33:08 titan [<ffffffff8027d749>] kmem_cache_alloc+0x49/0x6c
Oct 8 14:33:08 titan [<ffffffff8048bd71>] skb_clone+0x4e/0x6a
Oct 8 14:33:08 titan [<ffffffff80492c0c>] dev_hard_start_xmit+0xa8/0x283
Oct 8 14:33:08 titan [<ffffffff804a107d>] __qdisc_run+0xde/0x1d9
Oct 8 14:33:08 titan [<ffffffff804930d9>] dev_queue_xmit+0x1e5/0x2d9
Oct 8 14:33:08 titan [<ffffffff804c5afa>] ip_queue_xmit+0x2c3/0x319
Oct 8 14:33:08 titan [<ffffffff804d56da>] tcp_transmit_skb+0x5d3/0x612
Oct 8 14:33:08 titan [<ffffffff804d3c07>] tcp_rcv_established+0x55a/0x6eb
Oct 8 14:33:08 titan [<ffffffff804d9f5b>] tcp_v4_do_rcv+0x2c/0x1b1
Oct 8 14:33:08 titan [<ffffffff804da518>] tcp_v4_rcv+0x438/0x654
Oct 8 14:33:08 titan [<ffffffff804c18fd>] ip_local_deliver_finish+0xf8/0x194
Oct 8 14:33:08 titan [<ffffffff804c1512>] ip_rcv_finish+0x2d2/0x2ec
Oct 8 14:33:08 titan [<ffffffff804c1768>] ip_rcv+0x23c/0x280
Oct 8 14:33:08 titan [<ffffffff804063f7>] rtl8169_rx_interrupt+0x303/0x3cb
Oct 8 14:33:08 titan [<ffffffff8022b917>] enqueue_task_fair+0x13f/0x152
Oct 8 14:33:08 titan [<ffffffff80408ce8>] rtl8169_poll+0x3b/0x19e
Oct 8 14:33:08 titan [<ffffffff80490b11>] net_rx_action+0x69/0x127
Oct 8 14:33:08 titan [<ffffffff80235eeb>] __do_softirq+0x7c/0x115
Oct 8 14:33:08 titan [<ffffffff8020c57c>] call_softirq+0x1c/0x28
Oct 8 14:33:08 titan [<ffffffff8020db70>] do_softirq+0x2c/0x6c
Oct 8 14:33:08 titan [<ffffffff80235ca0>] irq_exit+0x3f/0x7b
Oct 8 14:33:08 titan [<ffffffff8020dc66>] do_IRQ+0xb6/0xd4
Oct 8 14:33:08 titan [<ffffffff8020be53>] ret_from_intr+0x0/0xa
Oct 8 14:33:08 titan <EOI> [<ffffffff80211713>] mwait_idle+0x6a/0x6d
Oct 8 14:33:08 titan [<ffffffff8020a6b9>] cpu_idle+0x4a/0x8b
Oct 8 14:33:08 titan Mem-Info:
Oct 8 14:33:08 titan DMA per-cpu:
Oct 8 14:33:08 titan CPU 0: hi: 0, btch: 1 usd: 0
Oct 8 14:33:08 titan CPU 1: hi: 0, btch: 1 usd: 0
Oct 8 14:33:08 titan DMA32 per-cpu:
Oct 8 14:33:08 titan CPU 0: hi: 186, btch: 31 usd: 137
Oct 8 14:33:08 titan CPU 1: hi: 186, btch: 31 usd: 185
Oct 8 14:33:08 titan Normal per-cpu:
Oct 8 14:33:08 titan CPU 0: hi: 186, btch: 31 usd: 79
Oct 8 14:33:08 titan CPU 1: hi: 186, btch: 31 usd: 75
Oct 8 14:33:08 titan Active_anon:183604 active_file:32239 inactive_anon:28455
Oct 8 14:33:08 titan inactive_file:1473410 unevictable:0 dirty:92861 writeback:27 unstable:0
Oct 8 14:33:08 titan free:7910 slab:125708 mapped:6360 pagetables:1777 bounce:0
Oct 8 14:33:08 titan DMA free:10592kB min:12kB low:12kB high:16kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:9604kB pages_scanned:0 all_unreclaimable? yes
Oct 8 14:33:08 titan lowmem_reserve[]: 0 3831 8060 8060
Oct 8 14:33:08 titan DMA32 free:18916kB min:5456kB low:6820kB high:8184kB active_anon:26352kB inactive_anon:5308kB active_file:10116kB inactive_file:3602928kB unevictable:0kB present:3923040kB pages_scanned:0 all_unreclaimable? no
Oct 8 14:33:08 titan lowmem_reserve[]: 0 0 4229 4229
Oct 8 14:33:08 titan Normal free:2132kB min:6024kB low:7528kB high:9036kB active_anon:708064kB inactive_anon:108512kB active_file:118840kB inactive_file:2290712kB unevictable:0kB present:4330880kB pages_scanned:0 all_unreclaimable? no
Oct 8 14:33:08 titan lowmem_reserve[]: 0 0 0 0
Oct 8 14:33:08 titan DMA: 4*4kB 6*8kB 4*16kB 3*32kB 2*64kB 2*128kB 3*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10592kB
Oct 8 14:33:08 titan DMA32: 9*4kB 54*8kB 61*16kB 2*32kB 37*64kB 21*128kB 2*256kB 1*512kB 1*1024kB 1*2048kB 2*4096kB = 18852kB
Oct 8 14:33:08 titan Normal: 257*4kB 21*8kB 11*16kB 2*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2076kB
Oct 8 14:33:08 titan 1514339 total pagecache pages
Oct 8 14:33:08 titan 0 pages in swap cache
Oct 8 14:33:08 titan Swap cache stats: add 0, delete 0, find 0/0
Oct 8 14:33:08 titan Free swap = 6297408kB
Oct 8 14:33:08 titan Total swap = 6297408kB
Oct 8 14:33:08 titan 2146304 pages RAM
Oct 8 14:33:08 titan 100234 pages reserved
Oct 8 14:33:08 titan 1433222 pages shared
Oct 8 14:33:08 titan 633231 pages non-shared |
server's heavy outgoing traffic doesn't produce any messages in logs.
any clue ?
_________________
gentoo user |
|
| Back to top |
|
 |
whig
l33t
Joined: 27 Nov 2004
Posts: 973
Location: New Zealand
|
| Posted: Thu Oct 08, 2009 9:45 pm Post subject: |
|
|
| I once had very poor outgoing bandwidth performance from a server. In the end upgrading the mobo bios was the answer. ymmv. |
|
| Back to top |
|
|
tnt
Veteran
Joined: 27 Feb 2004
Posts: 1227
|
| Posted: Thu Oct 08, 2009 11:00 pm Post subject: |
|
|
this mobo (complete hardware) works without a problem for 28 months (24/7).
maybe this is comething kernel 2.6.29 - related... 
_________________
gentoo user |
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Thu Oct 08, 2009 11:55 pm Post subject: |
|
|
| From those messages it sounds like it's trying to allocate RAM (for DMA?) and failing. You could try tweaking the memory usage controls in /proc/net/ but I'm not sure if it'd help. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Kernel & Hardware
