Gentoo Forums :: View topic - Keep running out of memory (solved)

Keep running out of memory (solved)

View unanswered posts
View posts from last 24 hours

Goto page Previous 1, 2

Gentoo Forums Forum Index

Kernel & Hardware

View previous topic :: View next topic

Author

Message

Jaglover
Watchman

Joined: 29 May 2005
Posts: 8291
Location: Saint Amant, Acadiana

Posted: Thu Mar 30, 2017 3:06 pm Post subject:

Turning COMPACTION on will not remove the cause of original issue ... _________________ My Gentoo installation notes. Please learn how to denote units correctly!

Bigun
Advocate

Joined: 21 Sep 2003
Posts: 2198

Posted: Sat Apr 01, 2017 2:19 pm Post subject:

So it's happening again:

Code:

# ~/ps_mem.py -S
Private + Shared = RAM used Swap used Program

100.0 KiB + 76.0 KiB = 176.0 KiB 0.0 KiB 0.0 KiB init
188.0 KiB + 80.5 KiB = 268.5 KiB 0.0 KiB 0.0 KiB rpc.idmapd
168.0 KiB + 110.5 KiB = 278.5 KiB 0.0 KiB 0.0 KiB cron
196.0 KiB + 88.0 KiB = 284.0 KiB 0.0 KiB 0.0 KiB rpcbind
492.0 KiB + 74.0 KiB = 566.0 KiB 0.0 KiB 0.0 KiB rpc.mountd
376.0 KiB + 211.0 KiB = 587.0 KiB 0.0 KiB 0.0 KiB su
592.0 KiB + 102.0 KiB = 694.0 KiB 0.0 KiB 0.0 KiB rpc.statd
728.0 KiB + 70.0 KiB = 798.0 KiB 0.0 KiB 0.0 KiB smartd
1.1 MiB + 147.0 KiB = 1.3 MiB 0.0 KiB 0.0 KiB systemd-udevd
784.0 KiB + 578.0 KiB = 1.3 MiB 0.0 KiB 0.0 KiB agetty (6)
664.0 KiB + 1.3 MiB = 1.9 MiB 0.0 KiB 0.0 KiB bash (2)
1.4 MiB + 636.0 KiB = 2.0 MiB 0.0 KiB 0.0 KiB syslog-ng (2)
1.9 MiB + 2.4 MiB = 4.2 MiB 0.0 KiB 0.0 KiB sshd (3)
-------------------------------------------------------------
14.3 MiB 0.0 KiB 0.0 KiB
=============================================================

Code:

top - 10:19:06 up 2 days, 40 min, 1 user, load average: 0.34, 0.36, 0.36
Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie
%Cpu(s): 3.4 us, 1.8 sy, 1.3 ni, 89.0 id, 4.4 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 8230380 total, 47512 free, 718668 used, 7464200 buff/cache
KiB Swap: 979836 total, 979624 free, 212 used. 7215344 avail Mem

Code:

# ps -A
PID TTY TIME CMD
1 ? 00:00:01 init
2 ? 00:00:00 kthreadd
3 ? 00:00:18 ksoftirqd/0
7 ? 00:00:14 rcu_sched
8 ? 00:00:00 rcu_bh
9 ? 00:00:00 migration/0
10 ? 00:00:00 lru-add-drain
11 ? 00:00:00 cpuhp/0
12 ? 00:00:00 cpuhp/1
13 ? 00:00:00 migration/1
14 ? 00:00:00 ksoftirqd/1
17 ? 00:00:00 kdevtmpfs
18 ? 00:00:00 netns
384 ? 00:00:00 oom_reaper
385 ? 00:00:00 writeback
387 ? 00:00:00 kcompactd0
388 ? 00:00:00 crypto
389 ? 00:00:00 bioset
391 ? 00:00:00 kblockd
566 ? 00:00:00 ata_sff
586 ? 00:00:00 md
595 ? 00:00:00 cfg80211
689 ? 00:00:00 rpciod
690 ? 00:00:00 xprtiod
723 ? 00:05:36 kswapd0
724 ? 00:00:00 vmstat
803 ? 00:00:00 nfsiod
812 ? 00:00:00 cifsiod
822 ? 00:00:00 bioset
830 ? 00:00:00 xfsalloc
831 ? 00:00:00 xfs_mru_cache
898 ? 00:00:00 acpi_thermal_pm
916 ? 00:00:00 i915/signal:0
917 ? 00:00:00 i915/signal:1
918 ? 00:00:00 i915/signal:2
935 ? 00:00:00 bioset
936 ? 00:00:00 bioset
937 ? 00:00:00 bioset
938 ? 00:00:00 bioset
939 ? 00:00:00 bioset
940 ? 00:00:00 bioset
941 ? 00:00:00 bioset
942 ? 00:00:00 bioset
943 ? 00:00:00 bioset
944 ? 00:00:00 bioset
945 ? 00:00:00 bioset
946 ? 00:00:00 bioset
947 ? 00:00:00 bioset
948 ? 00:00:00 bioset
949 ? 00:00:00 bioset
950 ? 00:00:00 bioset
985 ? 00:00:00 bioset
988 ? 00:00:00 bioset
991 ? 00:00:00 bioset
994 ? 00:00:00 bioset
997 ? 00:00:00 bioset
1000 ? 00:00:00 bioset
1003 ? 00:00:00 bioset
1007 ? 00:00:00 bioset
1009 ? 00:00:00 iscsi_eh
1042 ? 00:00:00 scsi_eh_0
1043 ? 00:00:00 scsi_tmf_0
1046 ? 00:00:00 scsi_eh_1
1047 ? 00:00:00 scsi_tmf_1
1050 ? 00:00:00 scsi_eh_2
1051 ? 00:00:00 scsi_tmf_2
1052 ? 00:00:00 scsi_eh_3
1055 ? 00:00:00 scsi_tmf_3
1058 ? 00:00:00 scsi_eh_4
1059 ? 00:00:00 scsi_tmf_4
1062 ? 00:00:00 scsi_eh_5
1063 ? 00:00:00 scsi_tmf_5
1066 ? 00:00:00 scsi_eh_6
1068 ? 00:00:00 scsi_tmf_6
1071 ? 00:00:00 scsi_eh_7
1073 ? 00:00:00 scsi_tmf_7
1112 ? 00:00:09 kworker/1:0H
1147 ? 00:00:00 raid5wq
1188 ? 00:00:00 bioset
1196 ? 00:00:00 bioset
1205 ? 00:00:00 bioset
1213 ? 00:00:00 bioset
1233 ? 00:00:00 bioset
1246 ? 00:00:00 bioset
1258 ? 00:00:00 bioset
1262 ? 00:00:00 bioset
1266 ? 00:00:00 bioset
1267 ? 00:01:40 md126_raid1
1269 ? 00:00:00 bioset
1273 ? 00:00:00 bioset
1274 ? 00:00:01 md125_raid1
1276 ? 00:00:00 bioset
1280 ? 00:00:00 bioset
1281 ? 00:00:00 md124_raid1
1284 ? 00:00:00 xfs-buf/md126
1285 ? 00:00:00 xfs-data/md126
1286 ? 00:00:00 xfs-conv/md126
1287 ? 00:00:00 xfs-cil/md126
1288 ? 00:00:00 xfs-reclaim/md1
1289 ? 00:00:00 xfs-log/md126
1290 ? 00:00:00 xfs-eofblocks/m
1291 ? 00:00:27 xfsaild/md126
1695 ? 00:00:00 systemd-udevd
1860 ? 00:00:00 bioset
1870 ? 00:00:00 bioset
1871 ? 00:55:37 md127_raid5
2062 ? 00:00:00 xfs-buf/md127
2063 ? 00:00:00 xfs-data/md127
2064 ? 00:00:00 xfs-conv/md127
2065 ? 00:00:00 xfs-cil/md127
2066 ? 00:00:00 xfs-reclaim/md1
2067 ? 00:00:00 xfs-log/md127
2068 ? 00:00:00 xfs-eofblocks/m
2069 ? 00:00:03 xfsaild/md127
2771 ? 00:00:00 syslog-ng
2772 ? 00:00:00 syslog-ng
2829 ? 00:00:00 rpcbind
2855 ? 00:00:00 rpc.statd
2905 ? 00:00:00 rpc.idmapd
2936 ? 00:00:00 rpc.mountd
2940 ? 00:00:00 nfsd4_callbacks
2941 ? 00:00:00 lockd
2943 ? 00:00:00 nfsd
2944 ? 00:00:00 nfsd
2945 ? 00:00:00 nfsd
2946 ? 00:00:00 nfsd
2947 ? 00:00:00 nfsd
2948 ? 00:00:00 nfsd
2949 ? 00:00:00 nfsd
2950 ? 00:00:00 nfsd
3153 ? 00:00:00 smartd
3183 ? 00:00:00 sshd
3208 ? 00:00:00 cron
3239 tty1 00:00:00 agetty
3240 tty2 00:00:00 agetty
3241 tty3 00:00:00 agetty
3242 tty4 00:00:00 agetty
3243 tty5 00:00:00 agetty
3244 tty6 00:00:00 agetty
3483 ? 00:00:45 cifsd
8959 ? 00:00:10 kworker/0:2
14053 ? 00:00:04 kworker/1:2
14067 ? 00:00:00 kworker/1:1H
14357 ? 00:00:00 kworker/u4:1
14960 ? 00:00:00 kworker/u4:2
29191 ? 00:00:00 kworker/0:1
29838 ? 00:00:00 kworker/0:0H
29860 ? 00:00:00 kworker/1:1
29861 ? 00:00:00 kworker/0:2H
29895 ? 00:00:00 kworker/1:0
29897 ? 00:00:00 kworker/u4:0
29898 ? 00:00:00 kworker/0:1H
29905 ? 00:00:00 sshd
29911 ? 00:00:00 sshd
29916 pts/0 00:00:00 bash
29970 ? 00:00:00 kworker/0:0
29976 pts/0 00:00:00 su
29979 pts/0 00:00:00 bash
29982 pts/0 00:00:00 ps

Code:

# tail -n 100 /var/log/messages
Apr 1 10:21:54 projector kernel: [ 3153] 0 3153 1098 128 7 3 0 0 smartd
Apr 1 10:21:54 projector kernel: [ 3183] 0 3183 1355 580 6 3 0 -1000 sshd
Apr 1 10:21:54 projector kernel: [ 3208] 0 3208 600 452 5 3 0 0 cron
Apr 1 10:21:54 projector kernel: [ 3239] 0 3239 1096 378 6 3 0 0 agetty
Apr 1 10:21:54 projector kernel: [ 3240] 0 3240 1096 351 6 3 0 0 agetty
Apr 1 10:21:54 projector kernel: [ 3241] 0 3241 1096 370 6 3 0 0 agetty
Apr 1 10:21:54 projector kernel: [ 3242] 0 3242 1096 359 6 3 0 0 agetty
Apr 1 10:21:54 projector kernel: [ 3243] 0 3243 1096 351 6 3 0 0 agetty
Apr 1 10:21:54 projector kernel: [ 3244] 0 3244 1096 349 5 3 0 0 agetty
Apr 1 10:21:54 projector kernel: [29905] 0 29905 2086 1172 7 3 0 0 sshd
Apr 1 10:21:54 projector kernel: [29911] 1000 29911 10412 1017 11 3 0 0 sshd
Apr 1 10:21:54 projector kernel: [29916] 1000 29916 925 741 4 3 0 0 bash
Apr 1 10:21:54 projector kernel: [29976] 1000 29976 755 578 5 3 0 0 su
Apr 1 10:21:54 projector kernel: [29979] 0 29979 925 761 6 3 0 0 bash
Apr 1 10:21:54 projector kernel: [29984] 0 29984 563 184 5 3 0 0 dmesg
Apr 1 10:21:54 projector kernel: Out of memory: Kill process 29905 (sshd) score 0 or sacrifice child
Apr 1 10:21:54 projector kernel: Killed process 29911 (sshd) total-vm:41648kB, anon-rss:736kB, file-rss:3332kB, shmem-rss:0kB
Apr 1 10:21:54 projector kernel: dmesg invoked oom-killer: gfp_mask=0x24040c0(GFP_KERNEL|__GFP_COMP), nodemask=0, order=2, oom_score_adj=0
Apr 1 10:21:54 projector kernel: dmesg cpuset=/ mems_allowed=0
Apr 1 10:21:54 projector kernel: CPU: 0 PID: 29984 Comm: dmesg Not tainted 4.9.6-gentoo-r1 #2
Apr 1 10:21:54 projector kernel: Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./H61M-DS2H, BIOS F5 04/02/2012
Apr 1 10:21:54 projector kernel: d5e75ca4 c12f0831 d5e75d7c c18ff2c4 d5e75ccc c10e86d5 00000000 d4c95900
Apr 1 10:21:54 projector kernel: 00200206 d5e75ccc c12f49e2 f0f40900 c18ff2c4 00000000 d5e75cf0 c10b571b
Apr 1 10:21:54 projector kernel: c10b556d 00000000 00000000 d5e75d7c d5e75d7c c19f2700 c19f29d4 d5e75d0c
Apr 1 10:21:54 projector kernel: Call Trace:
Apr 1 10:21:54 projector kernel: [<c12f0831>] dump_stack+0x47/0x5b
Apr 1 10:21:54 projector kernel: [<c10e86d5>] dump_header.isra.13+0x5d/0x168
Apr 1 10:21:54 projector kernel: [<c12f49e2>] ? ___ratelimit+0xa1/0xab
Apr 1 10:21:54 projector kernel: [<c10b571b>] oom_kill_process+0x66/0x2fe
Apr 1 10:21:54 projector kernel: [<c10b556d>] ? oom_badness+0xc5/0xfc
Apr 1 10:21:54 projector kernel: [<c10b5d08>] out_of_memory+0x254/0x28b
Apr 1 10:21:54 projector kernel: [<c10b89d3>] __alloc_pages_nodemask+0x91f/0x9c7
Apr 1 10:21:54 projector kernel: [<c12a9600>] ? avc_has_perm_noaudit+0x1/0x83
Apr 1 10:21:54 projector kernel: [<c10c8a25>] kmalloc_order+0x16/0x28
Apr 1 10:21:54 projector kernel: [<c106b16b>] devkmsg_open+0x39/0xc6
Apr 1 10:21:54 projector kernel: [<c1345a5b>] memory_open+0x48/0x4c
Apr 1 10:21:54 projector kernel: [<c10ee281>] chrdev_open+0x10c/0x12a
Apr 1 10:21:54 projector kernel: [<c10e8e24>] do_dentry_open+0x193/0x272
Apr 1 10:21:54 projector kernel: [<c10ee175>] ? cdev_put+0x1a/0x1a
Apr 1 10:21:54 projector kernel: [<c10e9be2>] vfs_open+0x45/0x4e
Apr 1 10:21:54 projector kernel: [<c10f634b>] path_openat+0xae7/0xcb2
Apr 1 10:21:54 projector kernel: [<c10f6547>] do_filp_open+0x31/0x77
Apr 1 10:21:54 projector kernel: [<c10ffc3b>] ? __alloc_fd+0x72/0x10f
Apr 1 10:21:54 projector kernel: [<c10e9ee1>] do_sys_open+0x12d/0x1a6
Apr 1 10:21:54 projector kernel: [<c10e9f72>] SyS_open+0x18/0x1a
Apr 1 10:21:54 projector kernel: [<c1001023>] do_fast_syscall_32+0x8b/0xf6
Apr 1 10:21:54 projector kernel: [<c16cf1ab>] sysenter_past_esp+0x40/0x6a
Apr 1 10:21:54 projector kernel: Mem-Info:
Apr 1 10:21:54 projector kernel: active_anon:354 inactive_anon:1532 isolated_anon:0\x0a active_file:773116 inactive_file:1064369 isolated_file:0\x0a unevictable:0 dirty:0 writeback:0 unstable:0\x0a slab_reclaimable:28383 slab_unreclaimable:13352\x0a mapped:1283 shmem:234 pagetables:134 bounce:0\x0a free:12096 free_pcp:528 free_cma:0
Apr 1 10:21:54 projector kernel: Node 0 active_anon:1416kB inactive_anon:6128kB active_file:3092464kB inactive_file:4257476kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:5132kB dirty:0kB writeback:0kB shmem:936kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
Apr 1 10:21:54 projector kernel: DMA free:3212kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15976kB managed:15360kB mlocked:0kB slab_reclaimable:92kB slab_unreclaimable:1648kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Apr 1 10:21:54 projector kernel: lowmem_reserve[]: 0 793 8021 8021
Apr 1 10:21:54 projector kernel: Normal free:5148kB min:3572kB low:4464kB high:5356kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:20kB unevictable:0kB writepending:0kB present:894968kB managed:813356kB mlocked:0kB slab_reclaimable:113440kB slab_unreclaimable:51760kB kernel_stack:1264kB pagetables:0kB bounce:0kB free_pcp:852kB local_pcp:144kB free_cma:0kB
Apr 1 10:21:54 projector kernel: lowmem_reserve[]: 0 0 57825 57825
Apr 1 10:21:54 projector kernel: HighMem free:40024kB min:512kB low:8644kB high:16776kB active_anon:1416kB inactive_anon:6128kB active_file:3092464kB inactive_file:4257456kB unevictable:0kB writepending:0kB present:7401664kB managed:7401664kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:536kB bounce:0kB free_pcp:1260kB local_pcp:648kB free_cma:0kB
Apr 1 10:21:54 projector kernel: lowmem_reserve[]: 0 0 0 0
Apr 1 10:21:54 projector kernel: DMA: 11*4kB (UE) 6*8kB (UE) 3*16kB (E) 6*32kB (UE) 1*64kB (U) 4*128kB (UE) 3*256kB (UE) 1*512kB (U) 1*1024kB (E) 0*2048kB 0*4096kB = 3212kB
Apr 1 10:21:54 projector kernel: Normal: 897*4kB (ME) 195*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5148kB
Apr 1 10:21:54 projector kernel: HighMem: 2744*4kB (UM) 2351*8kB (UM) 69*16kB (UM) 19*32kB (UM) 4*64kB (UM) 3*128kB (UM) 3*256kB (UM) 2*512kB (U) 4*1024kB (M) 1*2048kB (M) 0*4096kB = 40072kB
Apr 1 10:21:54 projector kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 1 10:21:54 projector kernel: 1837719 total pagecache pages
Apr 1 10:21:54 projector kernel: 0 pages in swap cache
Apr 1 10:21:54 projector kernel: Swap cache stats: add 462, delete 462, find 9/9
Apr 1 10:21:54 projector kernel: Free swap = 979624kB
Apr 1 10:21:54 projector kernel: Total swap = 979836kB
Apr 1 10:21:54 projector kernel: 2078152 pages RAM
Apr 1 10:21:54 projector kernel: 1850416 pages HighMem/MovableOnly
Apr 1 10:21:54 projector kernel: 20557 pages reserved
Apr 1 10:21:54 projector kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Apr 1 10:21:54 projector kernel: [ 1695] 0 1695 2740 730 6 3 0 -1000 systemd-udevd
Apr 1 10:21:54 projector kernel: [ 2771] 0 2771 1703 103 7 3 0 0 syslog-ng
Apr 1 10:21:54 projector kernel: [ 2772] 0 2772 6470 731 10 3 0 0 syslog-ng
Apr 1 10:21:54 projector kernel: [ 2829] 0 2829 613 39 5 3 0 0 rpcbind
Apr 1 10:21:54 projector kernel: [ 2855] 0 2855 730 454 5 3 0 0 rpc.statd
Apr 1 10:21:54 projector kernel: [ 2905] 0 2905 692 45 5 3 0 0 rpc.idmapd
Apr 1 10:21:54 projector kernel: [ 2936] 0 2936 817 123 5 3 0 0 rpc.mountd
Apr 1 10:21:54 projector kernel: [ 3153] 0 3153 1098 128 7 3 0 0 smartd
Apr 1 10:21:54 projector kernel: [ 3183] 0 3183 1355 580 6 3 0 -1000 sshd
Apr 1 10:21:54 projector kernel: [ 3208] 0 3208 600 452 5 3 0 0 cron
Apr 1 10:21:54 projector kernel: [ 3239] 0 3239 1096 378 6 3 0 0 agetty
Apr 1 10:21:54 projector kernel: [ 3240] 0 3240 1096 351 6 3 0 0 agetty
Apr 1 10:21:54 projector kernel: [ 3241] 0 3241 1096 370 6 3 0 0 agetty
Apr 1 10:21:54 projector kernel: [ 3242] 0 3242 1096 359 6 3 0 0 agetty
Apr 1 10:21:54 projector kernel: [ 3243] 0 3243 1096 351 6 3 0 0 agetty
Apr 1 10:21:54 projector kernel: [ 3244] 0 3244 1096 349 5 3 0 0 agetty
Apr 1 10:21:54 projector kernel: [29916] 1000 29916 925 741 4 3 0 0 bash
Apr 1 10:21:54 projector kernel: [29976] 1000 29976 755 578 5 3 0 0 su
Apr 1 10:21:54 projector kernel: [29979] 0 29979 925 761 6 3 0 0 bash
Apr 1 10:21:54 projector kernel: [29984] 0 29984 563 184 5 3 0 0 dmesg
Apr 1 10:21:54 projector kernel: Out of memory: Kill process 29916 (bash) score 0 or sacrifice child
Apr 1 10:21:54 projector kernel: Killed process 29976 (su) total-vm:3020kB, anon-rss:304kB, file-rss:2008kB, shmem-rss:0kB
Apr 1 10:21:54 projector kernel: oom_reaper: reaped process 29984 (dmesg), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Apr 1 10:22:21 projector sshd[29988]: SSH: Server;Ltype: Version;Remote: 192.168.0.104-15575;Protocol: 2.0;Client: PuTTY_Release_0.66
Apr 1 10:22:21 projector sshd[29988]: SSH: Server;Ltype: Kex;Remote: 192.168.0.104-15575;Enc: aes256-ctr;MAC: hmac-sha2-256;Comp: none [preauth]
Apr 1 10:22:26 projector sshd[29988]: Accepted keyboard-interactive/pam for bigun from 192.168.0.104 port 15575 ssh2
Apr 1 10:22:26 projector sshd[29988]: pam_unix(sshd:session): session opened for user bigun by (uid=0)
Apr 1 10:22:26 projector sshd[29994]: SSH: Server;Ltype: Kex;Remote: 192.168.0.104-15575;Enc: aes256-ctr;MAC: hmac-sha2-256;Comp: none
Apr 1 10:23:45 projector su[30003]: Successful su for root by bigun
Apr 1 10:23:45 projector su[30003]: + /dev/pts/0 bigun:root
Apr 1 10:23:45 projector su[30003]: pam_unix(su:session): session opened for user root by bigun(uid=1000)

_________________
"It's ok, they might have guns but we have flowers." - Perpetual Victim

cboldt
Veteran
Veteran

Joined: 24 Aug 2005
Posts: 1046

Posted: Sat Apr 01, 2017 4:10 pm Post subject:

One purported fix (found at unix.stackexchange.com ... I think linked above)

Code:

sync && echo 1 > /proc/sys/vm/drop_caches

Here is another shot in the dark ...

Code:

root@hypoid-2 [3] 26 /root # zgrep SL[AU]B /proc/config.gz
CONFIG_SLAB=y
# CONFIG_SLUB is not set
# CONFIG_SLAB_FREELIST_RANDOM is not set
CONFIG_SLABINFO=y
# CONFIG_DEBUG_SLAB is not set

I'm not getting close to full RAM use, so no swapping going on here - just saying, I'm not pushing up against the RAM limit, so don't know if those settings "work" to make swap happen, or not. The mention of SLAB/SLUB is in https://lkml.org/lkml/2016/12/12/49, you may have to follow some of the links around.

Edit to add a couple more remarks and references.

The preference to resolve this is SLUB, not SLAB. In other words, the settings on my system would not tend to resolve oom-killer acting before the kernel resorts to swap.

The "sync && echo 1 > /proc/sys/vm/drop_caches" has to be in a cronjob, not a one-shot. See https://bugzilla.redhat.com/show_bug.cgi?id=1373339

tholin
Apprentice

Joined: 04 Oct 2008
Posts: 207

Posted: Sat Apr 01, 2017 7:36 pm Post subject:

Bigun wrote:

So it's happening again:

You have plenty of available (easily freeable) ram so the OOM condition is probably a bug. You are not locking huge amounts of page cache right?
There have been a lot of problems with memory management and early OOM with recent kernels. You use 4.9.6 which is an old 4.9 version. Try 4.9.20. Here are some patches for fixing OOM conditions that went in 4.9.7. No idea if that's your problem though.

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?id=96e5cec10e7a75c931f8993633b3a5cedc99144e
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?id=ade7afe9dca6b13919f88abd38eefe32f22eaeb3
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?id=d1656c5aef4d72f03a7833d07a378c8f604b8307

In general I would advice against running any kernel that hasn't been "stable" for at least 6 months. Unless you really need some of the changes in 4.9 try downgrading to 4.4.59.

cboldt wrote:

sync && echo 1 > /proc/sys/vm/drop_caches

That's a horribly ugly workaround. Better to find what cause the problem than doing things like that.

Roman_Gruber
Advocate

Joined: 03 Oct 2006
Posts: 3846
Location: Austro Bavaria

Posted: Sat Apr 01, 2017 9:21 pm Post subject:

4.4 branch is also buggy. I think it depends on the platform and the use case. I had another topic htis week i responded where the 4.4. branch was the culprit for the hole topic. I update my kernel on a weekly basis. And i stay on 4.9 becasue 4.10 is not supported by the binary nvidia drivers afaik.

Bigun
Advocate

Joined: 21 Sep 2003
Posts: 2198

Posted: Sun Apr 02, 2017 9:37 am Post subject:

tholin wrote:

Bigun wrote:

So it's happening again:

...You are not locking huge amounts of page cache right?

I'm not sure what you mean by locking, but here is my current cache usage:

Code:

$ free
total used free shared buff/cache available
Mem: 8230380 425608 21528 776 7783244 7508572
Swap: 979836 0 979836

_________________
"It's ok, they might have guns but we have flowers." - Perpetual Victim

Bigun
Advocate

Joined: 21 Sep 2003
Posts: 2198

Posted: Sun Apr 02, 2017 9:40 am Post subject:

Now this is interesting.

Code:

[ 1774.540540] kworker/dying (1072) used greatest stack depth: 5424 bytes left
[54758.518469] kworker/dying (3320) used greatest stack depth: 5416 bytes left
[66456.307781] kworker/dying (30573) used greatest stack depth: 5296 bytes left

_________________
"It's ok, they might have guns but we have flowers." - Perpetual Victim

Bigun
Advocate

Joined: 21 Sep 2003
Posts: 2198

Posted: Sun Apr 02, 2017 9:49 am Post subject:

Roman_Gruber wrote:

4.4 branch is also buggy. I think it depends on the platform and the use case. I had another topic htis week i responded where the 4.4. branch was the culprit for the hole topic. I update my kernel on a weekly basis. And i stay on 4.9 becasue 4.10 is not supported by the binary nvidia drivers afaik.

I usually update to a newer kernel when it gets marked as stable anyway, upgrading now. We'll see if this works.

cboldt wrote:

One purported fix (found at unix.stackexchange.com ... I think linked above)

Code:

sync && echo 1 > /proc/sys/vm/drop_caches

...

Going to try this next. I want to pinpoint why it's happening.
_________________
"It's ok, they might have guns but we have flowers." - Perpetual Victim

NeddySeagoon
Administrator

Joined: 05 Jul 2003
Posts: 54830
Location: 56N 3W

Posted: Sun Apr 02, 2017 10:10 am Post subject:

Bigun,

Code:

sync && echo 1 > /proc/sys/vm/drop_caches

if that works, you have a kernel bug.
The kernel is supposed to do that for itself by way of normal operation.

sync flushes dirty buffers to the disc. The filesystem ensures that there is nothing more than a few seconds old in dirty buffers anyway.
drop_caches, flushes clean buffers, which the kernel is supposed to do when it needs RAM.
However, drop_caches, drops all caches, which will have a performance impact. The kernel mechanism only drops what's needed.

As has been said, its a horrible hack.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

Bigun
Advocate

Joined: 21 Sep 2003
Posts: 2198

Posted: Thu Apr 06, 2017 1:57 pm Post subject:

I'm starting to think it was a kernel bug. After upgrading to 4.9.16, I've noticed caching doesn't completely consume my RAM, and the SWAP is actually being used. Four days uptime, when I get to 14, I'll consider the issue resolved, but I'm starting to guess that was the issue. _________________ "It's ok, they might have guns but we have flowers." - Perpetual Victim

Bigun
Advocate

Joined: 21 Sep 2003
Posts: 2198

Posted: Mon Apr 17, 2017 4:11 pm Post subject:

Consider it solved. Upgrading to the new kernel version seems to have fixed the issue. _________________ "It's ok, they might have guns but we have flowers." - Perpetual Victim

Carnildo
Guru
Guru

Joined: 17 Jun 2004
Posts: 595

Posted: Mon Apr 17, 2017 10:56 pm Post subject:

Bigun wrote:

I'm starting to think it was a kernel bug. After upgrading to 4.9.16, I've noticed caching doesn't completely consume my RAM, and the SWAP is actually being used. Four days uptime, when I get to 14, I'll consider the issue resolved, but I'm starting to guess that was the issue.

It almost certainly was: https://bugzilla.kernel.org/show_bug.cgi?id=190351 and https://bugzilla.redhat.com/show_bug.cgi?id=1401012

Display posts from previous:

	Gentoo Forums Forum Index Kernel & Hardware	All times are GMT Goto page Previous 1, 2
Page 2 of 2

Jump to:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Copyright 2001-2025 Gentoo Foundation, Inc. Designed by Kyle Manna © 2003; Style derived from original subSilver theme. | Hosting by Gossamer Threads Inc. © | Powered by phpBB 2.0.23-gentoo-p11 © 2001, 2002 phpBB Group
Privacy Policy