Keep running out of memory (solved)

[Bigun]

Let me preface this whole post with this statement: This has only become a problem in the last few months, before that, this system has ran like a top for the last 5 years or so. With that, I'll move on.

For whatever reason, after about 7 to 18 days, the machine will start running out of memory and start killing off random processes to make RAM. From what I can tell, swap is hardly even being touched.

[Roman_Gruber]

my opinion

your kernel is outdated. 4.9.16 is stable for amd64.
+ experimental software like systemd + xfs

[Bigun]

[Bigun]

Also, this box is running a 32 bit kernel.

[Roman_Gruber]

Well i Know its very off topic

https://www.cnet.com/products/gigabyte-ga-h61m-ds2h-2-1-motherboard-micro-atx-lga1155-socket-h61/specs/

Any reason why you run 32bit on 64bit capable box?

It may be that you run in a not very well tested issue, bug. I assue most of the guys these days use amd64. Running the platform with more users may cause less issues in the long run => amd64, ext4, openrc / eudev

--

Only thing to ask is what you do with your server? Check regularly with top to see what causes this issue

Some software like google chrome have a memory hole since years ...

[Bigun]

[Leio]

I'd use something like https://github.com/pixelb/ps_mem to see where the memory goes and what is swapping (the -S option to it) things out (suggesting potential memory leaks or at least unused memory for a long while). I believe the ps_mem.py file can be used indepdently without dealing with the whole repo.
_________________
GNOME team lead; GStreamer; MIPS/ARM64

[Bigun]

[cboldt]

I doubt this issue is the age of your kernel. 4.9 is plenty recent.

I see no reference in your dmesg that /dev/md125 is being used for swap. This could be a kernel setting or an fstab issue

[Bigun]

[cboldt]

Yes, swapon will show what is available. I get similar report here, with `swapon` and `swapon -s`. And in your case that command does refer to the device /dev/md125.

But your kernel boot log does not refer to the device at all, while my kernel boot log does, and reports adding it.

Could be just a difference in logging. My swap is a normal partition. Something is resisting paging out to SWAP, in favor of dropping information from RAM.

I don't know enough off the top of my head to say whether our kernel config differences re:SWAP have a role.

[cboldt]

What do you get on `stat /dev/md125`

For comparison purposes, here are the relevant specifications on my swap space, /dev/sda5

[cboldt]

For anybody curious, the (8 Gb) size of my swap space is only because /dev/sda5 on this machine was created with an eye to using it for hibernate space, if I ever get around to working on that project. Other machines under my watchful eye have much less swap space, and swap space is barely touched on any machine, even after hundreds of days of uptime.

Edit to add, just to be the first to jump on "useless use of cat" -- you can grep the contents of .config directly
`grep SWAP /usr/src/linux/.config` works fine, and skips a pipeline.

[cboldt]

You aren't the only one, but this supposedly was fixed for 4.9 kernel

http://unix.stackexchange.com/questions/300106/why-is-the-oom-killer-killing-processes-when-swap-is-hardly-used

[NeddySeagoon]

Bigun,

swap can't be used for everything. Its only used for dynamically allocated RAM.
If something in RAM has a permanent home on your HDD, it won't be swapped to swap.
Dirty buffers will be written, so that the RAM they use can be freed.
Code or other read only data will be dropped. Next time its needed, you get a page fault and its reloaded form HDD.
It is swapping - just not using the swap space, which is why you should always have some swap.

When the kernel needs to invoke the OOM manager. Its done all of the above and has nowhere to go.
Its kill something or panic. That it choose sshd is unfortunate.

The usual cause in a memory leak, however, memory leaks come from dynamically allocated RAM that is not deallocated.
If turn, that means you can usually watch swap fill up before the OOM manager kicks in.

Your log shows

[Bigun]

[cboldt]

New toys! I never looked into the oom killer, but it is configurable. You can protect sshd if you want ...

http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html

In short, /proc/PID/oom_adj is available, and takes values from -16 (preserved) to +15 (vulnerable), and even recognizes a value of -17, which allows the protected app to commit suicide by bringing down the kernel.

[Bigun]

[Bigun]

[cboldt]

[Bigun]

[cboldt]

I suspect compaction is "naturally" turned on by other selections, or kernel default, but maybe not. In my case, it is turned on by other configuration options. It (compaction) has the -*- indicator next to it. It can't be toggled directly.

The kernel has tons of "weird" options and pitfalls. I chased "why does my system show a loadavg close to 1.0 with no CPU cycles?" for a few days, and found quite by accident that the loadavg calculation is greatly affected by high precision timers. I bet there are literally hundreds of interactions like that, and yours with oom-killer. It's tough enough to get the average computer user into a working system without scaring the crap out of them with so many issues and options.

[Bigun]

[NeddySeagoon]

Bigun,

You have the choice of forcing COMPACTION on by choosing

[cboldt]

The kernel default is CONFIG_COMPACTION=y (just tested with `make defconfig`)
Some kernel `make` options turn comapaction off. `make tinyconfig` results in "# CONFIG_COMPACTION is not set"

I don't know what genkernel does, or what the "Gentoo helpers" toggle does vs. COMPACTION

I think Bigun has a good point. If COMPACTION isn't the most likely outcome due to other causes (like genkernel, or start with default kernel, of "on" via the Gentoo switch, etc.), then a mention in the kernel config part of the handbook ought to be created.