| View previous topic :: View next topic |
| Author |
Message |
neuron
Advocate
Joined: 28 May 2002
Posts: 2371
|
 Posted: Sun Aug 10, 2008 11:10 am Post subject: hardlock on oom |
 |
|
With recent kernels (2.6.26, gentoo) and swap on an encrypted loopback device I'm getting hard locks oom.
I can reproduce it by building boost, this is the last thing I got from top before it died completely (magic key doesn't even work):
| Code: |
Mem: 2056660k total, 2042868k used, 13792k free, 268k buffers
Swap: 2023992k total, 789352k used, 1234640k free, 12032k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8313 portage 39 19 373m 190m 6256 D 39 9.5 0:04.75 cc1plus
8345 portage 39 19 334m 267m 4144 D 33 13.3 0:03.49 cc1plus
8322 portage 39 19 369m 174m 6260 D 32 8.7 0:04.60 cc1plus
8335 portage 39 19 372m 348m 6264 D 26 17.4 0:04.71 cc1plus
8331 portage 39 19 373m 125m 6264 D 20 6.2 0:04.77 cc1plus
5636 root 15 -5 0 0 0 D 14 0.0 0:00.46 kcryptd
5626 root 0 -20 0 0 0 R 11 0.0 0:00.32 loop0
|
Any suggestion on how to fix this? My system has failed gracefully on going oom before, but now it does horribly, and I'm not entirely sure where the problem lies. |
|
| Back to top |
|
 |
manaka
Apprentice
Joined: 23 Jul 2007
Posts: 178
Location: Spain
|
| Posted: Mon Aug 11, 2008 6:23 pm Post subject: |
|
|
oom-killer should normally resolve the situation by killing one or more processes.
Maybe it's some kind of kernel deadlock. See http://article.gmane.org/gmane.linux.kernel.device-mapper.dm-crypt/682/match=deadlock+swap+loop+aes
for a similar problem.
Having a separate swap device would be less prone to that kind of problems.
Increasing /proc/sys/vm/min_free_kbytes could also be of help.
_________________
Javier Miqueleiz
"Listen to your heart. It knows all things, because it came from the Soul of the World, and it will one day return there." |
|
| Back to top |
|
|
neuron
Advocate
Joined: 28 May 2002
Posts: 2371
|
| Posted: Tue Aug 12, 2008 7:48 am Post subject: |
|
|
I've done a bit more testing, disabling swap and oom-killer kicks in and solves it no problem, but with encrypted swap it'll hardlock, unencrypted swap works fine.
So it's either something in my loop device/the encryption itself that's causing problems. |
|
| Back to top |
|
|
manaka
Apprentice
Joined: 23 Jul 2007
Posts: 178
Location: Spain
|
| Posted: Tue Aug 12, 2008 4:49 pm Post subject: |
|
|
Seems a deadlock.
Did you try increasing min_free_kbytes? It should help (at least in theory)...
_________________
Javier Miqueleiz
"Listen to your heart. It knows all things, because it came from the Soul of the World, and it will one day return there." |
|
| Back to top |
|
|
neuron
Advocate
Joined: 28 May 2002
Posts: 2371
|
| Posted: Tue Aug 12, 2008 5:03 pm Post subject: |
|
|
| manaka wrote: | Seems a deadlock.
Did you try increasing min_free_kbytes? It should help (at least in theory)... |
Haven't had time to test it yet, but it's in my notes and I will give it a try when I get the time  |
|
| Back to top |
|
|
zeek
Guru
Joined: 16 Nov 2002
Posts: 480
Location: Bantayan Island
|
| Posted: Thu Aug 14, 2008 3:14 am Post subject: Re: hardlock on oom |
|
|
| neuron wrote: | With recent kernels (2.6.26, gentoo) and swap on an encrypted loopback device I'm getting hard locks oom.
I can reproduce it by building boost, this is the last thing I got from top before it died completely (magic key doesn't even work):
| Code: |
Mem: 2056660k total, 2042868k used, 13792k free, 268k buffers
Swap: 2023992k total, 789352k used, 1234640k free, 12032k cached
|
|
It didn't run out of memory, it went into swap thrashing. You have over 1GB of swap still available. |
|
| Back to top |
|
|
neuron
Advocate
Joined: 28 May 2002
Posts: 2371
|
| Posted: Thu Aug 14, 2008 7:52 am Post subject: Re: hardlock on oom |
|
|
| zeek wrote: | | neuron wrote: | With recent kernels (2.6.26, gentoo) and swap on an encrypted loopback device I'm getting hard locks oom.
I can reproduce it by building boost, this is the last thing I got from top before it died completely (magic key doesn't even work):
| Code: |
Mem: 2056660k total, 2042868k used, 13792k free, 268k buffers
Swap: 2023992k total, 789352k used, 1234640k free, 12032k cached
|
|
It didn't run out of memory, it went into swap thrashing. You have over 1GB of swap still available. |
I'm well aware of that, but that swap trashing with encryption enabled does cause a hardlock. |
|
| Back to top |
|
|
|
Kernel & Hardware
