Ultra-heavy swap-like disk thrashing - System unusable

[xenon]

Yet another time I am trying to resolve an issue that has been cursing me since I first installed Gentoo (also my first Linux) about six years ago.

A brief description: suddenly, or in occasion of trivial actions like opening a new tab in Firefox, the system becomes completely unusable because of an incredibly intense disk activity. Such activity manages to takeover all resources, to the point where turning Caps-Lock on takes several seconds if not minutes when inside an X enivironment.

First, what the problem is not:

A hardware problem. In these six years, I went through a lot of hardware, with at least three setups that had nothing in common with each other and several "incremental updates".

A performance problem: Currently I run a modern dual-core AMD CPU, 4 GB of memory and, to add insult to injury, a WD Raptor disk where the OS is installed.

Now, what could be the cause of the problem? It's not easy to diagnose a system that's unusable, but I've been able to collect some information (especially now that I discovered iotop):

• CPU activity is generally low during these episodes.
  
• Disk I/O is not incredibly high in terms of throughput, commonly ranging between 4 and 10 MB/s
  
• It seems most if not all of the activity is reading from the /usr directory (I can tell because it's on its own partition)
  
• iotop shows the kswapd0 process on top, but without any actual disk read or write activity. Instead, it seems reading activity is evenly spread between the running applications.
  
• These episodes can last several hours. The last one before the one happening right now lasted at least eight hours, after which I gave up and powered off the computer.
  
• The swap partition doesn't seem to be particularly involved in the issue. Right now, after about one hour of thrashing, I got 1381 of 2062 MB of swap unused.
  
• The disk activity itself wouldn't justify a completely unusable system: it seems to me the kernel is saying "I'm working, nobody move or do anything".
  
• Speaking of kernels, I went through so many versions in these years that it seems hard to blame a specific bug. It's always been gentoo-sources, from 2003 up to now.
  
• Obviously, lsof shows lots of open files in /usr, but since it's just natural for applications to have open files there, I don't think I can get much out of that.
  
  I blamed KDE, I blamed Firefox, I blamed memory-hogging applications like Azureus, I blamed X and then xorg, but I have always been proven wrong by facts. Currently I blame the kernel, but... where do I go from here?
  
  So, that's it. I'm frustrated, I'm angry and I don't know who or what to blame, nor what to do to fix this issue. I am more than willing to provide detailed information, to perform any sort of tests, to answer any questions. As soon as I regain control of my computer, I will update this post with my .config options.
  
  Please, help.

[xenon]

Here we go. I managed to kill X (and with it all applications) and all went back to normal. One more thing I should mention is that the last time I checked the output of free -m, all memory was used (as in common with Linux), and all but a tiny fraction was buffers, not cache. Before, it was more evenly ditributed. Could it mean something?

Here's my .config file. I know it could take a diet, but I do not think that's the problem. Again, please ask, comment, suggest. Each and every bit of help is welcome. Thanks.

[Tekeli Li]

.

From atop of my head, pun intended, I suggest you to try running atop, and switch to disk usage view (shift+d, iirc). During those episodes, it should show you which process is hogging the disc. Be sure you shorten the report interval which is 10 secs by default, might be too much. There is also the /proc/sys/vm/block_dump to view blocks read/written, but be careful with that.

[Carnildo]

[xenon]

[zyko]

[xenon]

Thank you for your post. And I agree with it. There's something wrong, I don't simply have too many applications open. This said:

- i have tried not using a swap file in the past, thinking I was simply experiencing massive swapping (although it can't last hours). These episodes still happened. Also, the swap partition doesn't seem really involved in this phenomenon, even though it obviously comes to mind. Last time it happened it was more than 70% free.

- I don't think there's a single application to blame. I say this both out of the experience that the phenomenon happened with different app combinations and the fact (shown to me yesterday for the first time by iotop) that every open application starts reading the /usr directory. Certainly some app may be hogging too much memory because of poor coding, but it wouldn't explain the whole thing, just facilitate it, I guess.

- I will try changing the sceduler, but I think these episodes already happened before such option even existed. But don't quote me on this. Also, it's hard to troubleshoot this issue since it happens at relatively distant intervals and I'm not really able to reproduce it at will. But I won't give up this time!

[HeissFuss]

It really sounds like your issue is due to swapping.

Here's the issue:
Some app or process comes along and consumes a large amount of memory. This causes pages of memory to be paged out to swap. The pages that are being kicked out are not from the process which is consuming the memory. They are from pages which haven't been accessed in a while (probably things like firefox tabs). When you quit that app, memory is freed back up. However, the pages from firefox which are in swap stay in swap. When you go to that firefox tab (perhaps hours later) it need to page those chunks of memory back in from swap, thus the slowness.

You should figure out what is actually consuming the memory. It could be a leak in firefox or a javascript page in one of the tabs, or it could be some app that just consumes a large amount. Keep in mind that shared memory is also something that can force pages out but only shows up in buffer/cache. The same is true for tmpfs filesystems. If you're storing something in a tmpfs, that may be the problem.

If you do see enough available memory, you can force swap to clear by running swapoff on that partition, and then 'swapon -a' after it finishes. That'll force the pages from swap back into memory and then remount swap afterward.

[xenon]

Thanks for your post. Here's why I don't think the issue is swap-related:

- The problem occurred even when I didn't have a swap partition at all.

- The disk activity is on the /usr partition, which I believe means shared objects and not application data.

- Swap activity, even when massive, doesn't occur at relatively low data transfer rates and doesn't last for hours.

[x22]

It seems like some kind of memory pressure.
- kswapd0 is working hard
- many different processes read from disk

When memory shortage occurs system will be paging anonymous memory from/to swap and file mapped memory from/to normal files. This includes most binaries and libraries many of which are in /usr.

Check /proc/meminfo and /proc/zoneinfo. And also dmesg and /var/log/messages for memory allocation failures, out of memory reports or anything unusual.

[MageSlayer]

Not quite sure about your symptoms, but take a look at
https://forums.gentoo.org/viewtopic-p-5419223.html#5419223

[xenon]

Thanks for your posts guys, but:

- @MageSlayer, that was an interesting read, but I think this is a software issue, not a hardware one. I think (if my assumptions are correct) it's just natural for a disk drive to crawl to a grinding halt when a dozen processes try to read their .so files with kernel-like priority, resulting in a sort of starvation.

- @x22, as I already said, this doesn't seem really swap-related, since it also happened without a swap partition. Certainly it's a memory management issue, but despite the slightly misleading topic title, it's not about moving data. Disk activity is basically read-only during the episodes.
At least once I found interesting traces in dmesg, something about the kernel having to kill a process because there was no memory left. While not optimal, killing a single app seems much better than paralyzing the system for hours. Nevertheless, other times I couldn't find anything of interest in there. What should I look for in meminfo and zoneinfo? Do you have any suggestions?

Considering the current costs of flash memory, I could put my /usr partition on a compact flash and probably work around this issue, but I find it crazy to need to recur to such tricks to get a usable Linux desktop. I would like to submit this to kernel people, do you think they'd listen?

[HeissFuss]

[xenon]

The point of the Compact Flash was that due to the nature of flash memory, I think the concurrent reads wouldn't render the system unusable (for hours). I understand it would be just a workaround, and a slightly sad one.

It seems the discussion is boiling down to either:

- I should find and abandon the application that is filling the memory.

or

- I should stop abusing and overloading my system.

Honestly, I find it hard to completely blame this problem to myself and/or to the application level. I mean, an OS can not allow eight hours of disk grinding and an unusable system because physical memory is full while the swap partition is 70% free. By the way, is it correct to say that only application data can be swapped out, not shared objects and the like?

[Carnildo]

Anything except the portions of the kernel relating to memory layout and swap can (theoretically) be swapped out. File-backed memory objects (memory-mapped files, shared objects, and the like) are usually discarded from memory rather than swapped out, because they're already present on disk.

Replacing your hard drive with a solid-state disk might hide the problem (solid-state disks are faster than spinning disks, but still much slower than RAM), but it won't fix it.