Extreme I/O without swap - What exactly is kswapd0 doing?

[xenon]

Preamble: this issue has intermittently plagued me for years, across kernel versions, mainboards, hard disks and every other hardware or software change and upgrade.

The issue is kswapd0, the kernel virtual memory handler, completely grinding my system to a halt for several minutes. The system becomes so unresponsive that even pressing Caps-Lock takes several seconds for the light on the keyboard to turn on. While it's quite hard to even switch to a terminal and log in, I managed to run top and check that CPU usage remains normal, while vmstat shows more interesting things. It showed 20 MB of free memory for a few lines, then in the span of about ten lines, it went from 20 MB to about 180, to about 300, down to 200 again, to 150 and then up to 270 and finally to about 300. At this point, the system became usable and I stopped vmstat. Everything was snappy and responsive again, the tray clock finally caught up and it was as if nothing ever happened. The accident I just described happened when I closed Akregator (an obvious resources hog, I know), and I had lowered kernel swappiness to 20. About an hour earlier, it was apparently triggered by closing Azureus (another resources hog, I know!) and again by restarting it.

Now I have two questions:

- I understand release of resources may cause a temporarily decrease of responsiveness, but 5 to 10 minutes of paralysis purely because of incredibly intense I/O, while the rest of the system is alive and well and the CPU is almost idle and able to handle running processes is just crazy. Before you ask, DMA is active and I'm using a WD Raptor, so this is NOT a performance issue. This much, I'm sure of.

- What exactly is kswapd0 doing, since I currently have NO swap partition at all? Why is it grinding the hard disk anyway and what does freeing memory has to do with the disk since we are talking about real RAM only?

I thought I'd never see this happen again, but today it occurred two times and I think it's time to understand it and fix it. Thanks in advance for any info.

[cyrillic]

[MostAwesomeDude]

SLAB or SLUB?
_________________
Don't believe the "n00b" under my name.

[xenon]

[cyrillic]

Maybe some process is generating a ton of errors (or other useful output), and the harddrive is grinding away while filling up /var/log/*

[MostAwesomeDude]

I was asking about which slab allocator you were using, SLAB or SLUB. (You're using SLAB.) SLUB can improve performance occasionally.

[xenon]

Thanks for your replies.

@cyrillic: no, I don't think so, and an inspection of /var/log doesn't show anything unusual.

@MAD: I agree I should recreate .config from scratch, but it is going to be hard to troubleshoot the issue that way. The problem is so intermittent (but still crazy and unnerving) that false positives or negatives are almost assured. I think I'll do it anyway, but any further information, hints and suggestions will be more than welcome.

[lonrot_m]

i had this problem some years ago but i was messing with the kernel to see what everything did, i fix it reconfiguring my kernel from scratch.
_________________
The only fool bigger than the person who knows it all is the person who argues with him.

[xenon]

Well, I finally reconfigured my kernel from scratch, too. We'll see if it ever happens again.

[xenon]

So yesterday I went to bed with the disk thrashing like mad, and it hadn't stopped when I fell asleep. Today it happened again and I was pretty pissed off. Almost by chance, I ran fsview and noticed that the .xsession-errors file was almost 4 GB! cyrillic had suggested to check for oversized log files, but I checked in the obvious places and not in my home. I can't be sure yet, but it seems it could be the culprit.
For now, my solution is

[Jaglover]

Hey, how comes your Xorg has so many errors? You better fix the cause, not the consequence.

[xenon]

For obvious reasons, I did not even try to open the log file. Empirically, I remeber critical situations when Azureus crashed and when Konqueror had to handle complex and/or poorly coded Web pages, also consider that most Flash objects cause GTK errors in Konqueror. Firefox and Opera are better browsers, I know, but in my experience they get sluggish after a while and don't like having many tabs open at the same time. At least here, Konqueror performs better. Maybe this issue is correlated to the other one... we'll see.

I know your approach is the correct one, but considering that Xorg works fine and that the consequences are terrible, I'm working on the latter first.

[Jaglover]

Such hard disk grinding as you describe because of X errors, I'd say some process is running in loop or something, it is going to eat resources, you better investigate it.

[pappy_mcfae]

Absolutely. Not only that, the harder you work the hard drive, the closer it comes to death. While doing intense disk operations once in a while is a good thing, for a computer to get caught in a loop that includes extended intense disk operation is just asking for your hard drive to go to its early grave.

And if I may ask, how much RAM do you have, and why don't you use swap? Just curious.

Blessed be!
Pappy
_________________
This space left intentionally blank, except for these ASCII symbols.

[xenon]

Hm. I know you guys are right. So let's see... My usual desktop population is Konqueror with many tabs open, Akregator, often Azureus and sometimes aMule. Now none of these applications is too nice on resources, I know. As I wrote above, Konqueror spews lots of outputs and errors, but it's obivious that 4 GB is not "a lot", it's HUGE and doesn't make sense. Now consider that I only found out about this today (the original topic title is about kswap0, as you can see). The applications I use are mainstream and have millions of users, and the fact that this behavior happened to me for years, across reinstalls and many hardware changes, led me to think that there wasn't much I could do about it on an application level. And yes, I know frequent grinding kills disks, that's one of the reasons why I am determined to fix this problem. I have 1280 MB of RAM and I have no swap because in a previous attempt of fixing this, I thought the thrashing was due to massive de-swapping stuff back to RAM and disabled it. I never missed swap after that, so I lived without it. Anyway, I'd be more than happy to get some help on the "bunch of errors" side of the issue, so if you guys have suggestions or need some info and testings, please speak. Do you think I should temporarily restore the .xsession-errors file and see what it gets filled with?

[Jaglover]

Sure, you didn't keep the 4 GB one? Running tail on it would have shown what's up. I remember I tossed my Debian years ago because famd went berserk on NFS mounts and this supercow apt-get refused to uninstall it.

[MostAwesomeDude]

[xenon]

Ok, currently I got a 1.4 GB file, here's a tail of it:

[MostAwesomeDude]

Looks like you've got some type of KDE app running in the background, churning out errors. Is it possible to try logging in with another WM, or to just start killing KDE apps until you find the one dumping the errors?
_________________
Don't believe the "n00b" under my name.

[xenon]

I will try that. In addition, I also get these quite often (probably every time I visit a https web page:

[i92guboj]

[xenon]

I ran revdep-rebuild, and it only found a gparted broken library (no problem, I downgraded parted because of HAL issues). Then I ran revdep-rebuild -L /usr/lib/libcrypto.so.0.9.8 and it literally found dozens of items; now it's willing to re-emerge all sorts of packages, from mysql to amarok to php and ruby to kdebase and network (as one would expect ). By the way, shouldn't revdep-rebuild -L whatever.so always produce a subset of the parameter-less command results? Am I supposed to run revdep-rebuild on every single library in my system?

At this point, I would like to be sure that the original problem is indeed caused by the rapid growth of .xsession-errors in certain circumstances; I'll work on this.

[xenon]

Current .xsession-errors is about 1.4 MB, not GB. It seems I'll have to wait some more for the huge growth to happen before drawing conclusions.