AMD64 system slow/unresponsive during disk access (Part 2)

tallica · Posted: Tue Nov 23, 2010 8:52 am Post subject:

Looks like there are some problems with patch v4: http://lkml.org/lkml/2010/11/21/41
_________________
Gentoo ~AMD64 | Audacious

kernelOfTruth · Posted: Fri Dec 03, 2010 6:36 pm Post subject:

if you're rsyncing regularly and apps are suffering during heavy reads, try the following:

[PATCH v3 0/3] f/madivse(DONTNEED) support
http://marc.info/?l=linux-kernel&m=129104424110018&w=2
http://marc.info/?l=linux-kernel&m=129104424210023&w=2
http://marc.info/?l=linux-kernel&m=129104424210027&w=2

there are several other potential improvements in this area but the most if not all, that are applicable to 2.6.36 to the current date are incorporated
already in the zen-kernel so give it a try:

git.zen-kernel.org

www.zen-kernel.org

dm crypt: scale to multiple CPUs
might be useful if you need it - in my experience it increases latency probably a little

there also seems to be some potential filesystem corruption being triggered with ext4 and 2.6.37-rc* right now which is under investigation so don't use it with 2.6.37-rc* - yet
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004

pste · Tux's lil' helper Joined: 14 Dec 2004 Posts: 106

I think this is a bug, an inconsistency, an unfortunate system design or something similar that has been around for quite a while, which makes me very annoyed, not least because it pokes a hole in the hallmark of linux - speed and stability! - and it seems never to get fixed!?!? Unfortunately I'm in no position (not timewise, nor knowledgewise) to pursue this problem myself, which leaves me with no other option than to provide background, some additional information and experiences and hope that someone clever can get something out of it and accept the challenge!

Quick summary:
Probably since kernel 2.6.18 (googling seems to converge on this kernel) there's been some kind of problem with I/O (disk I/O, ...??) on 64-bit kernels, resulting in high cpu load and a lagging or even hanging system. It seems like the problem begins when I/O reaches a certain amount, like copying many large files (taking a backup...) or doing many things concurrently, like copying files while rsync'ing through a vpn tunnel (high cpu load in general). It seems like it has something to do with the I/O scheduler, but I experience problems even if I use the "simplest" deadline scheduler. To me, zen-sources also works slightly better that gentoo-sources.

It's not hardware problems, because it works fine with windows (which is kind of extra annoying, isn't it?), and my impression (haven't tested thoroughly though - but my home servers, both with gentoo-sources-32bit with several usb drives on usb hubs seems to work flawlessly) is that there's no problem with 32-bit kernels. The external usb drives I'm using are of different brands and models, all have the same behavior. I've tried -a lot- of different kernel settings, including many minimal ones, but I cannot find a pattern.

What happens?
Personally I think it's connected with usb and usb-harddrives (it's at least a failsafe way to make the problem show!). Every time I make a backup, copying goes at 20MB/s plus for a while, but then it starts to slow down. I cannot say exactly when, but either when copied size reaches - say 8GB (just a guess - sometimes more), or when the file count is big! (no number...). Then transfer speed falls down to a few MB/s (2-5MB/s perhaps), system start to lag and cpu is 100%. Often (not always) I get something about "reset usb-device" in the log, sometimes "I/O-error on device" forcing me to shut-down drives and computer, start over, and fsck. Occasionally I get a complete system lock-up (my comp. freezes and all leds on keyb shine).

Making backups between two usb-drives on the same usb-hub, seems to create most problems, including total system hang-ups

Hopefully, someone that thinks this is something that is required to be fixed can make someting out of this start digging - I'm cheering loudly in that case!

Good luck!

/pste

idella4 · Posted: Tue Mar 01, 2011 9:48 am Post subject:

pste,

very general. use top, iotop, and setup the conditions that it occurs. You can at least capture some snapshots of the system state with the tops. Then dmesg content. Need some sort of baseline.
_________________
idella4@aus

frostschutz · Advocate Joined: 22 Feb 2005 Posts: 2977 Location: Germany

ppurka · Advocate Joined: 26 Dec 2004 Posts: 3256

See https://forums.gentoo.org/viewtopic-t-793263.html
_________________
emerge --quiet redefined | E17 vids: I, II | Now using kde5 | e is unstable :-/

tomk · Posted: Tue Mar 01, 2011 10:40 am Post subject:

Merged from here.
_________________
Search | Read | Answer | Report | Strip

pste · Tux's lil' helper Joined: 14 Dec 2004 Posts: 106

ppurka - yes, that (this) thread is one of the sources that made me say that this is a since long problem! Google gives you more...

frostschutz - yes I know that the setup with usb-drives is a hardware bottleneck, but I do believe that it should not mean anything else than that the backup takes a long time, it should -not- make the entire system lag, or hang! I do think this is caused by some kind of race condition that occur in 64-bit kernels... And, NO! it's not the hardware, I wrote above that the same setup works fine in windows and (similar setups) with 32-bit kernels! - but yes, hardware related, meaning (kernel) driver problems, perhaps? I do agree that overheating is a possible explanation for the I/O-error situations, but I find it strange that this differ between OS:es (or kernel types). Furthermore, the problem does also occur without the hub (e.g. copying from system drive to usb drive or between usb drives on different usb ports of the comp.), I stated the example because my impression is that it's the quickest way to create problems...

idella4 - sure, I'll try to capture something, although not today... (I need to recompile the kernel with a few new flags - the iotop emerge told me, but I need my comp running a while longer...) But a problem is that for the worst case (the most interesting one) I must try to create one of these total lock-ups and then hand-copy (or photograph) the tops and dmesg screens precisely because the system is frozen, and it feels a little risky to recover the filesystem(s) everytime it hangs... A concrete example (for anyone to try): try starting a rsync -avh --progress /home /media/your-usb-drive/ (or similar) and wait (of course, /home must be many GB large!). I'm doing precisely this at the moment! For me this command keeps showing about 15-20MB/s for every file. But after a while the system gets lagged (rsync keeps running at the same speed though), then if I for instance try starting a movie in vlc (having to read a big file from the harddrive) - this is (naturally) really slow, but sometimes the movie hangs, and closing vlc takes about 5 minutes! When sync is finished, system is back to normal responsivess...

Thanks for the response!

/pste

joeklow · n00b Joined: 23 Jan 2011 Posts: 46

Reporting 2.6.36 ck-sources running at multicore Phenom II.

Recompiled this kernel, changing deprecated SATA support ("ATA/ATAPI support") over to new (serial ATA/PATA drivers).
I/O scheduler: BFQ (was CFQ)
Profile: Desktop (was Server)
CPU scheduler: CFS+autogroups
Timer: 1000 (was 200)

Also, /etc/init.d/local.start has the following to disable cache (stupid XFS loves to flush data once in hour, and it would be stupid to let the flushed data stay in cache).

Yamakuzure · Posted: Wed Mar 02, 2011 4:32 pm Post subject:

Huh? I haven't had any lag since gentoo-sources-2.6.36-rsomething and with gentoo-sources-2.6.37 (okay, with cgroups hack) I have no lag even if I do a huge parallel merge (load between 25 and 40 on an i7 Dualcore laptop with HT) and have VMWare with WindowsXP open.

Is it just this cgroups stuff? I am basically using what is described here: https://forums.gentoo.org/viewtopic-t-852922.html
(And no, I do not have any problems with Amarok, DragonPlayer or any other multi media stuff)
_________________
Edited 220,176 times by Yamakuzure

devsk · Posted: Tue Mar 29, 2011 7:19 am Post subject:

2.6.38 with AUTOGROUP helps a lot with this issue.

devsk · Posted: Tue Apr 19, 2011 6:34 am Post subject:

Any news on this front? Does AUTOGROUP help people with this issue? Or this is a non-issue now?

kernelOfTruth · Posted: Tue Apr 19, 2011 6:26 pm Post subject:

TimeManx · n00b Joined: 11 Jul 2011 Posts: 55

I've configured 3.1.6 with autogroup, cfq, zram (128 MB as swap), zcache, transparent huge pages (madvise), memory compaction, preemptile kernel on a system with 2 GB of RAM. The system is quite responsive in the first half hour after boot but the performance keeps deteriorating.
Also, copying large amounts of data from one drive to another is sporadic and the drives become inaccesible during that time which causes dolphin to freeze.

TinheadNed · Posted: Thu Jan 05, 2012 8:45 pm Post subject:

From the changelog of the 3.2 kernel: "desktop reponsiveness in presence of heavy writes has been improved"

Holysword · l33t Joined: 19 Nov 2006 Posts: 946 Location: Greece

Is this problem still up? I'm suffering from unresponsiveness very often; even stupid facebook flash games can bring my i7 with 4GB down - and let's say, I've got an infinite swap.

I've tried to blame the kernel (was zen-sources-something, can't remember but it doesn't have more than 2 weeks that I've updated it), but gentoo-sources also freezes/slows down. Was using BFQ+BFS, and now I'm with CFQ+CFS; same. SLUB to SLAB? Same. Remarking that BFQ is incompatible with cgroups, one would see that I was not using cgroups initially. Now I am using cgroups+autogroups. Nothing. My main system lies in a ReiserFS3 partition, not ext4 though.

I wouldn't claim this is annoying; this is being dangerous for me, since at least once a day my system crashes hopelessly and I have to hard-reboot it. Have lost a couple of files so far. I was considering a depclean + emerge -e world but now I am wondering if this is worth to try. 4 months ago I didn't have this problem (was using zen-sources back then) and some of you seem to have been having this for years...
_________________
"Nolite arbitrari quia venerim mittere pacem in terram non veni pacem mittere sed gladium" (Yeshua Ha Mashiach)

depontius · Advocate Joined: 05 May 2004 Posts: 3526

Are you running /home from a network drive?

My performance problems were from /home being mounted on nfsv4, and were related to firefox and its sqlite sync() behavior. A year or two back I moved .mozilla and .thunderbird to local disk, then symlinked the nfs-mounted .mozilla and .thunderbird directories to the local ones. Problem gone.

Some time after moving that system to 3.2.x I saw the notice of improved responsiveness, and tried moving .mozilla back to nfs. My performance problems came back, though they didn't seem quite as bad. The other night I moved .mozilla back to local disk.

Other than that, I'm happy and even with that I wasn't having problems with crashing. Have you tried memtest86+?
_________________
.sigs waste space and bandwidth

Holysword · l33t Joined: 19 Nov 2006 Posts: 946 Location: Greece

Holysword · l33t Joined: 19 Nov 2006 Posts: 946 Location: Greece

Okay folks... it seems it was both my kernel (it was zen-sources with BFS+BFQ), as seen in this thread, and a memory leak problem with my wm (as seen here).
_________________
"Nolite arbitrari quia venerim mittere pacem in terram non veni pacem mittere sed gladium" (Yeshua Ha Mashiach)

xman1 · n00b Joined: 11 Apr 2004 Posts: 58

Has this been solved yet? I had these same issues and it turned out my Western Digital hard drive has a bug with APM. Pop into PM-utils default config and set APM to 255 to disable it and all works well now.

You can also do this with hdparm:

smlbstcbr · n00b Joined: 08 Apr 2006 Posts: 51

Bump. I still have those issues in 3.3.8-gentoo.

kernelOfTruth · Posted: Wed Jul 04, 2012 10:37 am Post subject:

in total we're trading in some throughput for interactivity & responsiveness:

you guys having the problems could give following tweaks a try:

smlbstcbr · n00b Joined: 08 Apr 2006 Posts: 51

I'll see how that works in my machine. How unfortunate to have such issues in the Gentoo Kernel. It seems to me that it has slowed since the change to 3.XX kernels.

kernelOfTruth · Posted: Sat Jul 07, 2012 11:39 am Post subject:

smlbstcbr · n00b Joined: 08 Apr 2006 Posts: 51

Well, I'm trying your solution (thank you for posting them). There's a slight improvement. Not as smooth as it used to be.
EDIT: I have been using a value of 200 for the last parameter and my system has improved significantly, though there's still some lag when swapping windows or opening some documents.