| View previous topic :: View next topic |
| Author |
Message |
lagalopex
Guru
Joined: 16 Oct 2004
Posts: 564
|
 Posted: Sat Jun 23, 2007 5:56 pm Post subject: |
 |
|
Btw, tried anybody a different io-scheduler?
| Code: | | for i in /sys/block/*/queue/scheduler ; do echo "anticipatory" > $i ; done |
as mentioned in kernel-bug #8636 that was also posted in our kernel bug #7372...
And its proven to be userland independend?
Last edited by lagalopex on Sat Jun 23, 2007 5:58 pm; edited 1 time in total |
|
| Back to top |
|
 |
moesasji
Apprentice
Joined: 10 May 2005
Posts: 263
|
| Posted: Sat Jun 23, 2007 5:58 pm Post subject: |
|
|
It's my feeling that we're just running around in circles and getting nowhere.
Basically the same suggestions have started repeating itself...
I myself have tested setting the qeue-depth to 1 already in the kernel bug, see comment 88 and 89 Note that it was not the first time the suggestion appeared in that bug, see for example comment #29 and the response of devsk to it.
I personally think it is most benificial if somebody could confirm that also in their case the bad commit I found it the origin of the problem. See post 42 in the kernel bugreport. Nobody has yet confirmed this.....Checking the bad commit is just a single reboot. You have to look at:
git checkout master
git revert <bad-commit-id>
to figure out how to do it. See: isolate-bugs-with-bisect
edit) in response to the previous poster: most people here have tested with all different schedulers. In my specific case changing the scheduler does not make a difference. The problem remains. |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 2998
Location: Bay Area, CA
|
| Posted: Sat Jun 23, 2007 6:55 pm Post subject: |
|
|
| moesasji wrote: | It's my feeling that we're just running around in circles and getting nowhere.
Basically the same suggestions have started repeating itself... |
I think you are right. But it is bound to happen when new folks hit this issue and the issue drags for so long that people forget what all has been tried. I actually booted into amd64 to try it...feeling silly now... 
| moesasji wrote: |
I personally think it is most benificial if somebody could confirm that also in their case the bad commit I found it the origin of the problem. See post 42 in the kernel bugreport. Nobody has yet confirmed this.....Checking the bad commit is just a single reboot. |
I actually wanted to do this but I am tied up with reiser4. I will need to have a clean vanilla setup to test it out, which means backup, wipe the partition and restore. I will probably take it on sometime next week. |
|
| Back to top |
|
|
ttuegel
Apprentice
Joined: 18 Jan 2005
Posts: 176
Location: Illinois, USA
|
| Posted: Sat Jun 23, 2007 7:35 pm Post subject: |
|
|
Just wanted to add my $0.02 about the NCQ thing:
The other day, my SATA drive died. My Gentoo installation is on my PATA drive, so I only lost all my files, but I can still boot into Gentoo. However, the problem remains, so unless the mere presence of the sata_nv kernel driver is enough to bugger up performance, I think I can rule it out (at least on my system).
Since hardware specs seem to be all the rage:
| Code: | Athlon 64 X2 4400+
MSI K8N Neo4 Platinum (nForce 4 Ultra)
2 GB DDR 400 RAM
1x Maxtor 250GB PATA
1x Samsung 500GB SATA (replacement for the old drive)
GeForce 6800 GT |
Just a note that my previous SATA drive was a 320GB Seagate Barracuda. Reading some users' experiences with AHCI vs legacy mode for SATA controllers in the BIOS, I checked the same on my motherboard, only to discover that my model doesn't support AHCI in the first place! Needless to say, the problem of DMA conflicts between drivers isn't the culprit here...
As far as CFLAGS and such, I've encountered this issue on a whole range of 'em, but right now I'm at:
| Code: | CFLAGS="-O2 -march=athlon64 -pipe -msse3"
LDFLAGS="-Wl,-O1 -Wl,--as-needed" |
on ~amd64. Still as stumped as I was when I first encountered this issue some time in August 2006... |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 2998
Location: Bay Area, CA
|
| Posted: Sat Jun 23, 2007 8:13 pm Post subject: |
|
|
| devsk wrote: | | moesasji wrote: | It's my feeling that we're just running around in circles and getting nowhere.
Basically the same suggestions have started repeating itself... |
I think you are right. But it is bound to happen when new folks hit this issue and the issue drags for so long that people forget what all has been tried. I actually booted into amd64 to try it...feeling silly now...
| moesasji wrote: |
I personally think it is most benificial if somebody could confirm that also in their case the bad commit I found it the origin of the problem. See post 42 in the kernel bugreport. Nobody has yet confirmed this.....Checking the bad commit is just a single reboot. |
I actually wanted to do this but I am tied up with reiser4. I will need to have a clean vanilla setup to test it out, which means backup, wipe the partition and restore. I will probably take it on sometime next week. |
Did I ever confirm that gentoo-sources-2.6.17-r7(with only reiser4 patch) don't exhibit the problems we are facing?
Well, if I didn't, I can now because I just booted into this old kernel and I can confirm that the problem is not there.
@moesasji, so it seems like we are at least facing the bug. |
|
| Back to top |
|
|
moesasji
Apprentice
Joined: 10 May 2005
Posts: 263
|
| Posted: Sun Jun 24, 2007 4:40 pm Post subject: |
|
|
| moesasji wrote: | | I myself have tested setting the qeue-depth to 1 already in the kernel bug, see comment 88 and 89 Note that it was not the first time the suggestion appeared in that bug, see for example comment #29 and the response of devsk to it. |
Based on the outcome of a new git-bisect run (yes I was bored enough) I have to withdraw this earlier conclusion. In my case NCQ clearly does play a role in the problem. Below is what I just added to the kernel-bug report on it:
| moesasji wrote: | The outcome of the new git-bisect attempt is slightly embarrassing....(blush)
---
12fad3f965830d71f6454f02b2af002a64cec4d3 is first bad commit
[PATCH] ahci: implement NCQ suppport
Implement NCQ support.
---
So I tried again to enable/disable NCQ as suggested earlier, but this time for
the vanilla 2.6.18 kernel as this is the kernel that most clearly shows the
problem in my case. And indeed enabling/disabling NCQ has a clear effect for
2.6.18!
**with echo 1 > /sys/block/sda/device/queue_depth it takes ~7s to open a
terminal while simultaneously calculating a checksum.
** with echo 31 > /sys/block/sda/device/queue_depth it takes >40s to open a
terminal under the exact same conditions.
So conclusion: NCQ plays an important role for the problem I have, BUT those 7s
are still much longer then under my 32bit environment. When doing exactly the
same test on the 32bit kernel(2.6.21) the terminal opens in 1-2s, so
substantially faster then the 7s I get with NCQ disabled!!
I would say that those 7s are still too long....
|
So even though disabling NCQ gives a very clear improvement the time it takes to open a terminal stays quite long....much too long for my taste. |
|
| Back to top |
|
|
lagalopex
Guru
Joined: 16 Oct 2004
Posts: 564
|
| Posted: Sun Jun 24, 2007 5:06 pm Post subject: |
|
|
| On the lkml they (Michael Tokarev) ran some tests and it seems to be _very_ harddisk dependend, wether ncq improves the performance or hits it very hard... link |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 2998
Location: Bay Area, CA
|
| Posted: Sun Jun 24, 2007 5:43 pm Post subject: |
|
|
| moesasji wrote: | | So even though disabling NCQ gives a very clear improvement the time it takes to open a terminal stays quite long....much too long for my taste. |
Here are my disk specs:
2x WD raptors with DMRAID - No NCQ on these. I am not sure TCQ works. This is where my '/' sits.
1x 300G maxtor sata - NCQ queue length can't be changed from 31 to 1 : permission denied.
1x 500GB WD sata - NCQ q length can't be changed from 31 to 1: permission denied.
1x 200GB IDE - No NCQ.
Am I supposed to chmod that queue_length file in /sys/block writable? is that even a good idea? How do the sata drives (and NCQ enabled on them) affect the freezes which involve only the IDE drive and network txr of 2GB file i.e. read from /dev/hda5 and write over cifs? It sure seems like couple of problems getting mixed in here. |
|
| Back to top |
|
|
moesasji
Apprentice
Joined: 10 May 2005
Posts: 263
|
| Posted: Sun Jun 24, 2007 6:40 pm Post subject: |
|
|
| devsk wrote: | | How do the sata drives (and NCQ enabled on them) affect the freezes which involve only the IDE drive and network txr of 2GB file i.e. read from /dev/hda5 and write over cifs? It sure seems like couple of problems getting mixed in here. |
It could still be that we are experiencing the same problem. I just confirmed that with NCQ disabled on my system the 2.6.17.14 kernel has a better response than the 2.6.18 kernel. So clearly there is a bug independent of NCQ.
It could just be that enabling the NCQ on a drive interacts in some way with the bug itself and because the delay due to enabling the NCQ is so much longer/more obvious it is clear that running the git-bisect will give the patch that enabled it on my system. And that is exactly what has happened!! (remember that 2.6.17.14 has queue_depth set to 1, while in 2.6.18 it is 31)
So bottomline is that the git-bisect I ran (probably) points in the wrong direction....so I'm now going to spend/waist another evening running one with NCQ disabled (I start to enjoy bootscreens....) Hopefully I will get something usefull for the devs to look at.
Last edited by moesasji on Sun Jun 24, 2007 10:57 pm; edited 1 time in total |
|
| Back to top |
|
|
likewhoa
l33t
Joined: 04 Oct 2006
Posts: 778
Location: Brooklyn, New York
|
| Posted: Sun Jun 24, 2007 7:31 pm Post subject: |
|
|
seems 2.6.17 is not hit by this, good to know and will confirm this if 2.6.17 has supports for my current setup.
I see people are trying different I/O Schedulers I myself made a small bash function to let me switch between schedulers.
| Code: |
ios() {
case $1 in
cfq|deadline|anticipatory)
echo "switching scheduler to $1"
while read line; do echo $1 >$line; done < <(find /sys/block/ -type f -name 'scheduler')
if (($?>0)); then echo "error occurred.";fi
;;
-l|--list) echo "listing IO Scheduler(s) in use"
find /sys/block/ -type f -name 'scheduler' -exec cat {} \;
;;
*) echo "one argument is needed you noob"
;;
esac
}
|
I don't think the problem is directly related to the SATA_NV drivers are others are running VIA,SILICON,PROMISE and others SATA drivers and still experiencing the lockups. I hope this week i'll have time to test out different kernel setups so that I can pin point a possible conflict. |
|
| Back to top |
|
|
mirekm
Apprentice
Joined: 12 Feb 2004
Posts: 210
Location: Gliwice
|
| Posted: Sun Jun 24, 2007 7:42 pm Post subject: |
|
|
| I think you have to look into begin of this thread, that is started on 2.6.16. So very probably that 2.6.17 is affected too. |
|
| Back to top |
|
|
likewhoa
l33t
Joined: 04 Oct 2006
Posts: 778
Location: Brooklyn, New York
|
| Posted: Sun Jun 24, 2007 10:59 pm Post subject: |
|
|
| mirekm wrote: | | I think you have to look into begin of this thread, that is started on 2.6.16. So very probably that 2.6.17 is affected too. |
yes i remember but not according to devsk
| devsk wrote: |
Did I ever confirm that gentoo-sources-2.6.17-r7(with only reiser4 patch) don't exhibit the problems we are facing?
|
and to respond to the above,.. no devsk you haven't  |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 2998
Location: Bay Area, CA
|
| Posted: Sun Jun 24, 2007 11:28 pm Post subject: |
|
|
I didn't reply because I had already said that. The only kernel where I don't have this problem is 2.6.17-r7 (this is oldest I have) on amd64 and any kernel on x86. May be there are multiple problems we are looking at and some existed before 2.6.17 and were fixed during 2.6.17 series.
CONFIG_ATA wasn't there in 2.6.17-r7 i.e. SATA was supported thru SCSI layer and I had CONFIG_SCSI_SATA_NV=y in .config in 2.6.17 instead if CONFIG_SATA_NV=y in 2.6.[18|19|20|21]. |
|
| Back to top |
|
|
moesasji
Apprentice
Joined: 10 May 2005
Posts: 263
|
| Posted: Mon Jun 25, 2007 10:52 am Post subject: |
|
|
1) I've been trying to narrow down a problem area with NCQ disabled yesterday-evening, but did not (yet) get very far as my previous testing method is to rough to clearly distinguish between good/bad. Even though the response for 2.6.18 is worse the for 2.6.7.14 the difference is subtle.
A recipy suggested in the kernel-bug report is:
| Code: |
mount -o loop $iso /tmp/iso
sudo /tmp/cold # fairly typical drop_caches / BLKFLSBUF script
cp -a /tmp/iso /tmp/foo-slow &
sleep 1 ; time ls /usr
|
Does anybody know how to do the drop_caches part for Gentoo as that is what makes testing difficult??
This dropping of the cache is very relevant as I would need to run the test a couple of time to eliminate statistical fluctuations to some extend.
**) the git-tester script suggested earlier in this thread does not really work for the same reason.
2) I'm no longer sure that 2.6.17.14 is the last good kernel.....the fact that NCQ plays such a big role could be very misleading here. If you are not able to disable NCQ this is something to think about. |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 2998
Location: Bay Area, CA
|
| Posted: Mon Jun 25, 2007 2:56 pm Post subject: |
|
|
| moesasji wrote: | Does anybody know how to do the drop_caches part for Gentoo as that is what makes testing difficult??
|
To drop all (which is preferred in this case) caches:
| Code: | | echo 3 > /proc/sys/vm/drop_caches |
|
|
| Back to top |
|
|
engineermdr
Apprentice
Joined: 08 Nov 2003
Posts: 297
Location: Altoona, WI, USA
|
| Posted: Mon Jun 25, 2007 10:03 pm Post subject: |
|
|
All,
I've switched to a 32-bit system and reported that initially it appeared better. Even still, by the numbers, my iowaits and thoughput remain excellent. But I'm experiencing the same problem. Today, I was running mkisofs to generate a DVD data image. I clicked on my konsole button and waited, and waited. Click it again, and waited and waited 10s of seconds. I suspended my mkisofs and presto, two new konsoles instantly appeared. I'm not so sure this is limited to amd64. While it's certainly better running i686, I don't think it's gone completely.
I'm using 2.6.20-gentoo-r8. I think I'll try some of the older kernels when I get some time. I'm not using any special kernel modules or patches keeping me at any particular version. |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 2998
Location: Bay Area, CA
|
| Posted: Tue Jun 26, 2007 7:21 am Post subject: |
|
|
| mdr wrote: | All,
I've switched to a 32-bit system and reported that initially it appeared better. Even still, by the numbers, my iowaits and thoughput remain excellent. But I'm experiencing the same problem. Today, I was running mkisofs to generate a DVD data image. I clicked on my konsole button and waited, and waited. Click it again, and waited and waited 10s of seconds. I suspended my mkisofs and presto, two new konsoles instantly appeared. I'm not so sure this is limited to amd64. While it's certainly better running i686, I don't think it's gone completely.
I'm using 2.6.20-gentoo-r8. I think I'll try some of the older kernels when I get some time. I'm not using any special kernel modules or patches keeping me at any particular version. |
mkisofs is writing and konsole wants to read, and your scheduler is being fair. deadline scheduler should alleviate this problem. Its gives 10:1 advantage to read over writes as far as scheduling deadline (500ms vs. 5000ms for writes) is concerned. Try different values for read_expire, write_expire and writes_starved (default 2, try 5).
CFQ will try to be fair but mkisofs is a b/w hog, your reads won't complete unless you suspend or complete the write operation. Or you could ionice mkisofs and see it will help.
So, this is unrelated (somewhat). |
|
| Back to top |
|
|
moesasji
Apprentice
Joined: 10 May 2005
Posts: 263
|
| Posted: Tue Jun 26, 2007 8:27 am Post subject: |
|
|
| devsk wrote: | | mkisofs is writing and konsole wants to read, and your scheduler is being fair. deadline scheduler should alleviate this problem. |
I've been testing schedulers on my system (outcome reported to kernel-devs)...deadline scheduler is the only one that actually behaves decently on my system. But indeed I'm calculating a checksum of an ISO, so not creating one.
The test: I calculate the checksum for a DVD-iso and while at 40% into the checksum I try to open a terminal and
measure how much seconds it takes. (between each run I drop caches and change scheduler)
Here comes the statistics:
1) The bad 2.6.18 kernel with NCQ enabled
- cfq: >43s,>43s,>45s (after this time the checksum is done)
- anticipatory: 16s, 13s, 14s
- deadline: 5s, 3s, 5s
2) The bad 2.6.18 kernel with NCQ disabled
- cfq: 15s, 5s, 7s, 8s
- anticipatory: 13s, 13s, 13s
- deadline: 2s, 2s, 4s, 2s
3) The supposedly good 2.6.17.14 kernel with NCQ disabled (by definition)
- cfq: 11s, 12s, 12s
- anticipatory: 14s, 14s, 15s
- deadline: 2s, 2s, 2s
4) For a rough comparison a 32-bit 2.6.21 kernel
- cfq: 2s, 2s
- anticipatory: 3s, 2s
- deadline: 2s, 2s
**) Note that the 32-bit kernel is located on a different HD then the iso-image
and the 64-bit kernels, so it might not be a completely fair comparison.
Conclusion
- the cfq and anticipatory schedular give a pretty bad response under all tests
with the 64-bit kernels, while it works fine with a 32-bit kernel.
- enabling NCQ seems to make the response worse. In particular the cfq
scheduler takes a serious hit.
- Note that with NCQ disabled the 2.6.17.14 kernel behaves just as bad as the 2.6.18 kernel!!!
Todo
I now have to get the 32-bit system installed on the same HD and see if that does behave as expected or also shows the problem.
ps) Thanks for giving me the hint on drop_caches devsk!! 
That was pretty essential for doing this test. |
|
| Back to top |
|
|
gringo
Advocate
Joined: 27 Apr 2003
Posts: 3793
|
| Posted: Tue Jun 26, 2007 10:22 am Post subject: |
|
|
i experienced the same troubles on my amd64 box till 2.6.19 ( can´t remember exactly what version though) and have been following this thread for a while, but it disappeared since then and i cant reproduce this anymore. My guess is that this strongly depends on the chip/hd combination, specially sata2 drives using NCQ ( mine are raptors attached to a sata_sil chip).
If you want me to run some tests, share numbers or whatever to track down the problem let me know, but i´m not sure this will help as i have no ncq available.
cheers
_________________
Error: Failing not supported by current locale |
|
| Back to top |
|
|
likewhoa
l33t
Joined: 04 Oct 2006
Posts: 778
Location: Brooklyn, New York
|
| Posted: Tue Jun 26, 2007 1:45 pm Post subject: |
|
|
| mdr wrote: | All,
I've switched to a 32-bit system and reported that initially it appeared better. Even still, by the numbers, my iowaits and thoughput remain excellent. But I'm experiencing the same problem. Today, I was running mkisofs to generate a DVD data image. I clicked on my konsole button and waited, and waited. Click it again, and waited and waited 10s of seconds. I suspended my mkisofs and presto, two new konsoles instantly appeared. I'm not so sure this is limited to amd64. While it's certainly better running i686, I don't think it's gone completely.
I'm using 2.6.20-gentoo-r8. I think I'll try some of the older kernels when I get some time. I'm not using any special kernel modules or patches keeping me at any particular version. |
running mkisofs is about the only time my system locks up too, even using deadline but it seemed to help with lower nice levels. |
|
| Back to top |
|
|
engineermdr
Apprentice
Joined: 08 Nov 2003
Posts: 297
Location: Altoona, WI, USA
|
| Posted: Tue Jun 26, 2007 1:58 pm Post subject: |
|
|
| Quote: | | CFQ will try to be fair but mkisofs is a b/w hog, your reads won't complete unless you suspend or complete the write operation. Or you could ionice mkisofs and see it will help. |
Whoa!  This has been my problem all along and now you're saying it's expected?!? I'll bet this is what most people are calling "the problem". If this is the expected behavoir of CFQ, then CFQ is completely useless in my mind. If I've got 2 jobs doing i/o, I expect them to share equally. How is it fair that one takes over the machine?
Yes, I'm definitely going to try the other schedulers tonight when I get home. |
|
| Back to top |
|
|
neuron
Advocate
Joined: 28 May 2002
Posts: 2371
|
| Posted: Tue Jun 26, 2007 2:10 pm Post subject: |
|
|
| mdr wrote: | | Quote: | | CFQ will try to be fair but mkisofs is a b/w hog, your reads won't complete unless you suspend or complete the write operation. Or you could ionice mkisofs and see it will help. |
Whoa! This has been my problem all along and now you're saying it's expected?!? I'll bet this is what most people are calling "the problem". If this is the expected behavoir of CFQ, then CFQ is completely useless in my mind. If I've got 2 jobs doing i/o, I expect them to share equally. How is it fair that one takes over the machine?
Yes, I'm definitely going to try the other schedulers tonight when I get home. |
absolutely not, CFQ should cause some overhead, but not make the operation go from taking 2 seconds to 40 seconds. If the problem was 1 second difference then I'd consider blaming the scheduler. |
|
| Back to top |
|
|
likewhoa
l33t
Joined: 04 Oct 2006
Posts: 778
Location: Brooklyn, New York
|
| Posted: Tue Jun 26, 2007 3:28 pm Post subject: |
|
|
I updated my ios function to allow editing of the fifo_batch,front_merges,read_expire,write_expire,writes_starved values for block devices.
| Code: |
ios() {
case $1 in
cfq|deadline|anticipatory)
echo "switching scheduler to $1"
while read line; do echo $1 >$line; done < <(find /sys/block/ -type f -name 'scheduler')
if (($?>0)); then echo "error occurred.";fi
;;
-l|--list) echo "listing IO Scheduler(s) in use"
find /sys/block/ -type f -name 'scheduler' -exec cat {} \;
;;
fifo_batch|front_merges|read_expire|write_expire|writes_starved)
if (($#<2)); then
echo "not enough arguments giving, try again."
else
echo "changing $1 value to $2."
while read line; do echo $2 >$line; done < <(find /sys/block/ -type f -name "$1")
fi
if (($?>0)); then echo "error occurred.";fi
;;
*) echo "atleast one argument is needed. try again."
;;
esac
}
|
|
|
| Back to top |
|
|
ohadbasan
n00b
Joined: 16 May 2007
Posts: 34
|
| Posted: Wed Jun 27, 2007 9:11 pm Post subject: forgive me for the noobness |
|
|
but i find this thread a bit hard to understand.
this thread started by someone who tells that cdrom access halts his system.
and later on it developed to IO problems and thoughts about problematic sata drivers.
I'm experiencing the problem of the first kind.
the cdrom access makes my computer completely laggy
when i shut down HAL the system functions again normally.
it's an dfi lanparty ut ultra-d nfroce4 ultra athlon64 dualcore 3800+
and a gentoo-soruces 2.6.20-r8 kernel.
can someone please simply the solution for me?
this thread looks to me like chinese.
thanks for your patience! |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 2998
Location: Bay Area, CA
|
| Posted: Wed Jun 27, 2007 9:44 pm Post subject: |
|
|
HAL related slowdowns are already documented elsewhere. My personal suggestion: don't use that stuff. Its moronic [windows like moronic] and is hated by kernel devs. They won't entertain any CDROM or DVD related bug if you have that thing running.
I don't see this thread ever was about HAL related slowdowns. OP clearly started this thread about slowdowns when disk is heavily used. |
|
| Back to top |
|
|
|
Gentoo on AMD64
