View previous topic :: View next topic |
Author |
Message |
madbiker Guru
Joined: 12 Oct 2003 Posts: 439 Location: Victoria, BC, Canada
|
Posted: Mon Feb 07, 2005 10:23 pm Post subject: Scheduler questions. |
|
|
I searched the forums, but didn't really see much that really explained things well.
So from what I've read, there are I/O and CPU schedulers, and there are a few major choices for each. Different kernels include different schedulers in them, and different ones work better for different purposes.
Using ck sources, I compiled in support for all 3 schedulers that I was given the option for. I can change which one I'm using by passing elevator=deadline or elevator=cfq or whichever one I want in my grub.conf. So these are I/O schedulers correct? Is there any way to determine which one I'm currently using? How about CPU schedulers? How do I know which one I'm using, how do I change?
Finally, are there generally accepted standards for schedulers for desktop use and server use? I imagine some of them are going to be considered "unstable" or in "testing" phases, but while being unstable also provide better desktop performance (much like ck, love and nitro kernel sources)?
Thanks for any light you can shed on the subject. |
|
Back to top |
|
|
choco n00b
Joined: 10 Jun 2003 Posts: 49 Location: Jackson, MI
|
Posted: Tue Feb 08, 2005 1:37 am Post subject: |
|
|
I'm interested in learning more about the various schedulers too. http://kerneltrap.org has been an informative site for this sort of thing, although it's so technical (it refers directly to the email lists of kernel developers).
I'm not as sure about CPU schedulers, but you can tell which I/O scheduler is in use by looking at the kernel boot log. Doing a should give you that. The general guideline I've heard is that anticipatory I/O or deadline is good for servers and databases because of better throughput, while cfq is good for the desktop because no program is able to slow the system down by monopolizing I/O.
I don't believe the CPU scheduler can be chosen by a boot parameter - it requires a patch. I don't know what CPU schedulers are available/in development. A selling point of the Gentoo kernel prior to 2.6 was that it had the O(1) scheduler. I don't know if any special CPU scheduler patch goes into the main Gentoo kernel now. _________________ Who would win in a fight between Sandman and Clayface? |
|
Back to top |
|
|
commonloon Tux's lil' helper
Joined: 13 Apr 2004 Posts: 133 Location: Marin, CA
|
Posted: Tue Feb 08, 2005 5:33 am Post subject: a little more on io schedulers |
|
|
You can find some documentation in /usr/src/linux/Documentation/block where linux is the symlink to your kernel version.
2.6 has the as (default - anticipatory), cfq (completely fair), deadline (good for databases) and noop (good for devices such as usb drives).
You can find out your scheduler via dmesg as described above or in /sys/block/sdb/queue/scheduler on _some_ newer (2.6.10?) or patched kernels. On these kernels you may change it w/o rebooting by echo cfq > /sys/block/sdb/queue/scheduler (sdb is the device in question) for example -- auto-magically you'll then notice it will look like:
porky queue # cat scheduler
noop anticipatory deadline [cfq]
You can adjust io scheduler parameters by adjusting things in: /sys/block/sda/queue/iosched where sda is the device your interested in changing. The parameters and files within the iosched dir vary -- see the respective documentation.... seems more options (and more complex) that elvtune for 2.4. You can change via standard echo value > iosched/param
Hope this helps... _________________ Calling fly fishing a hobbie is like calling brain surgery a job. |
|
Back to top |
|
|
madbiker Guru
Joined: 12 Oct 2003 Posts: 439 Location: Victoria, BC, Canada
|
Posted: Fri Feb 11, 2005 2:36 am Post subject: |
|
|
Can't believe I missed this reply.
Yes that helps a lot! I wasn't getting any mention of which scheduler I was using with dmesg, but I'm using a 2.6.10 and had that scheduler bit in my /sys. Works great.
Now my only questions relate to CPU schedulers. I'm using ck-sources, so does that mean I'm using the 'staircase' scheduler automagically? Is it simply determined by the kernel patchset you're using, and there's no way to change it other than go to a patchset that's patched with a different scheduler? |
|
Back to top |
|
|
steve_d555 Guru
Joined: 07 Nov 2004 Posts: 458 Location: Belmont, Massachusetts
|
Posted: Fri Feb 11, 2005 5:36 am Post subject: |
|
|
I have a question, I hear that Deadline/Anticipatory is better for gaming as it allows it to take up 100%. Does anyone have any thoughts? Has anyone done any benchmarks against CFQ or something? I would be really interested to know, because ive been using Nitro sources with CFQ and was wondering what was better.
Thanks,
--Steve |
|
Back to top |
|
|
commonloon Tux's lil' helper
Joined: 13 Apr 2004 Posts: 133 Location: Marin, CA
|
Posted: Fri Feb 11, 2005 6:23 am Post subject: |
|
|
As far as cpu schedulers, I can't help much as I only know enough to be dangerous. In 2.6 there is O(1) "Ultra-scalable" (damn cool name) scheduler that is the main one (/usr/src/Documentation/sched-design.txt). There is also, I think, the SC or staircase and the PB or priority based; I'm pretty sure that you need to have a specific patchset to have either of the alternative cpu schedulers.
Regarding gaming and schedulers -- haven't seen specific benchmarks.
I'm actually in the middle of doing some benchmarks comparing various schedulers and parameters for a database server. I'll post some results when I'm done, but they may not help w/ destop or gaming. _________________ Calling fly fishing a hobbie is like calling brain surgery a job.
Last edited by commonloon on Fri Feb 11, 2005 6:38 am; edited 2 times in total |
|
Back to top |
|
|
steve_d555 Guru
Joined: 07 Nov 2004 Posts: 458 Location: Belmont, Massachusetts
|
Posted: Fri Feb 11, 2005 6:32 am Post subject: |
|
|
OK, Ya I heard somewhere that the Staircase/CFQ is not very good for gaming as it doesnt allow users to take 100% of the CPU or something.
I heard that the best setup would be the default Scheduler with Deadline in gentoo sources. I'm not sure, but I might try testing it with a new rig I'm getting just to try and get the full performance outta my 6600GT
--Steve |
|
Back to top |
|
|
oggialli Guru
Joined: 27 May 2004 Posts: 389 Location: Finland, near Tampere
|
Posted: Fri Feb 11, 2005 8:42 pm Post subject: |
|
|
Hmm, you probably won't gain anything in games using those things compared to what you gain using cfqv2 and a recent staircase in multitasking/general desktop usage. For example I get 64,1 fps in Doom3 timedemo demo1 usecache second run with 1024x768, 4xAA, 8xAF, Ultra Quality. I think that is enough, and with the stock scheduler my desktop feels a lot more choppy.
Also, batch scheduling absolutely rocks. _________________ IBM Thinkpad T42P - Gentoo Linux |
|
Back to top |
|
|
masseya Bodhisattva
Joined: 17 Apr 2002 Posts: 2602 Location: Baltimore, MD
|
Posted: Fri Feb 11, 2005 9:33 pm Post subject: |
|
|
Moving from Gentoo Chat. _________________ if i never try anything, i never learn anything..
if i never take a risk, i stay where i am.. |
|
Back to top |
|
|
fallow Bodhisattva
Joined: 08 Jan 2004 Posts: 2208 Location: Poland
|
|
Back to top |
|
|
commonloon Tux's lil' helper
Joined: 13 Apr 2004 Posts: 133 Location: Marin, CA
|
Posted: Sun Feb 13, 2005 2:32 am Post subject: Some benchmarks comparing schedulers |
|
|
...Thanks Fallow for the info....
What follows is some of the benchmarks I've been running on a new server. They are only benchmarks (for fun). I don't really think they show much difference between schedulers (I'm sticking w/ deadline, since this is a database server). I'm also testing using a SQL test script to test closer to the real world. The server's specs are:
Tyan 2882 mobo
16GB ram
2x 250 Opterons
2x10K 73GB RAID1 OS
6x15K ~200GB RAID10 data
gentoo-dev-sources 2.6.9-r14
I have booted w/ mem=4G to make the tests run a bit quicker, as w/ 16GB of ram running a non-caching test (~32GB min) is time consuming. I use ext2 instead of xfs (what I will be using), so that I could eliminate any effects of writing the journal from latency timings. XFS results in better read, write bandwidth and non-write latencies. Also, note that I've found that bonnie++ 1.93c does not show per char K/sec correctly(?), version 1.03 does, and I used 1.93c below.
Code: |
Anticipatory, ext2
Tiotest results for 4 concurrent io threads:
,----------------------------------------------------------------------.
| Item | Time | Rate | Usr CPU | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write 8192 MBs | 152.7 s | 53.637 MB/s | 0.7 % | 11.4 % |
| Random Write 16 MBs | 4.1 s | 3.818 MB/s | 0.1 % | 1.3 % |
| Read 8192 MBs | 224.7 s | 36.454 MB/s | 0.4 % | 5.1 % |
| Random Read 16 MBs | 7.4 s | 2.102 MB/s | 0.1 % | 0.4 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item | Average latency | Maximum latency | % >2 sec | % >10 sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write | 0.196 ms | 1137.279 ms | 0.00000 | 0.00000 |
| Random Write | 2.395 ms | 343.993 ms | 0.00000 | 0.00000 |
| Read | 0.424 ms | 537.257 ms | 0.00000 | 0.00000 |
| Random Read | 6.895 ms | 104.739 ms | 0.00000 | 0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total | 0.318 ms | 1137.279 ms | 0.00000 | 0.00000 |
`--------------+-----------------+-----------------+----------+-----------'
Tiotest results for 1 concurrent io threads:
,----------------------------------------------------------------------.
| Item | Time | Rate | Usr CPU | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write 8192 MBs | 109.0 s | 75.176 MB/s | 0.7 % | 13.1 % |
| Random Write 4 MBs | 1.3 s | 3.020 MB/s | 0.0 % | 1.5 % |
| Read 8192 MBs | 42.5 s | 192.654 MB/s | 1.6 % | 25.6 % |
| Random Read 4 MBs | 3.8 s | 1.029 MB/s | 0.1 % | 0.2 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item | Average latency | Maximum latency | % >2 sec | % >10 sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write | 0.031 ms | 33147.868 ms | 0.00005 | 0.00005 |
| Random Write | 0.566 ms | 9.925 ms | 0.00000 | 0.00000 |
| Read | 0.020 ms | 17.176 ms | 0.00000 | 0.00000 |
| Random Read | 3.794 ms | 20.777 ms | 0.00000 | 0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total | 0.027 ms | 33147.868 ms | 0.00002 | 0.00002 |
`--------------+-----------------+-----------------+----------+-----------'
Version 1.93c ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
taz 7G 920 99 77348 12 52209 10 1424 98 201612 25 792.5 6
Latency 9104us 670ms 630ms 23415us 26568us 548ms
Version 1.93c ------Sequential Create------ --------Random Create--------
taz -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 6846 94 +++++ +++ +++++ +++ 7173 96 +++++ +++ 22568 93
Latency 9174us 14us 30us 5238us 12us 277us
Deadline, ext2
Tiotest results for 4 concurrent io threads:
,----------------------------------------------------------------------.
| Item | Time | Rate | Usr CPU | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write 8192 MBs | 148.0 s | 55.339 MB/s | 0.7 % | 11.4 % |
| Random Write 16 MBs | 3.6 s | 4.390 MB/s | 0.1 % | 1.9 % |
| Read 8192 MBs | 84.2 s | 97.271 MB/s | 0.8 % | 13.6 % |
| Random Read 16 MBs | 5.2 s | 3.010 MB/s | 0.1 % | 0.7 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item | Average latency | Maximum latency | % >2 sec | % >10 sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write | 0.187 ms | 948.769 ms | 0.00000 | 0.00000 |
| Random Write | 1.733 ms | 239.241 ms | 0.00000 | 0.00000 |
| Read | 0.155 ms | 138.513 ms | 0.00000 | 0.00000 |
| Random Read | 3.972 ms | 22.511 ms | 0.00000 | 0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total | 0.177 ms | 948.769 ms | 0.00000 | 0.00000 |
`--------------+-----------------+-----------------+----------+-----------'
Tiotest results for 1 concurrent io threads:
,----------------------------------------------------------------------.
| Item | Time | Rate | Usr CPU | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write 8192 MBs | 112.6 s | 72.784 MB/s | 0.7 % | 13.2 % |
| Random Write 4 MBs | 1.3 s | 3.082 MB/s | 0.2 % | 0.8 % |
| Read 8192 MBs | 42.4 s | 193.102 MB/s | 1.6 % | 24.3 % |
| Random Read 4 MBs | 4.0 s | 0.974 MB/s | 0.0 % | 0.2 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item | Average latency | Maximum latency | % >2 sec | % >10 sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write | 0.032 ms | 1641.053 ms | 0.00000 | 0.00000 |
| Random Write | 0.504 ms | 9.520 ms | 0.00000 | 0.00000 |
| Read | 0.020 ms | 16.559 ms | 0.00000 | 0.00000 |
| Random Read | 4.011 ms | 31.399 ms | 0.00000 | 0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total | 0.027 ms | 1641.053 ms | 0.00000 | 0.00000 |
`--------------+-----------------+-----------------+----------+-----------'
Version 1.93c ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
taz 7G 1055 98 74916 12 50356 10 1610 97 202775 24 1150 9
Latency 7936us 35201ms 383ms 22374us 25876us 589ms
Version 1.93c ------Sequential Create------ --------Random Create--------
taz -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 6934 95 +++++ +++ +++++ +++ 7149 97 +++++ +++ 22126 93
Latency 6253us 12us 56us 7395us 12us 288us
CFQ, ext2
Tiotest results for 4 concurrent io threads:
,----------------------------------------------------------------------.
| Item | Time | Rate | Usr CPU | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write 8192 MBs | 133.8 s | 61.235 MB/s | 0.8 % | 13.0 % |
| Random Write 16 MBs | 3.6 s | 4.388 MB/s | 0.0 % | 1.6 % |
| Read 8192 MBs | 87.6 s | 93.490 MB/s | 0.8 % | 13.2 % |
| Random Read 16 MBs | 6.5 s | 2.419 MB/s | 0.0 % | 0.7 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item | Average latency | Maximum latency | % >2 sec | % >10 sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write | 0.145 ms | 13981.799 ms | 0.00129 | 0.00038 |
| Random Write | 1.980 ms | 499.414 ms | 0.00000 | 0.00000 |
| Read | 0.148 ms | 124.167 ms | 0.00000 | 0.00000 |
| Random Read | 3.931 ms | 92.474 ms | 0.00000 | 0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total | 0.152 ms | 13981.799 ms | 0.00064 | 0.00019 |
`--------------+-----------------+-----------------+----------+-----------'
Tiotest results for 1 concurrent io threads:
,----------------------------------------------------------------------.
| Item | Time | Rate | Usr CPU | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write 8192 MBs | 90.0 s | 90.997 MB/s | 0.9 % | 17.1 % |
| Random Write 4 MBs | 1.8 s | 2.156 MB/s | 0.0 % | 0.9 % |
| Read 8192 MBs | 42.1 s | 194.630 MB/s | 1.7 % | 25.8 % |
| Random Read 4 MBs | 3.9 s | 1.014 MB/s | 0.1 % | 0.2 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item | Average latency | Maximum latency | % >2 sec | % >10 sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write | 0.029 ms | 16312.277 ms | 0.00005 | 0.00005 |
| Random Write | 1.043 ms | 9.102 ms | 0.00000 | 0.00000 |
| Read | 0.020 ms | 19.132 ms | 0.00000 | 0.00000 |
| Random Read | 3.852 ms | 27.849 ms | 0.00000 | 0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total | 0.026 ms | 16312.277 ms | 0.00002 | 0.00002 |
`--------------+-----------------+-----------------+----------+-----------'
Version 1.93c ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
taz 7G 911 99 84611 16 50582 11 1313 98 199747 25 1131 10
Latency 10157us 16763ms 289ms 24113us 24422us 589ms
Version 1.93c ------Sequential Create------ --------Random Create--------
taz -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 6039 82 +++++ +++ +++++ +++ 6171 84 +++++ +++ 21658 91
Latency 7275us 17us 30us 6350us 19us 231us
|
_________________ Calling fly fishing a hobbie is like calling brain surgery a job. |
|
Back to top |
|
|
commonloon Tux's lil' helper
Joined: 13 Apr 2004 Posts: 133 Location: Marin, CA
|
Posted: Mon Feb 14, 2005 2:46 am Post subject: |
|
|
This came across LKML. While I haven't read it completely, it seems to have a nice overview of the O(1), etc. scheduler.
http://josh.trancesoftware.com/linux/ _________________ Calling fly fishing a hobbie is like calling brain surgery a job. |
|
Back to top |
|
|
bollucks l33t
Joined: 27 Oct 2004 Posts: 606
|
Posted: Mon Feb 14, 2005 3:56 am Post subject: |
|
|
steve_d555 wrote: | I have a question, I hear that Deadline/Anticipatory is better for gaming as it allows it to take up 100%. Does anyone have any thoughts? Has anyone done any benchmarks against CFQ or something? I would be really interested to know, because ive been using Nitro sources with CFQ and was wondering what was better.
|
There's no such thing as "not allowing 100%". They all (cpu and i/o schedulers) try to use all the resources as much as possible, as efficiently as possible. It makes no sense to have cpu or i/o bandwidth available and not use it. The closest any of these schedulers comes to not using 100% is the anticipatory I/O scheduler which pauses briefly after a read (up to 6ms) anticipating another read because it would be cheaper to perform a related read after a short wait than starting a write or different read. It still does this to try and maximise throughput. None of them try to decrease throughput in any way, but they definitely do perform differently in different workloads. |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|