[RFC/RFT PATCH v3] sched: automated per tty task groups

[albright]

[SlashBeast]

I wonder if I can mount no only cpu there but also something to control IO. Anyway I did it the-shellscript-way a little different, If anyone care:

[Dont Panic]

I'm always dubious about patches like this one, especially when Phoronix starts shouting from the tops of the towers how fantastic the feature is.

But I've done a fair amount of benchmarking, and there seems to be a lot of places where task group scheduling helps (sometimes very noticeably), and I couldn't find much downside in my testing.

My tests used a kernel compilation with -j6 on a single CPU system (AMD Athlon(tm) XP 2600+) in conjunction with the latency test wakeup-latency cited in Mike Galbraith's LKML posts discussing the patch (http://lkml.org/lkml/2010/10/19/212). In one console I would make a kernel after a fresh reboot and a 'make clean' using: 'time make -j6 && killall wakeup-latency'. In another window, I ran the wakeup-latency benchmark, which was terminated at the end of the kernel compilation by the 'killall' command.

Please keep in mind that even though the wakeup-latency test shows eye-popping numbers, it is a purpose-built benchmark to test latency, and these results don't always translate into real-life improvement. Having said that, I do notice an improvement in Xorg performance (in terms of latency) when running CGROUPS.

I also wanted to compare a kernel with the "sched: automated per tty task groups" patch versus Lennart Poettering's manual method (See Fitzcarraldo's post above) versus Group Scheduling disabled versus Group Scheduling not built (I'd heard that even having Group Scheduling would degrade performance).

My conclusion was that latency could be improved considerably by implementing CGROUPs scheduling without impacting system performance as measured by a kernel compilation (as with anything like this, I'm sure there are performance regressions out there somewhere).

There's certainly considerable variation in the wakeup-latency results, but I was more looking at when it made changes of two orders of magnitude.

I wish I knew how to do tables on this forum, but here's my results

Test #1 (baseline): average latency: 2592.5 µs Kernel Build Time: real 21m13.003s user 18m49.836s sys 1m32.097s
Test #2: average latency: 8.1 µs Kernel Build Time: real 21m13.089s user 18m53.186s sys 1m30.387s
Test #3: average latency: 32.0 µs Kernel Build Time: real 21m20.134s user 18m25.428s sys 1m49.036s
Test #4: average latency: 5600.8 µs Kernel Build Time: real 20m58.894s user 18m39.197s sys 1m31.424s
Test #5: average latency: 15.7 µs Kernel Build Time: real 21m1.724s user 18m45.550s sys 1m29.031s
Test #6: average latency: 6595.3 µs real Kernel Build Time: 20m45.245s user 18m34.281s sys 1m25.808s

Trial Descriptions:
Trial #1: Baseline trial. I used a 2.6.35.7 kernel without any "sched: automated per tty task groups" patch or manual task group scheduling. CGROUP task scheduling was built into the kernel (which seems to be the default usually, even with mrproper), but not implemented. Ran from console in TTY1 and TTY2.
Trial #2: I used a 2.6.36 kernel patched with the "sched: automated per tty task groups V3" patch, and allowed automated task groups to schedule the tasks. Ran from the console in TTY1 and TTY2.
Trial #3: A repeat of trial #2 with a patched 2.6.36 kernel, but I ran it under X11 (using IceWM, a fairly lightweight wm). Since I started X on TTY1 in this trial, I was curious to see if performance was affected by effectively running all tasks under the same TTY (the patch normally separates task groups based on TTY I believe).
Trial #4: Essentially a repeat of trial #2, but booted the kernel with the "noautogroup" boot parameter. The purpose of this trial was just to confirm that the "noautogroup" boot parameter really does turn off automatic group scheduling since I want to test the manual method in the next trial. Ran in console on TTY1 and TTY2.
Trial #5: A repeat of trial 4 with the "noautogroup" boot parameter on a patched 2.6.36 kernel. However, I implemented manual task group scheduling using Lennart Poettering's manual method (See Fitzcarraldo's post above for links). Tasks were run in console on TTY1 and TTY2, however, both tasks were grouped in the same category.
Trial #6: Essentially a repeat of trial #1, except this was a 2.6.36 kernel built with CGROUPs totally disabled. I had read that merely having CGROUPs enabled in your kernel incurred a performance penalty. However, the difference was probably within the noise level.

Even though I haven't performed a statistically rigorous number of tests, the kernel compilation times are fairly tightly grouped, so I've concluded that having CGROUPs will not significantly hurt or help things like compiling. The results of the wakeup-latency benchmark seem to vary. I'm not sure you'll notice the difference between an average latency of 25 and an average latency of 10, but when the latency changes by two orders of magnitude from 25 to 2500, this is a change I can feel when running X11.

But it's important to qualify the wakeup-latency result in that you're not really running 100X better with group scheduling. But you can now officially count me in the group that concludes there's a noticeable improvement running some implementation of group scheduling without much downside that I could identify.

[Fitzcarraldo]

Thank you for putting in all that effort to benchmark the different kernels and userspace approaches, Dont Panic. Very interesting and helpful.

I notice from LKML that the patch is still evolving. The latest iteration, Version 4, has moved away from using task groups per TTY to using task groups per session (http://lkml.org/lkml/2010/11/20/91). I think this new approach could be more relevant to 'normal' desktop users than the per-TTY approach.

[Shining Arcanine]

[asturm]

Am I the only one experiencing drawbacks of that patch? Something as simple as emerge calculating dependencies (utilizing only one of my two cores) brings sound output (plain ALSA here) with Amarok2 to its knees (stuttering playback).

I am using the version from Gentoo patchset v4 (the one in masked gentoo-sources-2.6.36-r2) against tuxonice-sources-2.6.36 running on a decent enough dual-core system (X200s). My system might be exotic in that it is equipped with a Seagate Momentus XT 500GB HDD including 4GB SLC SSD cache, with the portage tree quite likely residing inside the cache due to its regular use (and emerge --sync really is pretty fast).

[kernelOfTruth]

[asturm]

No issues with mouse yet. However, I can also reproduce the problem with kaffeine - output stops reliably when firing off an 'emerge -uvaDN world'.

[kernelOfTruth]

[asturm]

Btw, mouse and system just completely hung for several seconds during kmplayer playing an Xvid movie until the 'emerge -uvaDN world' had finished. I definitely didn't have those issues without the patch.

I am operating a fairly basic system without fancy stuff, no crypt, only regular ext4 partitions +mounting options (only noatime), as for IOSCHED:

[Shining Arcanine]

[asturm]

KDE-4.5.3 and xf86-video-intel-9999, already tried with and without compositing - with the same result. Any things that could stand out on your system compared to the average setup?

@EDIT: CONFIG_IO_DELAY_NONE=y made no difference whatsoever.

@EDIT #2: ...as did CONFIG_DEFAULT_IOSCHED="noop"

[Yamakuzure]

As I am not doing any multimedia stuff on my companies laptop, I gave the "alternative" (like described by SlashBeast and Fitzcarraldo) a try.

I have to do a lot of stuff inside a Virtual Machine (Windows XP-32, VMware Workstation 7, 768MB Ram from my laptops 2GB) and normally KDE4 apps, like thunderbird and firefox are quite unresponsive when the VM is running. (Even when Windows idles.)

With the "alternative" patch I started VMware Workstation via Konsole, thus with cgroups support, and while I am typing this everything is quite responsive. In the meantime Windows has installed Microsoft Office 2007 Service Pack 1 (268MB) and I had no issues with it. The only hint that the VM is working is the noise from the CPU fan and the hard drive.

So if this kernel patch (not the tty-only alternative) is going into the kernel for genral scheduling rather than tty-only, I think it's a more or less huge leap.
Well... at least when the drawbacks are sorted out...
_________________
Edited 220,176 times by Yamakuzure

[.yankee]

[devsk]

The kernel patch has run into some issues. There is more discussion around it on lkml. Looks like more changes are coming in sched. Looks like the final patch may not be safe for anything other than 2.6.38.

https://lkml.org/lkml/2010/11/28/47

[.yankee]

The patch works for me without noticeable drawbacks - I used the one posted on 24.11.10 against vanilla-sources-2.6.37_rc3 (for autogrouping) as well as userspace cgroups managing implementation (modifying .bashrc and .xinitrc). With these, I haven't noticed the downsides genstorm described.

[gringo]

[Dont Panic]

I see Mike Galbraith posted a v4+ patch on Tue, 30 Nov 2010.

https://lkml.org/lkml/2010/11/30/121

The notes indicate this patch is against "tip". Is the linux "tip" branch 2.6.38, or still with the 2.6.37_rc branch?

Has anybody tried this version yet against a 2.6.36 or 2.6.35 kernel?

There was some discussion about this patch addressing some issues strictly in the "tip" branch, so I was wondering if it was worthwhile to backport this patch.

[.yankee]

[Yamakuzure]

[asturm]

[.yankee]

[DaggyStyle]

I've heard that there are performance hits when system isn't in stress, did anyone noticed it?
_________________
Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein

[wrc1944]

I just booted the new OpenSuse kde-4.6 beta live cd (milestone 4, 32 bit version) because I read it had adopted this patch.
FWIW, here's the cgroup config section from their default boot kernel "Linux/i386 2.6.37-rc3-git1 Kernel Configuration" file.

[Dont Panic]

If you're trying to see if the patch is enabled (or even there), look for the CONFIG_SCHED_AUTOGROUP setting.