2.6.12-rc2-JadeX1 [[EBUILD ADDED]]

[cokey]

hyp0r makes a very good point, Tiger683, you have to be able to explain why you added things into your patchset.

Having lots of different patchsets can be a good idea IF they are made competently by people that know exactly what they are doing otherwise it just does Gentoo harm.

Tiger683, dont stop with the enthusiasm, take what Hyp0r and others have said on board and use it to your advantage. To be an established patchset you need trust and commitment, that takes time.
_________________
https://otw20.com/ OTW20 The new place for off the wall chat

[rhill]

there's no need to flame the guy. he's just creating something and having fun. i wouldn't touch it with a ten foot pole, but it's still interesting to look at... from a minimum safe distance. it's like a car accident (without the dead people).
_________________
by design, by neglect
for a fact or just for effect

[cokey]

i didn't flame him, i gave good advice
_________________
https://otw20.com/ OTW20 The new place for off the wall chat

[Tiger683]

Hi,
i basically like the way this is evolving, but rather would like to
admit i already read some questions from hyp0r i cannot __yet__
answer in a way satisfying for him and others apparently knowing
more about the subject than i do....

And no, i don't really feel flamed at all...

Two things to point out:

1) im currently doing some hardcore reading on the subject to get enough
insights about the material to have a base knowledge for discussions with
ppl like hyp0r being lead on a fairly equal level of competence

2) implied by the fact i am doing lots of advanced literature, i plan to
take a step back in the defaults in this patchset, voluntary preempt is
obviously better for __all__ users ever interested in using a custom patchset.

For now, i really want the dust to settle down a little, unless someone really
DEMANDS some answers from me not being a support requests.

And hyp0r, i think you __might__ have overestimated my skills, although mybe im wrong,
no idea what you currently think of them, but lets just say, im having it as fun AND a learning process by now.
But thx for interest, as i said, next one will be more "sane" and less "coloured"

cheers

T

PS: No, im not ashamed and not afraid to say i found s.o. knowing more than i do,
and yes i do admit it..... Do with it whateber you want, but do it wisely....

PS2: remember, even some hardcore kernel hackers started off only knowing SOMETHING about what they were doing....
but its the only really motivating kickstart IMHO
_________________
Retired gentoo user

[hyp0r]

Wow. I'm really impressed by your point of view and the way you admit it. Many other people use to start insulting and become abusive, when they recognize they are going to "lose" the discussion.
Maybe there's still hope for this world.

[rhill]

[cokey]

[cokey]

Tiger683, excellent choice. Dont dare stop thinking about the patchsets, i can tell you will be alot of help in the community. Just remember 1 thing, not everyone is as sensible as you. When love-sources was started out lovechild and the vanilla dev (i think it was Johnm) worked together to bring a patchset that was experimental but also stable and helping the gentoo community. Things went wrong when the people just wanted speed, speed, speed at the sake of stability. Love was not meant to be that, thats why lovechild left and love-sources got then got a bad rep with the devs (so did lovechild because it had his name). Stability MUST be paramount with any sources or there is no upstream use which is why you will be flamed.

Dont let this stop you young padawan, there is hope for you yet...
_________________
https://otw20.com/ OTW20 The new place for off the wall chat

[predatorfreak]

[hyp0r]

Apart from several unsuitable comments, I share the interest and enthusiasm which has been risen in this thread. Coupling an interesting topic like this with the needed competence for these elements, the outcome might become a promising step for the consuming linux-community. You won't ever achieve realtime with the current task-environment, but you can set up a kernel, that enables a highly flexible scheduling and a real dynamic behaviour of tasks to adapt user requirements, that change unexpectedly. This is currently a problem, that has not really been solved. Most solutions behave in a way, that my grandma could do better. I think "stability" is often used as an excuse and on the expense of what could actually be done. Just a hint: Andrew Morton already includes many scheduling-patches into his mm-sources. You should always make sure you don't do work that others already did.

[Tiger683]

True.

I take one step back with the hard realtime preemption, i found it more promissing coupling Desktop-suited preemption from ingo's solution
with the improvements of scheduling code from others as well as taking a look at other highly interesting patches i.e. i follow the development of high
resolution timers, although it looks like it is moving slowly towards the "mainline". Other than that i was experimenting with the solution
presented here under heavy load multitasking, and it shows its major weaknesses (also thx to feld for reports about problems
on changing the load of system drastically, audio might skip then, which means we need something more adaptable).
For now, before i come up with something new, people should choose Desktop under preemption mode.

T
_________________
Retired gentoo user

[schnake]

Hi,
as there seem to be some common doubts "Why on earth realtime preemption could be needed for Linux" I will try to explain the need from the so called "professional audio" point of view. I know that this not the top priority usage for everyone, but even then it may be a useful example to explain the "What" and "Why". BTW, I am no expert neither on "professional audio" nor on "Realtime-Kernel programming" (hey, I'm not even a native english speaking person ), but by lurking on the LAD, LAU, ardour and jack mailing lists I may have gained enough understanding to provide at least a "birds view" on the subject.

Let's assume an application that produces some sound (PLAYER), and a hardware device to convert and output the (digital) samples as (analog) sound (SOUNDCARD). Let us further assume the SOUNDCARD has two hardware buffers (A and B) for storing the samples to be output, each big enough to hold the samples for 1 second of sound (actually the buffers would be a lot smaller (more like 1 millisecond), but I have the impression that it is easier to follow when using units we are used to ).

So playing sound would be like:
1. PLAYER starts by filling buffer A with samples
2. As soon as buffer A is full the PLAYER tells the SOUNDCARD to start playing from buffer A
3. PLAYER switches to buffer B and starts filling that

Now the PLAYER has exactly 1 second to finish filling buffer B, because as soon as the SOUNDCARD is done with processing the samples from buffer A it will switch over to buffer B. If the PLAYER did not manage to fill buffer B in time the sound will "skip".

Now, for any "normal" sound work (like OGG playback) every "recent" CPU will easily be able to calculate the samples for buffer B in time (and even use only a small percentage of the available cycles for that). But the PLAYER must get the chance to run its calculation in time.

If somewhere within the kernel there is a codepath being executed that
- takes a good part of (or even exceeds) 1 second runtime, and
- that can not be "interrupted" (preempted),
then the PLAYER will surely miss its deadline to fill buffer B within 1 second from step 2, regardless how powerful the CPU is.

That is what the patchset deals with. Kernel code that (potentially) blocks the CPU for a significant amount of time without being able to be preempted is changed (that may range from "simply" changing lock types to completely refactor that codepath) so that it can be preempted. The "worst case" (= the longest running codepath that still can not be preempted) basically is the system latency.

If that latency (= "the worst case") is known (let's say 0.5 seconds) a realtime task like the PLAYER can be sure that it will get its chance to run at least every 0.5 seconds if it needs so. If the PLAYER itself is "determinstic" and can guarantee (because it does not do anything "risky" like blocking I/O, GUI work etc.) that it needs max. 0.3 seconds for its own calculations it now in turn can be guaranteed that it will take a maximum of 0.5 + 0.3 = 0.8 seconds ("time", not "CPU usage"!) to fill the next buffer. Or put the other way round, if the SOUNDCARD buffers are at least "0.8 seconds big" the system can reliably output sound in realtime. And if you use buffers > 0.8 seconds there would even be some spare CPU cycles left for other things to do

Ok, but what if I simply use huge buffers (maybe 5 seconds each) and do not care about high (but surely below 5 seconds) system latency at all?

This is basically what "consumer audio" (I personally do not like that "consumer" versus "professional" wording, but anyway) sound deamons like ARTS do. The effect is that it takes some time until you actually hear the sound after you pressed "Play" (until the first buffer is filled), and that the sound continues playing for some time after you pressed "Stop" (until the last buffer provided is "empty"). In reality that time is mostly short enough to be hardly even recognized, so no big deal at all?

For simply playing some OGG's this maybe is not a big deal indeed. But what if, as one example, you want to use your computer as a realtime effects processor? You plug the microphone into your soundcard, have the computer calculate some nice effects on the input and output the result via the soundcard to your amplifier. Now you start singing along some playback. It takes 5 seconds to fill the soundcards input buffer, 0.3 seconds to calculate the effect and again 5 seconds to fill the output buffer. So everything you sing will take 10.3 seconds until it actually comes out of your speakers. On a karaoke event it might even be funny for the singer to be 10 seconds late, but any musician trying do something serious under such "live software monitoring" conditions will surely grow mentally ill in no time

Please keep in mind that everywhere within this post you should replace "seconds" with "milliseconds" to get some more realistic scale. And I heard somewhere that humans start to "unpleasingly" recognize audio latency at as low as 3 milliseconds (e.g. from the moment you "sing" until the moment you "hear").

So you need low audio roundtrip latency (< 3 ms) - and to that end kernel realtime preemption - as soon as you only do such "highly professional audio work" as simply trying to use your computer as an effects processor on your next karaoke party. Perhaps for some people this could already make the difference between "shit-hot" and "hot air" And note that realtime capabilities are absolutely not about providing better overall system perfomance, but "only" to enable code with realtime demands to be able to "keep promises".

[DrWoland]

I don't know anything about this RTP stuff, but I compiled the kernel with it and it works fine. There are a few things missing, but I'm really just waiting on fallow to come out with whatever he makes of his new patchset so I can have vesa-tng back.
_________________
I'm not a Guru, I just ask a lot of questions.

[Tiger683]

Heh, currently i've cut the work on it a little to release a new nitro, until then, nothing new will happen.
Next release will be a little less "experimental", but hey, its not as broken as some previous attempts....

stuff that isn't in: it's because it was broken while being in...

T
_________________
Retired gentoo user

[DrWoland]

[Tiger683]

Don't even mention it again

It was one of the most sick misfeatures in a patchset since a long time....
We don't want to be the retards of the forums, and it had more breakage than use anyways...

T
_________________
Retired gentoo user

[DrWoland]

[hyp0r]

@schnake
What's the stuff you're smoking called? You are describing a scenario, that is not doable in an environment that underlies non-rt circumstances. Scheduling tasks with deadlines in an environment, where one have and other's don't is impossible. That's what buffers are for - storing data, so a process does not get blocked. Otherwise prefetching is a solution, which is done very often, when it comes to fullfilling timing issues - at least within a limited degree of timing constraints.
Applications must be designed for using deadlines (compare the typical rt-dispatcher, that use c-tasks for this issue, which can be found in any trivial rt-book). RTAI executes the conventional linux within a scheduled thread besides deadline-scheduled threads, which is the opposite to jadex' approach, so jadex won't reach rt this way.
Anyway, if the player is that deterministic as you say, you can prefetch data, just like pipelines and prediction "buffers"(here's it again) do, so why should the be a need for deadline-scheduled requests, which provide their real strength in a non-deterministic environment?
Easier said: There are many IO-actions, that need to be changed in the current kernel, before the rt-plan will work at all and before the kernel developers do that, the user who requires such functionality will go for RTAI, RTL or QNX.

[schnake]

I'm not smoking anything particular dangerous, i hope

I already admitted upfront that I'm by no means an "technical" expert about those things. But I got the feeling that there sometimes is a basic misunderstanding when discussing this topic. Maybe the misunderstanding is on my side, but anyway it will be a win for me understand. Perhaps simply the term "realtime" leads into the wrong heading, too.

As I understand, you (and others) are talking mainly about how to correctly schedule task that have to meet deadlines. But I think the "Realtime-preemption" patch does not target scheduling, but latency.

Scenario: The soundcard raises an interrupt that it just switched to the second buffer. Now the audio producer must "re-fill" the first buffer until the second one is "eaten" by the soundcard. If at time of the interrupt the kernel just executes some long un-preemtable codepath the audio producer is "delayed" until that codepath finished (regardless on how smart any scheduler may be, the currenly running kernel code simply can not be preempted). The patch just tries to cut down the runtime of those un-preemptable kernel codepaths to a minimum. The audio processing code simply must get to run ASAP on some event (like a soundcard interrupt).

And as to prefetch (I assume this means something like preparing the "future" data forehand outside of the RT thread and only do a simple and cheap copy in RT) this is not an option for realtime audio, as the current "output" may well be a function of the current "input", so you can calculate only the current data, but never the future. For that reason the audio producer also may not be determinstic (as the complexity of the input may vary). But if it gets run "immediately" on the trigger chances are much better it will be able to keep its deadline than if it gets run delayed because of some non-preemptable kernel code. And it is not only about producing data "in time", but more like running a "read-process-write" cycle ASAP.

As to buffers, of course you always deal with any high latency simply by making then "big enough". But exactly that is contrary to "realtime"-audio, where buffers must be as small (or "short") as possible. If a human brain can notice a sound latency of as low as 3 ms you simply can not achieve "perceived" realtime audio as long as there may be a (let's say) 20 ms system latency from some non-preemptible kernel codepath (and what I heard reiser4 seems to want to win the price for introducing the longest non-premptible codepaths ever )

So, as far as I understand this is not about "real hard realtime" at all (as for scheduling to deadlines, RT-Plan etc.), but simply about to get some code running ASAP on some triggering event by reducing the potential delay that comes from non-preemptible kernel code.

But maybe I am wrong

[hyp0r]

[schnake]

[Tiger683]

[schnake]

@hyp0r:

[schnake]

[hyp0r]

The "beef" is, that one should be aware, what's this about and where the patches strength is in.
As always, you know that you cannot tune a particular function for absolute, ultimate, ultraperformance, while others stay the same.
This is like OPs that are more often used will have lower execution times, i.e. lower latency, while less used OPs will have higher execution times.

That's why I asked Tiger683 for some "scientific" evalution on this, to see where the balance went to. Some people certainly like lower latency for audio applications, but are not willing to do this on the expense of their usually used applications.
That's what I was after the whole time. :p