kernel patches in portage

rbr28 · Tux's lil' helper Joined: 09 Feb 2004 Posts: 126

Has anyone ever considered having kernel patches in portage so that kernel building could be as modular as the rest of gentoo? Even if the patches didn't apply, but were simply dropped into /usr/src, it would still be a huge benefit. To be able to look in one place and just select the specific combination of security, bugfix, or performance/feature patches that one wanted to apply would save so much time.

I know so many people who either patch from vanilla, or complain about numerous things missing from the sources they use. Example, I patch my own vanilla kernel with Reiser4, grsecurity, and numerous other patches becaues I don't like how unstable the mm-sources are, the hardened-dev-sources are missing too many features I want, and I don't want everything in the nitro-sources either.

Just being able to emerge all the patches and then apply them, rather than having to search the net for a dozen patches, would save so much time. I'm also guessing that patches could be offered via portage the day they are released or at least a lot sooner than developers could get the patches into an official kernel release.

scristian · n00b Joined: 11 Sep 2004 Posts: 28

the patches are in /usr/portage/distfiles/genpatches-2.6.x for 2.6.x kernel

rbr28 · Tux's lil' helper Joined: 09 Feb 2004 Posts: 126

That's not quite what I'm suggesting. Those patches are only for a specific kernel that you emerge. If I wanted to build my own sources with half a dozen patches, I might have to emerge gentoo-dev-sources, hardened-dev-sources, and maybe even something else, and then extract those patches and apply them. Considering that there are times where those sources wouldn't even be on the same version of the kernel, it would be even more difficult to get the patches this way, than just collecting them from various ftp sites.

The whole point would be to enable emerging any number of patches you want. The ebuild would actually have to do nothing more than drop the patch in /usr/src/. It would be the shortest and easiest solution to building kernels very specific to ones needs, and I think much more realistic and more flexible than trying to have the patches actually applied to the kernel sources.

Again, I think this is really in-line with the way Gentoo works in general, and is probably one of the more important ways in which the system should be flexible and customizable. Look at how many patches are in something like the wolk-sources or mm-sources. A lot of people actually do use only a few of those patches, yet they are not in any other kernel source. Wolk sources are usually behind a bit, mm-sources are often too unstable, etc. Having the flexibility to patch your own kernel is always there, but having easy access to all the available patches, in one place...all within the software management system, would be very unique and beneficial.

dsd · Posted: Sun Jan 16, 2005 10:53 pm Post subject: Re: kernel patches in portage

rbr28 · Tux's lil' helper Joined: 09 Feb 2004 Posts: 126

They would learn...hopefully. That's what Gentoo is all about anyway. I'm sure most users had never even compiled a kernel or used the command line to create a filesystem, among other things, before using Gentoo.

If all else failed, we could start a whole new forum dedicated to patch questions

wolf_99 · Apprentice Joined: 22 Oct 2003 Posts: 213

I have to addmit, an easy way to find patches and to patch them into the kernel dose sound good.

Gentoo is a distro for the one's who seek to learn, I think that such an option would hit the nail right on it's head. Maybe you should talk with one of the emerge developers?
_________________
To err is human. To really fuck up, use a computer.

rbr28 · Tux's lil' helper Joined: 09 Feb 2004 Posts: 126

The amount of discussion on this topic, doesn't bode well for my suggestion. The gentoo community always surprises me, in both good and bad ways. You can say something controversial like Gentoo sucks, and the thread will go on for months with dozens of responses, but hardly anyone even responded to a topic that is really a big part of building and maintaining a stable and secure Linux system.

Even more surprising is that so few responded to the comment about most Gentoo users being incapable of applying a patch! Unfortunately it seems that may be the case, and the reason why there is such a lack of interest.

Maybe I should have labeled the topic, why is there no porn in Portage?

Pink · Veteran Joined: 24 Jul 2003 Posts: 1062

To be honest, I'm not totally sure what you are suggesting.

So you want the ability to emerge a patch which then places it in /usr/src without applying it to the kernel? (Because there is no way on earth you could do that automatically because of any rejects).

I just don't get it.

Unless it comes down to not being able to find the patches. Is it that you are after a huge reposity of all the available kernel patces? A server full of literally thousands upon thousands of patches from all over the web.

Lets say it was done, and now portage is 50% larger and more cumbersome with the additional 50,000 files needed to cope with every single kernel patch available (Obviously that is a guess but there would be many thousands of files needed to be able to handle the thousands of kernel patches available).

For me (apart from just not getting why you would need it) this suggestion brings up so many questions:

Who maintains it? Who adds to it? Who does the QA? How does it keep up with the dozens of new patches released every day? Do we allow my win4lin patches in there, nitro patches, or just Linus sanctioned ones?

Would it allow you to 'emerge kgbd-ga.patch'? If so, will it bring the patch dependencies? Would it block it if you have conflicting patches already installed?

Would each ebuild explain what the patch is, what it does, what it changes and so on?

Then we come to the really interesting stuff - how do you offer help for this monumental repository? Do you allow the Gentoo devs to waste valuable time fixing bugs on patches that aren't their own (this is why no third party patchset will make it into portage)? Or do you expect the patch maker/builder/maintainer to offer help?

I don't think you have actually thought about the consequences or feasability of what you are suggesting - which, bared to the bone, is an end users ability to download a patch because they can't be bothered to look for it, and then expecting some sort of official help when the ebuild doesn't work, doesn't patch or the user experiences a reject, can't reboot and so on. (Which, believe me will happen!).

rbr28 · Tux's lil' helper Joined: 09 Feb 2004 Posts: 126

To be honest, I'm not totally sure what you are suggesting.

I'm suggesting the ability emerge patches. That means nothing more than having an ebuild in portage for each individual patch, that merely drops the patch in /usr/src.

So you want the ability to emerge a patch which then places it in /usr/src without applying it to the kernel?

Yes

Unless it comes down to not being able to find the patches.

Yes, this takes forever. Do you not believe this? How about I give you what I want in a kernel and see how long it takes you to find all the appropriate patches. Now do this every time you upgrade a kernel. I'm not talking about someone who applies 1 patch to a kernel on one machine. I'm talking about the benefit to people who support numerous servers and/or workstations, and regularly build new ones, update kernels, patch their kernels, etc. Yes anyone can find a single patch, and download it once in a while, without portage. For that matter anyone can download a single application, configure the makefile, and compile it. That doesn't mean having stuff in portage is unnecessary!

Lets say it was done, and now portage is 50% larger and more cumbersome with the additional 50,000 files

Don't talk nonsense. The ebuilds, not the patches would be in portage. 50% larger, 50000 files, that's just nonsense. Regardless, not adding something to portage because portage couldn't handle it doesn't mean it's not important. Even if that were the case it would just mean something had to change with portage. The portage tree has continued to get bigger and bigger, and this isn't something that is going to break portage on it's own.

Who maintains it, who adds it?

Who cares? How is it any different than how any other ebuilds are maintained? No one has to worry about the patches in any way. Ebuild developers don't worry about the quality of code for the app they build, and they don't support the application itself. If the ebuild does what it's supposed to, the support goes to the application developer, the patch developer or whoever. This is no different than having kernel sources or anything else in portage that isn't actually compiled.

Would each ebuild explain what the patch is, what it does, what it changes and so on?

Again, this is no different than any other ebuild. Have a link to the site, have a changelog, whatever.

Do you allow the Gentoo devs to waste valuable time fixing bugs on patches that aren't their own (this is why no third party patchset will make it into portage)? Or do you expect the patch maker/builder/maintainer to offer help?

Again, this isn't different than anything else in portage. The developer supports the ebuild. The application developer supports the application. People are free to ask questions on the forums, but the developer of an ebuild has never had any obligation to support the application itself, at least not that I know of. So where is the time being wasted? How much time is wasted answering questions about things that dont work or machines that have been hacked, just because someone hasn't applied patches? Sure having patches in portage isn't going to make everyone start patching, but it will make it easier.

I don't think you have actually thought about the consequences or feasability of what you are suggesting -

Yes, I have. I'm a technology expert in a huge enterprise environment and I build kernels almost daily and support a huge number of machines and users. I know that patching is absolutely critical, and I know it's a big pain. I know that portage is one of the best software management systems out there, and adding this functionality to it would really be an almost trivial task, and very beneficial to a lot of people.

which, bared to the bone, is an end users ability to download a patch because they can't be bothered to look for it, and then expecting some sort of official help when the ebuild doesn't work, doesn't patch or the user experiences a reject, can't reboot and so on. (Which, believe me will happen!).

Why won't the ebuild work when all it has to do is drop a text file into /usr/src/ ? If that doesn't work the ebuild developer has some serious problems. Your argument is like saying, why have portage, it's just for people too lazy to download applications from all over the net, compile the, etc. Maybe that's true, but that's what makes portage and Gentoo so easy to use, even though there is so much flexibility and configurability. Having kernel patches available via portage would contribute to Gentoo's flexibility, configurability, security, stability, and ease of use and ease of administration, very significantly.

Other benefits to these being in portage...search capability. Try searching smbfs patch on the internet. How much easier would it be to do this in portage? Lots of patches would be much easier to find in portage than on the Internet. You would also search descriptions so you wouldn't even have to know the exact name of the patch, which is sometimes the only way you can find one on the Internet. Being able to quickly see what patches you have applied, dependency checking in some way or at least narrowing your options (i.e. you could see all patches that were relevant to x86 and leave out patches specific to other platforms). Most of the advantages that anything else has in portage would apply to patches just as well.

Finally, there is such a fear of support, but again this would just be a learning process, just like everything else with Gentoo. Yes people would ask questions about why a patch didn't apply, or how to make it apply, and people would respond, just like they respond to 1000's of other questions, some much more irrelevant. Again, not doing this because people might ask lots of questions, certainly is no justification for not doing it. How many questions are there already on the forums about other patched sources like nitro or speedy that aren't even official gentoo sources?

There's no arguing that kernel patching is important to both security and stability, and arguably important for other things such as cutting edge hardware support, or other new features. Why should gentoo not "try" to make this as easy as possible? This is something I think would really make people take notice...that Gentoo is doing something that really makes life easier for serious linux users and administrators who know the benefits of patching, and as far as I know, Gentoo would be the first to make this significantly easier for the average user.

dsd · Posted: Wed Jan 19, 2005 12:37 pm Post subject:

rbr28 · Tux's lil' helper Joined: 09 Feb 2004 Posts: 126

Well you have gotten me to agree on one thing...that it would be useless for the majority of people

. That's unfortunate, but it certainly does appear that people would rather have someone else provide a patched kernel for them, than do it themselves. I can understand that, since it can be a lot of work, but I think it would change if patching was easier.

I do fault a lot of the arguments though. Everyone is focusing on how difficult patches would be to apply, and how they often don't work, and how there are compatibility issues and so on. So what! That all exists already. Having the patches in portage would make the first step easier...that's all. It would simply make it easier to get the patches and keep up with them. It doesn't have to make it any easier to apply them, or have them applied. Yes, that would be ideal, but it is obvious that it's a near impossible task.

I guess though, with that in mind, I have to agree that maybe just having a website or ftp server that collects the patches would be the best option. I might work on that. I already keep all the patches I use regularly on an ftp server, and it doesn't appear that serving up thousands of bzipped patches via an ftp server would be as big a project. The real work would still be just in collecting the patches and keeping up with the latest ones. If anyone thinks this would be useful, let me know. It would be simple if I'm doing it myself, since website building is incredibly boring to me. I would probably just dump stuff on my ftp server, and provide a web page that somehow explained and/or organized what was there, for those who didn't want to simply browse the ftp server. Again, it would be really simple, and I would start out small, for example, only collecting patches for the 2.6.x kernel at first.

I think a lot of the Gentoo users would think it's awful how a Windows user may accept that MS has built and configured Windows well, but so many accept that someone has patched their kernel well, to provide the security and features they need. Not a whole lot different than trusting who created the patch, or anything else you use I guess, just a matter of degree. At least with Linux there are so many more options, and you still have that option to do it your way, but it would be nice to make patching easier for the average linux user.

I think some people believe patching is for advanced users only, and that's a problem. Here's a good example, in our university environment, almost anyone using Linux would need mppe support for our vpn. As far as I know, the wolk sources are one of the only kernel sources that has that patch (and I may be wrong on this, but I didn't think Gentoo provided the 2.6.x Wolk sources). They need the vpn to use wireless anywhere on campus, and they need the vpn to access many campus resources from off campus. Just the fact that they need that one patch would be prohibitive for many linux users. Who would tell them to use the wolk sources for that one patch? Most likely we would have to provide instructions for patching their kernel (which we do), but as many have said, that is difficult for many users, and that is just one of the easiest patches to apply.

I'm not going to belabor the point that a method for patching kernels should be in portage, but regardless I think the Linux community in general needs to work on a better way for the average user to patch a kernel. Whenever someone says something is too difficult for most users, that's a sure sign it needs to be worked on, but usually that's an accepted answer, and it really hurts Linux's more widespread use. I know you could argue that a user should just use a distribution that provides a kernel patched with mppe for example, or they should know how to patch their kernel, but I don't think avoiding the problem is the answer.

dsd · Posted: Wed Jan 19, 2005 1:48 pm Post subject:

rbr28 · Tux's lil' helper Joined: 09 Feb 2004 Posts: 126

I know what you mean about moving away from mppe. Our university is now providing ipsec/l2tp in addition to mppe so that users can access the vpn either way. Unfortunately ipsec is not much easier to setup for Linux users, at least not in our environment. VPN support is a real sore spot for me when it comes to Linux. It's getting there, but setting up client access on Linux is not close to where it is on Windows, and that's a major issue for our users.

nerdbert · l33t Joined: 09 Feb 2003 Posts: 981 Location: Berlin

Rather than using packages for patching, which seems to be complicated, I'd suggest to have USE flags for common patches.

It would be great for all the Reiser4 folks for example to simply type:

rbr28 · Tux's lil' helper Joined: 09 Feb 2004 Posts: 126

ftp://namesys.com/pub/reiser4-for-2.6/

I've applied the reiser4 patch over everything from vanilla sources to kernels that already had numerous other patches applied. I can't remember though if I had to do any manual merging, but if it was anything really difficult, I'm sure I would have remembered it.

Also, gentoo-dev-sources, do not contain all that many patches. If you are only using the sources on x86, you can eliminate several patches for other architectures. There may only be half a dozen patches you really need out of gentoo-dev-sources, and as I've found, it's often easier to just do your own patching.

Grsecurity was one of the things that really got me going on patching, because I felt like there was no reason ever to not use at least some of the features it offered. I'm not going to get into that debate again, but in my opinion it too should be in the kernel. I know hardened-dev-sources and some others are pre-patched with grsecurity, but I would have had to patch those with other things I wanted, so I just always do it from scratch now.

Same goes for Reiser4. mm-sources aren't intended for mainstream use, and some of the other patch sets with reiser4 either had much more than I wanted, or were still missing things, so I just started patching the vanilla kernel for that too.

In most cases, even merging patches manually (meaning looking at the reject files and fixing failed merges), isn't that difficult. The only time I've seen major difficulties is if you try to patch with something like mm-sources or another patch-set that is really a huge number of individual patches, and then you try to apply patches on top of that. You end up with so many files that have been modified, so many incompatibilities, etc, you really have to know what you are doing to not break something applying patches over a kernel already patched with something like that. In many cases though, when you apply numerous individual patches, they aren't really related and they only occasionally change a file that has already been changed by a previous patch, or at least fail on that file because the changes are too significant.

If you like to just play around and learn about the kernel, it's a good way to get started. I have guessed my way through merging some patches, just experimenting to see what effect something would have, or if my intuition about where something was supposed to be was right. It does get easier and easier the more you do it and I certainly do not consider myself a programmer or kernel developer.

dsd · Posted: Wed Jan 19, 2005 3:22 pm Post subject:

i should add a FAQ item about this..

USE flag based patch application is not something we want to get into. it implies that some patches are not ready for mainstream use (i.e. use for everyone), in which case we dont really want the patch at all. it also makes maintenance that much harder as we have to check different combinations of use flags.
_________________
http://dev.gentoo.org/~dsd

Pink · Veteran Joined: 24 Jul 2003 Posts: 1062

I've been away for a couple of days and I think most points have been covered pretty well by both sides.

I'll just pick up on one point if I may:

nerdbert · l33t Joined: 09 Feb 2003 Posts: 981 Location: Berlin

Pink · Veteran Joined: 24 Jul 2003 Posts: 1062

OOZafle · Apprentice Joined: 19 May 2004 Posts: 150

maybe instead of taking all this time to add all these ebuilds into portage someone could just put up a webpage on the gentoo site with a collection of URLs to these patches with a disclaimer on the top. Would be as simple as knowing basic HTML. alot simpler than adding all this stuff to portage and complicating developers lives. anyone agree?
_________________
Time to kick ass and chew bubble gum, and i'm all out of gum.