View previous topic :: View next topic |
Author |
Message |
smartass Apprentice
Joined: 04 Jul 2011 Posts: 189 Location: right behind you ... (you did turn around, didn't you?)
|
Posted: Sat Nov 03, 2012 11:51 pm Post subject: The quest for the ultimate .config |
|
|
Hi,
this topic may also fit into the Kernel category, but I think it fits here even more, as it's more about sharing opinions and experiences. Mods may observe and move it later if appropriate.
I'm sure I'm not the only one who has (successfully) configured and compiled several kernels, but after some time has become tired of adding modules to the .config as he encounters various hardware.
Sure, usually it's just
Code: |
cd /usr/src/linux
make nconfig #and find the relevant driver and hope it can be configured as a module without rebuilding the kernel
make modules
make modules_install
|
but sometimes that doesn't work or I need to test some hardware ASAP and telling someone "Give me a sec, I'll configure my kernel" or "Hmm, I cannot tell whether it's faulty or not, I'd have to find, configure and compile the driver first".
AFAIK if I need a new module and must enable some bistate option to get the module tristate option, I have to rebuild the kernel right? Usually it wouldn't compile otherwise.
Don't get me wrong, I don't have the *buntu mentality and I DO care about having my system as lean as possible, however, the environment I'm in forces me to have a more versatile kernel, almost a distro-level kernel.
So, I was thinking about how to configure a kernel which would be almost as sane and lean as a Pappy's kernel seed and yet would have most drivers for most hardware as modules so I wouldn't have to recompile it each time I encounter some new hardware.
Well, here are the most obvious approaches:
- make a .config which has most hardware options enabled in such a way, that the relevant driver can be configured as a module on-demand. E.g. if you find out you need some special broadcom wifi card module, you should have the relevant scsi option enabled beforehand
- use something as crazy as allmodconfig and try to turn off at least the most useless stuff
- start with a kernel seed or my own custom .config and use common sense to enable the most possible modules
For some time I've been using the 3. approach, but even then I always encounter such hardware that I didn't anticipate. The 2. option is quite close but I fear such a kernel would be terribly fat and slow, possibly prone to errors and unwanted enterprise policy/product options restricting normal desktop behavior. The 1. option is quite hard to achieve, because you may not know which drivers require what option and you don't want all drivers too (just for hardware mainly).
So today I've been playing with the idea I've come up with, fusing the 3 approaches together using join, comm, diff, sort to compare and merge .config files generated by allmodconfig, allnoconfig and pappy's kernel seeds, but it didn't work out, because make oldconfig just restarts the whole config if there are too many unmet dependencies.
I've found kconfglib.py, will try to use that instead:
- start with a sane base config, e.g. pappy's kernel seed
- enable everything tristate as module and enable their respective reverse deps
- disable everything tristate, leaving the options enabling their existence as tristate enabled
I've almost managed to do this, but oldconfig wouldn't check all the reverse deps
Now, let me know what you think |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54550 Location: 56N 3W
|
Posted: Sun Nov 04, 2012 12:02 am Post subject: |
|
|
smartass,
A kernel with lots of modules isn't slow and bloated if unwanted modules are not loaded.
It takes a long time to build and it take s more disk space.
Start with allmodconfig then turn on <*> everything needed to boot, so you don't need an initramfs and all the rubbish that goes with it.
Now blacklist anything that loads that you don't want. You can still load it by hand.
You must never edit the .config file bu hand, or with anything except the provided tools as maky config options flip several CONFIG_ flags.
Make oldconfig cannot sort out the mess, so you get a kernel full of strange bugs. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
smartass Apprentice
Joined: 04 Jul 2011 Posts: 189 Location: right behind you ... (you did turn around, didn't you?)
|
Posted: Sun Nov 04, 2012 8:44 am Post subject: |
|
|
Thanks for the feedback Neddy.
I've already tried what you described, but I soon realized, that many modules require many other options aimed at specific enterprise platforms.
The inconvenient thing about allmodconfig is that its' too close to allyesconfig. When I make allmodconfig and then find an option which I want to disable (e.g. something for embedded systems or servers), then I must first find all the modules that depend on it. Not impossible, but neither easy.
Although the idea with module blacklisting sounds interesting, it would still require a lot of human intervention.
Well, I got very close to starting at point 2. and getting to point 3., but now I'm wondering how to merge it with the sane base without disabling all the options necessary for later module options. E.g. pappy's config disables many things that I'd like to remain possible.
Guess I'll just have to do it by hand, I have to make some modules as built-in anyways.
Code: | # start with total bloat
make allmodconfig
# disable all modules
sed -i 's/=m/=n/g' .config
# validate and repair the config. This results in a config capable of enabling modules later
# could also use `yes n` as it asks about stuff I don't need (AppArmor, SElinux, TOMOYO,..)
# or just do it manually, as it's not very long
yes | make oldconfig
# set the sane base by hand, build in necessary module-capable stuff
# possibly could have the sane config opened in another window as KCONFIG_CONFIG=config.base make nconfig
# this step could be also done after make allmodconfig instead
make nconfig
|
This would work. My question is, what's users' experiences with adding modules to the kernel later on vs. really compiling all modules in the beginning?
If that turned out out be a bad idea, I'd think of doing something like
Code: | # start with total bloat
make allmodconfig
#record what was set as module
sed -n '/=m/p' .config |sort -t = > config.mod_only
# disable all modules
sed -i 's/=m/=n/g' .config
# validate and repair the config. This results in a config capable of enabling modules later
# could also use `yes n` as it asks about stuff I don't need (AppArmor, SElinux, TOMOYO,..)
# or just do it manually, as it's not very long
yes | make oldconfig
#now it's easy to disable really unnecessary options and set the sane base
make nconfig
#will work with it later
sed -e 's/# \(.*\) is not set/\1=n/g' -e '/#.*\|^$/d' .config |sort -t = > config.allmod_capable
#enable again all modules
join -v 1 -j 1 -t = config.allmod_capable config.mod_only | sort -t = - config.mod_only > .config
#fix and validate the config
make oldconfig
|
The last step still restarts the whole config, trying to get my head around it.... Might have to use that kconfiglib.py
EDIT:
Found out this actually works, but feels like a big hack:
Code: | # start with total bloat
make allmodconfig
# record what was set as module
sed -n '/=m/p' .config |sort -t = > config.mod_only
# disable all modules
# it's important that the symbols are completely removed and oldconfig is not called
sed -i '/=m/d' .config
# now it's easy to disable really unnecessary options and set the sane base
make nconfig
mv .config config.base
# add the modules back
cat config.mod_only >> config.base
# make a new config enabling all modules but respecting our disabled options
KCONFIG_ALLCONFIG="config.base" make allmodconfig
# may be necessary to review the .config, as not all disabled options may have been respected
make nconfig
|
The last step spits out many errors, but works as expected in most cases and is a valid config. |
|
Back to top |
|
|
ulenrich Veteran
Joined: 10 Oct 2010 Posts: 1483
|
Posted: Sun Nov 04, 2012 10:49 am Post subject: |
|
|
cd /usr/src/linux
make help |less
shows:
localmodconfig - Update current config disabling modules not loaded
localyesconfig - Update current config converting local mods to core |
|
Back to top |
|
|
smartass Apprentice
Joined: 04 Jul 2011 Posts: 189 Location: right behind you ... (you did turn around, didn't you?)
|
Posted: Sun Nov 04, 2012 12:13 pm Post subject: |
|
|
ulenrich wrote: | cd /usr/src/linux
make help |less
shows:
localmodconfig - Update current config disabling modules not loaded
localyesconfig - Update current config converting local mods to core |
Hi,
I am aware of these options, but as I said, I cannot know beforehand what driver I may need tomorrow and I'd like to be prepared for most HW. |
|
Back to top |
|
|
ulenrich Veteran
Joined: 10 Oct 2010 Posts: 1483
|
Posted: Sun Nov 04, 2012 1:25 pm Post subject: |
|
|
Uuuh, excuse me, I didn't read your starting article
... because I don't understand what you want (when tried to read it).
I do compile my little vmlinuz faster than a fat kernel. Just ten minutes to go on a tmpfs. It is a throw away commodity for me ... if updated external modules need to be emerged. |
|
Back to top |
|
|
smartass Apprentice
Joined: 04 Jul 2011 Posts: 189 Location: right behind you ... (you did turn around, didn't you?)
|
Posted: Mon Nov 05, 2012 6:34 pm Post subject: |
|
|
ulenrich, I'm not exactly sure what I'm looking for, that's why I started this thread to gather experiences and opinions of other users, e.g. compiling all modules vs. adding them in later.
I'm hoping someone with similar trouble will point me in the right direction |
|
Back to top |
|
|
augury l33t
Joined: 22 May 2004 Posts: 722 Location: philadelphia
|
Posted: Wed Nov 07, 2012 11:49 pm Post subject: |
|
|
smartass wrote: |
Hi,
I am aware of these options, but as I said, I cannot know beforehand what driver I may need tomorrow and I'd like to be prepared for most HW. |
Sometimes you build 'em all. Sometimes you don't. < > Dallas's 1-wire support ---> is not tomorrows option IMO. |
|
Back to top |
|
|
smartass Apprentice
Joined: 04 Jul 2011 Posts: 189 Location: right behind you ... (you did turn around, didn't you?)
|
Posted: Thu Nov 08, 2012 7:27 am Post subject: |
|
|
Yes, some options seem more unlikely than others, but then I'm still left just with my best bet.
What I'm asking about is whether people usually just include most drivers as modules, excluding the really strange ones OR whether it's a good idea to build a kernel with most modules disabled, but with the possibility to build them later. |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54550 Location: 56N 3W
|
Posted: Thu Nov 08, 2012 7:27 pm Post subject: |
|
|
smartass,
My kernels contain everything I use every day built in. Thats not much more than items needed to boot without and initramfs and the network card.
Eveything else I use occasionally is <M>.
I take pot luck about building modules for new toys without doing a full kernel rebuild and try <M> first.
If it works - fine, if not you usually know before you power down to install the new toy, so very little time is lot in a kernel rebuild.
I assume you don't start from make clean.
Further, I upgrade my kernel every change in x where kernels are numbered 3.x.y.
That means when I try out new hardware I am often due a kernel update anyway. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
smartass Apprentice
Joined: 04 Jul 2011 Posts: 189 Location: right behind you ... (you did turn around, didn't you?)
|
Posted: Sun Dec 02, 2012 4:31 pm Post subject: |
|
|
Spent almost the whole day playing with kconfiglib.py, didn't get very far, because the API is !@$#* inconsistent and mostly hidden from the user so getting only the specific kind of a dependency is a lot harder than I thought.
I've seen some Kconfig->XML converter, didn't work. Seems I need something like that, because kconfiglib.py didn't give me direct access to the dependency system.
Anyone know of some other working Kconfig API? At the worst I'll dive into C...
Neddy: Sometimes you just need to test something on the spot and can't restart the computer, because you have other important things connected to or running on it. Otherwise you're right. |
|
Back to top |
|
|
steveL Watchman
Joined: 13 Sep 2006 Posts: 5153 Location: The Peanut Gallery
|
Posted: Sun Dec 02, 2012 7:07 pm Post subject: |
|
|
smartass wrote: | I've seen some Kconfig->XML converter, didn't work. Seems I need something like that, because kconfiglib.py didn't give me direct access to the dependency system.
Anyone know of some other working Kconfig API? At the worst I'll dive into C... |
I was going to suggest this. It'd actually be quite an interesting thing to do.
Quote: | Sometimes you just need to test something on the spot and can't restart the computer, because you have other important things connected to or running on it. Otherwise you're right. |
This seems odd. If it's work situation, why can't you just set aside a machine for hardware testing? I'm not saying you could never use it for anything else, but it should be understood that anyone can get kicked off it, and it'll likely be rebooted to try out something. |
|
Back to top |
|
|
smartass Apprentice
Joined: 04 Jul 2011 Posts: 189 Location: right behind you ... (you did turn around, didn't you?)
|
Posted: Sun Dec 02, 2012 7:20 pm Post subject: |
|
|
Hi Steve,
What were you going to suggest? Diving into C or some XML frontend? I've had a look at the C files and it's not very intuitive, can't find any API-like docs.
I wish I had an always accessible testing machine...well, I don't, so I have to find a different way. |
|
Back to top |
|
|
steveL Watchman
Joined: 13 Sep 2006 Posts: 5153 Location: The Peanut Gallery
|
Posted: Wed Dec 05, 2012 9:21 am Post subject: |
|
|
smartass wrote: | Hi Steve,
What were you going to suggest? Diving into C or some XML frontend? I've had a look at the C files and it's not very intuitive, can't find any API-like docs. |
Yeah, the C stuff. I haven't looked at the code at all, yet, but I'm doing some work on make and build stuff, as well as C programming, so it might be interesting from that perspective, though I'm not sure how much time I would have.
Quote: | I wish I had an always accessible testing machine...well, I don't, so I have to find a different way. |
Why not try with a virtual machine? I've had good results with vmware on gentoo (you just have to download a free serial) even on low-powered hardware, though that was a while ago. |
|
Back to top |
|
|
smartass Apprentice
Joined: 04 Jul 2011 Posts: 189 Location: right behind you ... (you did turn around, didn't you?)
|
Posted: Wed Dec 05, 2012 10:33 am Post subject: |
|
|
Well, AFAIK from looking at the sources the main kconfig routines all spawn from the lkc.h header, using expr.c, util.c, symbol.c. But they lack any consistent docs, or I haven't looked in the right places.
I'm ok with working in C, but only if the API is consistent, hunting down structs and functions in a bad C API is a nightmare. And so far it doesn't look like a consistent one. |
|
Back to top |
|
|
alkan Guru
Joined: 06 Aug 2004 Posts: 385 Location: kasimlar yaylasi
|
Posted: Wed Dec 05, 2012 5:24 pm Post subject: |
|
|
Here what i've been doing with kernel config.
It served me well over the years on different hardware.
For once manually: set everything as "module" and as "yes" if it is not a moduleable option (I admit it takes a lot of effort)
For a new kernel: start from the old config, apply the first step for the new features (this is minimal from one version to the next).
It takes a lot of effort for initial configuration and then a lot of compile time. But it works across machines and across kernel versions. Changes needed are usually minimal (like CPU option with different machines). I've been maintaining my kernels like this since 2004. |
|
Back to top |
|
|
smartass Apprentice
Joined: 04 Jul 2011 Posts: 189 Location: right behind you ... (you did turn around, didn't you?)
|
Posted: Wed Dec 05, 2012 6:05 pm Post subject: |
|
|
alkan wrote: |
For once manually: set everything as "module" and as "yes" if it is not a moduleable option (I admit it takes a lot of effort). |
Are you aware of ? Sounds like it does what you want automatically. |
|
Back to top |
|
|
fpemud Guru
Joined: 15 Feb 2012 Posts: 349
|
Posted: Thu Dec 06, 2012 3:57 am Post subject: |
|
|
I have encountered the same problem and I write a small project fpemud-kernelmanager.
In this project I encapsulate the Kconfiglib.py to a neater API set by which I can select the .config items like I do in make menuconfig. It use user defined rule files to select the .config item automatically (in cfg/rules.d).
This project is in initial state, it only supports .config checking. But it already helped me find some errors in my .config.
I managed to get in touch with the author of Kconfiglib.py who is a very nice and active guy and he is trying to improve it.
I'm learning the markdown language of github, after this I will write a README to fpemud-kernelmanager. |
|
Back to top |
|
|
smartass Apprentice
Joined: 04 Jul 2011 Posts: 189 Location: right behind you ... (you did turn around, didn't you?)
|
Posted: Thu Dec 06, 2012 5:37 am Post subject: |
|
|
Hi fpemud, didn't know you're on Gentoo
I noticed your fork and active discussion on Github.
I myself engaged in discussion with ulfalizer, but as he put it, the API is not capable of what I'm asking for right now. As far as I can see, to support full dependency evaluations the core of the API would have to be redone and ulfalizer is a very busy man.
I think I'll give Cocxml another go, but it fails, it appears to be too strict when it comes to whitespace.
I think I'll look for some lexical analysis tool for Bison or Flex (that's how Kconfig is defined) in Python, PyBison or FBModule look promising. |
|
Back to top |
|
|
fpemud Guru
Joined: 15 Feb 2012 Posts: 349
|
Posted: Thu Dec 06, 2012 10:47 am Post subject: |
|
|
nice to meet you, smartass
Yeah, I'm also having one point unimplemented yet, really unsure how to name it...
I call it "selects calculation", I think it should be the same thing as "dependency evaluations".
Wish I can better describe it in my README.
But to me it's not on the key path, so I can still do many things without it.
I'd like to learn your discussion with ulfalizer, can I see it on web?
I think I still need a better understanding on Kconfig syntax and structure.
Willing to see your work, good luck, buddy. |
|
Back to top |
|
|
smartass Apprentice
Joined: 04 Jul 2011 Posts: 189 Location: right behind you ... (you did turn around, didn't you?)
|
Posted: Thu Dec 06, 2012 12:38 pm Post subject: |
|
|
There was no real discussion with ulfalizer, I just asked what other dependency access there is besides Config.dep[Symbol] and Symbol._get_dependent().
If you want to know more about kconfiglib.py, dive into the source, there are many private methods that don't show up in the docs. You'll also find that instances have many attributes, which can be useful. Sadly, most of this stuff is not well documented and isn't very intuitive.
Dependency evaluations is somethign I'm looking for too.
Good luck |
|
Back to top |
|
|
alkan Guru
Joined: 06 Aug 2004 Posts: 385 Location: kasimlar yaylasi
|
Posted: Fri Dec 07, 2012 6:17 am Post subject: |
|
|
smartass wrote: | alkan wrote: |
For once manually: set everything as "module" and as "yes" if it is not a moduleable option (I admit it takes a lot of effort). |
Are you aware of ? Sounds like it does what you want automatically. |
I am aware, I tried at some time in the past, I can't remember the specifics but it didn't generate a usable config. And i couldn't boot into it. I guess it is not all that intelligent. |
|
Back to top |
|
|
Ulfalizer n00b
Joined: 09 Dec 2012 Posts: 3
|
Posted: Sun Dec 09, 2012 1:15 am Post subject: |
|
|
I'm the author of Kconfiglib (https://github.com/ulfalizer/Kconfiglib) and came across this while Googling.
You're right in that the internal dependency APIs probably aren't very useful. They work the way that is handiest for the implementation, without any concern for what the user might find useful. That's why they're internal though! If there's some functionality that would make your life easier, just suggest it and I might implement it . Consider that defining what e.g. "A requires B" means might be tricky with cases such as "depends on FOO || BAR" though.
I was thinking of adding some APIs for getting expressions, seeing what symbols are in them, walking over them, etc. That would be the most general thing you could do.
Oh, and if some part of the code seems especially cryptic, tell me and I could add some more comments. I consider Kconfiglib pretty clean compared to the scripts/kconfig/*conf utilities, and some of the mess might be because parts of Kconfig are a mess.
Edit: I cleaned up the installation to be minimally invasive. No files need copying anywhere now, and the comments that were added to various files previously were a bit silly; if you've installed the lib, you already know where to find the docs.
Last edited by Ulfalizer on Sun Dec 09, 2012 2:09 am; edited 3 times in total |
|
Back to top |
|
|
Ulfalizer n00b
Joined: 09 Dec 2012 Posts: 3
|
Posted: Sun Dec 09, 2012 1:23 am Post subject: |
|
|
Suggestions for improvements to the existing APIs are welcome too of course. |
|
Back to top |
|
|
smartass Apprentice
Joined: 04 Jul 2011 Posts: 189 Location: right behind you ... (you did turn around, didn't you?)
|
Posted: Sun Dec 09, 2012 9:22 am Post subject: |
|
|
Damn this world is small
Ulfalizer, no offense, but your API isn't well suited for dependency evaluations, that's all.
If you'd want it too, you'd most likely have to reimplement the core to support having tokens as instances and what not.
But that may not be worth it, because you'd be reimplementing the wheel, since there's already the flex/bison Kconfig parser. So maybe it would be better, if we tried to reimplement kconfiglib.py on top of something like pyBison, because then me and fpemud could get what we want and you could get an API that would be faster as it could be based on a C library. Also, if it used some of the C headers for parser definitions, you could be sure it is be compatible with the C implementation of the Kconfig version at hand.
And if it was implemented in such a way, maybe people would consider integrating it into the mainline kernel tree, as it would have a lot more potential with a an API exposing the Bison parsing. |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|