Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Why try and re-invent the wheel? [Solved]
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
yldouright
n00b
n00b


Joined: 29 Mar 2016
Posts: 6

PostPosted: Tue Mar 29, 2016 11:58 am    Post subject: Why try and re-invent the wheel? [Solved] Reply with quote

I know there must be some on this forum who have successfully built custom kernels or compile scripts for standardized hardware from Dell, HP and Lenovo. I think it might be a good idea to collect all these in this thread for people to try them out and evaluate them against each other. As an example, here are some common laptops that have been around a while:

Dell E6500, E6510, E6520, M4400, M4500, M4600
HP Elitebook 8530W, 8540W, 8560W
Lenovo W500, W510, W520

Has anyone here done custom compiles for any of the above?


Last edited by yldouright on Wed Mar 30, 2016 1:40 pm; edited 1 time in total
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6227
Location: Room 101

PostPosted: Tue Mar 29, 2016 2:37 pm    Post subject: Re: Why try and re-invent the wheel? Reply with quote

yldouright wrote:
Has anyone here done custom compiles for any of the above?

yldouright ... that is effectively what every user is doing when installing gentoo regardless of the hardware (though the degree of customisation may differ).

As for "custom kernels" I assume you mean .config files ... these tend to be big, and so not suitable for posting on the forum (and pastebin/gist files tend to expire quickly), so, a thread to act as a clearing house for such things is probably not such a good idea.

best ... khay
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6920

PostPosted: Tue Mar 29, 2016 3:17 pm    Post subject: Reply with quote

The Linux kernel already has `make localmodconfig`
Back to top
View user's profile Send private message
yldouright
n00b
n00b


Joined: 29 Mar 2016
Posts: 6

PostPosted: Tue Mar 29, 2016 4:59 pm    Post subject: Reply with quote

Thanks for the input. I'm a complete Gentoo noob so please forgive my possibly silly questions. Am I to understand that the 'make localconfig' command will optimize the kernel automatically without having to define the hardware and tick any flags?
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6227
Location: Room 101

PostPosted: Tue Mar 29, 2016 5:38 pm    Post subject: Reply with quote

yldouright wrote:
Am I to understand that the 'make localconfig' command will optimize the kernel automatically without having to define the hardware and tick any flags?

yldouright ... no, it won't do that ... from the kernel README

README wrote:
"make localmodconfig" Create a config based on current config and loaded modules (lsmod). Disables any module option that is not needed for the loaded modules.

best ... khay
Back to top
View user's profile Send private message
yldouright
n00b
n00b


Joined: 29 Mar 2016
Posts: 6

PostPosted: Tue Mar 29, 2016 5:56 pm    Post subject: Reply with quote

khayyam
The link was helpful. Is there any reason why why these localmodconfig files couldn't be linked in this thread? I would be happy just to have config files for the hardware in the opening post. The demands on bandwidth is not likely to overshadow the benefit to this website.
Back to top
View user's profile Send private message
Roman_Gruber
Advocate
Advocate


Joined: 03 Oct 2006
Posts: 3846
Location: Austro Bavaria

PostPosted: Tue Mar 29, 2016 6:30 pm    Post subject: Reply with quote

Well we have the genkernel package for the lazy guys and kernel seeds.

As things evolve there is no thing as static config anymore.

I hardly found anyone in the past on this forum with my hardware, although i buy prebuild notebooks always.

There are hardly guys on this forum with the same hardware

there is hardly a demand for a standardized config

and we have genkernel which is a generic kernel,

maybe those binary kernel can be abused / reused for htis purpose.

A kernel seed is a big mess to maintain, as every new kernel release introduce / drop config options and the user space always evolve.

we are not windi 95 platform where the same software is used for 20 years.

--

Use genkernel and your kernel trouble may be solved / use the kernel from sysrescue-cd
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6227
Location: Room 101

PostPosted: Tue Mar 29, 2016 6:41 pm    Post subject: Reply with quote

yldouright wrote:
The link was helpful. Is there any reason why why these localmodconfig files couldn't be linked in this thread? I would be happy just to have config files for the hardware in the opening post. The demands on bandwidth is not likely to overshadow the benefit to this website.

yldouright ... sure, if people provided them (or more tailored config's), the problem is that these links (and the config files themselves) tend to get stale, or removed (as the services for such things tend to have a short life span). A much better approach imo is the (unfortunately defunct ... but still useful) kernel seeds.

best ... khay
Back to top
View user's profile Send private message
yldouright
n00b
n00b


Joined: 29 Mar 2016
Posts: 6

PostPosted: Tue Mar 29, 2016 7:36 pm    Post subject: Reply with quote

tw04l124
I look at this as addressing the inefficiency of duplicate effort and not as an act of laziness. Ideally, you want a configuration script which will acoomodate a range of values for each component group. For example:

1. a script which optimizes for processors with the same socket so that you can take advantage of all the features available to that CPU but tick off the features that aren't available.
2. a script which creates a kernel that optimizes its operation during boot up for a range of memory bandwidth configurations.
3. a script which creates a kernel that dynamically optimizes the disk I/Oat boot up for access time and bandwidth.

These seem to be very doable and if that is so, why couldn't the same kind of systemic customization be done for standardized hardware like that described in the first post of this thread?
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2569

PostPosted: Tue Mar 29, 2016 7:46 pm    Post subject: Reply with quote

Kernel-seeds was awesome, but really nothing beats keeping track of things yourself.

I maintain a script and two files. They check the config I'm building for values I want, and prints any line not found and any line whose value is not what I expect.

kcheck:
Code:

#!/bin/bash

echo "Watched:"
cat .config \
| grep -f /root/kernel.grepf \
| grep -vf /root/kernel.grepv

echo "Not Found:"
cat .config \
| grep -Lf /root/kernel.grepf


kernel.grepf starts out like this:
Code:

CONFIG_64BIT\>
CONFIG_9P_FS\>
CONFIG_9P_FS_POSIX_ACL\>
CONFIG_BRIDGE_EBT_MARK_T\>
CONFIG_BRIDGE_EBT_T_NAT\>
CONFIG_BRIDGE_NF_EBTABLES\>


and kernel.grepv starts out like this:
Code:

CONFIG_64BIT=y
CONFIG_9P_FS_POSIX_ACL=y
CONFIG_9P_FS=y
CONFIG_BRIDGE_EBT_MARK_T=y
CONFIG_BRIDGE_EBT_T_NAT=y
CONFIG_BRIDGE_NF_EBTABLES=y



  1. I need to be in /usr/src/next (my symbolic link to my next kernel source directory).
  2. zcat /proc/config.gz > .config
  3. make nconfig
  4. Edit things like localsuffix which I always change.
  5. kcheck
  6. ...
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2569

PostPosted: Tue Mar 29, 2016 7:50 pm    Post subject: Reply with quote

Note that I don't keep every last little option in there, only my deal-breakers.

Any hardware support I actually rely on, any functionality I pay attention to.

I suppose you could build these up to be complete, but the more complete they are the higher maintenance they become. The kernel config is a fluid space. Each new kernel changes some things around as well as adding new functionality. If the kernel config changes one of the values you have in these files, a line is printed and you have to go figure out what happened. It may be a setting wrong or it may be something that changed in the kernel.
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6227
Location: Room 101

PostPosted: Tue Mar 29, 2016 8:02 pm    Post subject: Reply with quote

yldouright wrote:
1. a script which optimizes for processors with the same socket so that you can take advantage of all the features available to that CPU but tick off the features that aren't available.
2. a script which creates a kernel that optimizes its operation during boot up for a range of memory bandwidth configurations.
3. a script which creates a kernel that dynamically optimizes the disk I/Oat boot up for access time and bandwidth.

yldouright ... sorry, what is it that these scripts are acting on? Take note of the following:

grep 'DO NOT EDIT' /usr/src/linux/.config:
# Automatically generated file; DO NOT EDIT.

So you can't simply parse .config and add/remove entires ... you basically need to run 'make <something>config' to generate the .config, see the documentation here.

best ... khay
Back to top
View user's profile Send private message
yldouright
n00b
n00b


Joined: 29 Mar 2016
Posts: 6

PostPosted: Tue Mar 29, 2016 9:52 pm    Post subject: Reply with quote

khayyan
I think I get what you're saying, the parameters need to be input before you can compile or there is no optimization. There has to be some utility that can output all the relevant hardware specs of the system in a form readable by the compiler. If there isn't one, why hasn't someone made it? There is no reason why this process shouldn't be automated. Why can't a basic kernel dynamically compile the memory and disk subsystem at boot time? I remember a little program on the classic mac called silverlining which ran sample read and write scripts to disk to sample different drivers. You chose the best performing driver after these short tests which took about 3 minutes on a 68020 processor with drives of that era. The drivers you selected from were optimized for disk access, latency and throughput and the results were dramatic in some cases with a 260% speed improvement! Given the power of processors today, it seems plausible that these kind of optimizations could be done at boot time. I have nowhere near the skills to do something like this but my gut tells me it's doable and worth doing.

1clue
Thanks.
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6920

PostPosted: Tue Mar 29, 2016 10:03 pm    Post subject: Reply with quote

yldouright wrote:
I think I get what you're saying, the parameters need to be input before you can compile or there is no optimization. There has to be some utility that can output all the relevant hardware specs of the system in a form readable by the compiler. If there isn't one, why hasn't someone made it?

GCC has had -march=native for about a decade. Portage has a script to convert /proc/cpuinfo to correct USE flags. The Linux kernel itself already makes it brain-dead simple to convert a running system from a livecd to a permanent, optimized configuration with all necessary drivers as I already mentioned. What wheel are you trying to invent that we don't already have?

Quote:
Why can't a basic kernel dynamically compile the memory and disk subsystem at boot time? [...] Given the power of processors today, it seems plausible that these kind of optimizations could be done at boot time.

It's called an initramfs... you've already used one if you've ever booted a livecd.
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2569

PostPosted: Tue Mar 29, 2016 11:22 pm    Post subject: Reply with quote

yldouright wrote:
khayyan
I think I get what you're saying, the parameters need to be input before you can compile or there is no optimization. There has to be some utility that can output all the relevant hardware specs of the system in a form readable by the compiler. If there isn't one, why hasn't someone made it? There is no reason why this process shouldn't be automated. Why can't a basic kernel dynamically compile the memory and disk subsystem at boot time? I remember a little program on the classic mac called silverlining which ran sample read and write scripts to disk to sample different drivers. You chose the best performing driver after these short tests which took about 3 minutes on a 68020 processor with drives of that era. The drivers you selected from were optimized for disk access, latency and throughput and the results were dramatic in some cases with a 260% speed improvement! Given the power of processors today, it seems plausible that these kind of optimizations could be done at boot time. I have nowhere near the skills to do something like this but my gut tells me it's doable and worth doing.

1clue
Thanks.


yldouright,

I had a mac in that era. I remember silverlining. There's one fairly important thing you're forgetting.

Apple, especially at that time, had very few hardware options. Hardware manufacturers who built for Mac had to support the Apple Way. Programmers had to follow the standard API. I know this because I was a programmer for this system, and I owned the set of books describing everything.

Right now any given vendor may have more PC-compatible hardware than Mac OS had in its entire supported universe at that time, without even counting things which might be obsolete by now, and there are thousands of vendors.


Getting back to your premise, it's certainly possible to create a Linux kernel with everything turned on, but it's not desirable to do so for most people who run Gentoo. Typical Linux distros will support the most likely hardware scenarios and turn everything else off. Gentoo is aimed at people who want to do it their own way, which means we generally don't take the defaults on anything.

If one wants a secure system, one very good way of preventing some subsystem from being exploited is to remove all support for that subsystem from the operating system. If there is no software it cannot be exploited. This starts with drivers and extends throughout all the software on the disk. Generally speaking, a secure system is purpose-built for one task and everything not related to that task is removed.

Moving on to other things, many of the kernel options are to support older protocols, which depends not only on what hardware you have on your Linux box but also what sort of equipment is hooked to your network. Why should I add in support for an antiquated version of NFS if I have nothing using it? Why should I support Amiga hardware if I don't have any Amiga hardware and don't plan on getting any?

Good luck and have fun.
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6227
Location: Room 101

PostPosted: Wed Mar 30, 2016 6:39 am    Post subject: Reply with quote

yldouright wrote:
I think I get what you're saying, the parameters need to be input before you can compile or there is no optimization. There has to be some utility that can output all the relevant hardware specs of the system in a form readable by the compiler. If there isn't one, why hasn't someone made it?

yldouright ... well, no, not exactly, its not about "optimization" but about driver selection, and the dependencies of those selections. There is a method of doing this, its the build system of the kernel (which uses 'make'). So, "all the relevent hardware specs" is a misnomer, because that system isn't aware of the hardware, its only concerned with how to generate a consistent .config.

yldouright wrote:
There is no reason why this process shouldn't be automated. Why can't a basic kernel dynamically compile the memory and disk subsystem at boot time?

Because that involves the bootstraping problem ... the kernel doesn't include a 'compiler', and for a compiler to function you would need a memory, and possibly disk, subsystem. The kernel's basic remit is to boot, and to initialise the hardware in such a way that subsequent processes can use that hardware, and in that regard its bound by the process that went into generating that kernel (so, driver selection, and so on).

yldouright wrote:
I remember a little program on the classic mac called silverlining which ran sample read and write scripts to disk to sample different drivers. You chose the best performing driver after these short tests which took about 3 minutes on a 68020 processor with drives of that era. The drivers you selected from were optimized for disk access, latency and throughput and the results were dramatic in some cases with a 260% speed improvement! Given the power of processors today, it seems plausible that these kind of optimizations could be done at boot time. I have nowhere near the skills to do something like this but my gut tells me it's doable and worth doing.

As 1clue points out above, that is a lot easier to do when the target is fairly limited in number, this is not the case for the linux kernel, it runs on a wide variety of machine types, architectures, hardware. Also, "done at boot time" involves something which is easy to conceptualise, but very complex to design and make work.

best ... khay
Back to top
View user's profile Send private message
yldouright
n00b
n00b


Joined: 29 Mar 2016
Posts: 6

PostPosted: Wed Mar 30, 2016 1:40 pm    Post subject: Reply with quote

khayyan and 1clue
You've both put me on a steep learning curve I may not be able to climb. I thank you both for your attention.
Back to top
View user's profile Send private message
Tony0945
Watchman
Watchman


Joined: 25 Jul 2006
Posts: 5127
Location: Illinois, USA

PostPosted: Wed Mar 30, 2016 2:27 pm    Post subject: Reply with quote

yldouright wrote:
khayyan and 1clue
You've both put me on a steep learning curve I may not be able to climb. I thank you both for your attention.


Many good comments on this thread. The "press a button and it gets auto-magically done" software that you seem to be looking for would be very time consuming to build and test. I'm not sure that it's even desirable. However, I do think you have a good suggestion that the wiki could have a listing of CONFIG's for various CPU's and maybe a wiki article on what can be shut off for a PC. Because a linux kernel contains support for many many systems besides PCs. For a lot of years, I carried support for using a PC as a router needlessly in my kernel, unknowingly. Yet I'm glad it's there in the kernel because now I'm thinking about using an old PC as a Gentoo router because the major manufacturers are sticking spyware into their routers.

But a procedure for the less knowledgeable/not technically inclined/newbie:
    1. start with either the sysrescuecd kernel or the genkernel default for your architecture. Note that sysrescuecd has most options as modules, that's good.
    2. from your running kernel run "make localmodconfig" this will eliminate the stuff you don't have.
    3. run "make menuconfig" or "genkernel --menuconfig" and look through the file systems and decide what you want or don't want"
    4. run "lspci" and try to determine what hardware you have in terms of auxiliary chips for temperature monitoring and such.


I might add "5. if you have something oddball that doesn't work, post a help request here"

It IS a shame that the major mobo manufacturers that have the resources, don't have one or two experts on staff to develop and maintain downloadable Linux kernels for their products
on their websites. I would definitely prefer that manufacturer. A case in point is the ubiqutous Realtek LAN controller which is so heavily customized more a concept than a hardware. The team that designs their Windows drivers could have one or two guys who provide the Linux driver, even as a blob.
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2569

PostPosted: Wed Mar 30, 2016 4:36 pm    Post subject: Reply with quote

It might be interesting to have a wiki or handbook tutorial to see what hardware is on your box, verify kernel options for it and then look at what portage configuration might be dictated from it.

Re: Realtek. I have some hardware yet with Realtek nics. As soon as somebody pointed out the downfalls of these (high interrupt rates, high cpu overhead) I stopped using them and went with Intel. Things work much more smoothly with Intel nics. The LAN throughput is the same, but Realtek throws a lot more interrupts at the CPU than Intel does, which slows down everything else. If you have high cpu load you might choose Intel. They're only slightly more expensive.

Edit: I also noticed that if you buy only motherboards with Intel nics built in, they tend to have higher quality components in general and work more easily with Linux.
Back to top
View user's profile Send private message
Tony0945
Watchman
Watchman


Joined: 25 Jul 2006
Posts: 5127
Location: Illinois, USA

PostPosted: Wed Mar 30, 2016 9:51 pm    Post subject: Reply with quote

1clue wrote:
It might be interesting to have a wiki or handbook tutorial to see what hardware is on your box, verify kernel options for it and then look at what portage configuration might be dictated from it.

Re: Realtek. I have some hardware yet with Realtek nics. As soon as somebody pointed out the downfalls of these (high interrupt rates, high cpu overhead) I stopped using them and went with Intel. Things work much more smoothly with Intel nics. The LAN throughput is the same, but Realtek throws a lot more interrupts at the CPU than Intel does, which slows down everything else. If you have high cpu load you might choose Intel. They're only slightly more expensive.

Edit: I also noticed that if you buy only motherboards with Intel nics built in, they tend to have higher quality components in general and work more easily with Linux.


Thanks, 1clue. I have done that with two PC's. The other two have older Realtek chips (RTL111?) that work fine with the tulip driver. It does irk me to pay for a mobo with LAN then pay $35-$40 for an ethernet card also.

I do look for boards with Intel on them but they seem to be gaming boards with two or three x16 slots for additional GPU's that could better be used for other boards.

I'm always open to suggestions for boards well supported by Linux.
Back to top
View user's profile Send private message
Proinsias
Tux's lil' helper
Tux's lil' helper


Joined: 06 Oct 2014
Posts: 136
Location: Scotland

PostPosted: Wed Mar 30, 2016 11:47 pm    Post subject: Reply with quote

modprobed-db in Arch keeps track of modules loaded over time. It was a lazy way for me to run slimmer kernels and avoid interacting with the linux kernel config.
Back to top
View user's profile Send private message
1clue
Advocate
Advocate


Joined: 05 Feb 2006
Posts: 2569

PostPosted: Thu Mar 31, 2016 12:26 am    Post subject: Reply with quote

Tony0945 wrote:
1clue wrote:
It might be interesting to have a wiki or handbook tutorial to see what hardware is on your box, verify kernel options for it and then look at what portage configuration might be dictated from it.

Re: Realtek. I have some hardware yet with Realtek nics. As soon as somebody pointed out the downfalls of these (high interrupt rates, high cpu overhead) I stopped using them and went with Intel. Things work much more smoothly with Intel nics. The LAN throughput is the same, but Realtek throws a lot more interrupts at the CPU than Intel does, which slows down everything else. If you have high cpu load you might choose Intel. They're only slightly more expensive.

Edit: I also noticed that if you buy only motherboards with Intel nics built in, they tend to have higher quality components in general and work more easily with Linux.


Thanks, 1clue. I have done that with two PC's. The other two have older Realtek chips (RTL111?) that work fine with the tulip driver. It does irk me to pay for a mobo with LAN then pay $35-$40 for an ethernet card also.

I do look for boards with Intel on them but they seem to be gaming boards with two or three x16 slots for additional GPU's that could better be used for other boards.

I'm always open to suggestions for boards well supported by Linux.


My current favorite is http://www.supermicro.com. It might be a little too high-end for most desktop needs. Dual socket e5-2600-based maybe.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum