| View previous topic :: View next topic |
| Author |
Message |
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20476
|
 Posted: Sat Oct 26, 2024 3:14 am Post subject: RFC: linux-firmware ebuild patch, include firmware filter |
 |
|
This would replace my old method that altered the upstream "truth" file. That worked for ~2 years, but as of linux-firmware-20241017, it will no longer work.
Before submitting a bug as a proposed change, I'm looking for feedback on what I should fix.
This version requires USE=savedconfig but with NO savedconfig containing a list of firmware. I suspect this isn't what would be implemented, but it's hat I could think to do with existing functionality.
Instead of the current savedconffig file, I'm using a "filter" file that contains one awk pattern per line, later used to select only the firmware to be installed to the image / live file system. This does cause the ebuild to issue the warning about "no savedconfig" file.
Nevertheless, when that condition is true (no savedconfig file) the patch next checks for the existence of the filter file. If present, it "mimics" similar functionality as the old savedconfig file method, but using the awk pattern matching on the firmware to be included.
If adopted, I expect those who maintain the ebuild would do it differently, but I'm working with the constraints of it being a new capability and my lack of knowledge about "the right way" to do things with ebuilds.
My two (tested and working) examples: | /etc/portage/savedconfig/sys-kernel/linux-firmware.filter: | rtl_nic/rtl8(168|411)
amd-ucode/microcode_amd(_fam16h)?.bin |
| /etc/portage/savedconfig/sys-kernel/linux-firmware.filter: | i915/kbl_
iwlwifi-8265 |
The ebuild patch...
I couldn't find how to refer to the savedconfig file using variables, which is similar to how I'm referencing the "filter" file, so that is hard coded. I thought that's what "${S}/${PN}.conf" did, but no.
The if block could be streamlined, but for now, I've chosen to leave the original in place and duplicate some of the settings.
Regarding the method using awk, the find output didn't like being piped to awk, so find is wrapped with redirection. My original method used an actual file in the same position, so maybe it's a requirement rather than relying on a pipe?
sed is used to strip the leading './' path characters from find's output.
It's mostly straightforward if a bit ugly.
(edit) An updated patch for ebuild -r3 is here. | Code: | # diff -u gentoo/sys-kernel/linux-firmware/linux-firmware-20241017-r2.ebuild local/sys-kernel/linux-firmware/linux-firmware-20241017-r2.ebuild
--- gentoo/sys-kernel/linux-firmware/linux-firmware-20241017-r2.ebuild 2024-10-22 12:11:21.000000000 -0600
+++ local/sys-kernel/linux-firmware/linux-firmware-20241017-r2.ebuild 2024-10-25 17:10:22.186148992 -0600
@@ -284,6 +284,7 @@
local FW_OPTIONS=( "-v" )
local files_to_keep=
+ local filter_file="/etc/portage/savedconfig/sys-kernel/linux-firmware.filter"
if use savedconfig; then
if [[ -s "${S}/${PN}.conf" ]]; then
@@ -291,6 +292,14 @@
grep -v '^#' "${S}/${PN}.conf" 2>/dev/null > "${files_to_keep}" || die
[[ -s "${files_to_keep}" ]] || die "grep failed, empty config file?"
FW_OPTIONS+=( "--firmware-list" "${files_to_keep}" )
+ elif [[ -s "${filter_file}" ]]; then
+ files_to_keep="${T}/files_to_keep.lst"
+ pushd "${WORKDIR}/${P}" &>/dev/null || die
+ awk 'NR==FNR{patterns[$1]; next} {for (p in patterns) if ($0 ~ p) print $0}' \
+ "${filter_file}" <(find |sed -e 's:\./::') > "${files_to_keep}" || die
+ popd &>/dev/null || die
+ [[ -s "${files_to_keep}" ]] || die "filter failed, pattern match error?"
+ FW_OPTIONS+=( "--firmware-list" "${files_to_keep}" )
fi
fi |
One thing that may be useful is an example of how to test the patterns. Perhaps a separate script?
_________________
Quis separabit? Quo animo?
Last edited by pjp on Thu Oct 31, 2024 12:20 am; edited 1 time in total |
|
| Back to top |
|
 |
pa4wdh
l33t
Joined: 16 Dec 2005
Posts: 881
|
| Posted: Sat Oct 26, 2024 7:09 am Post subject: |
|
|
I like your idea.
Last time i took the time to decrease the number of firmware files on my system (which i admit is quite some time ago ...) i had to list the files i do not want. The problem is that i don't know what has been added or where versions have changed.
Your method seems to allow to select files i do want, which is much more useful as i know what hardware i have and which files they need. The use of patters allows to write a pattern that matches the current and future versions so you don't have to track updates manually.
_________________
The gentoo way of bringing peace to the world:
USE="-war" emerge --newuse @world
My shared code repository: https://code.pa4wdh.nl.eu.org
Music, Free as in Freedom: https://www.jamendo.com |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20476
|
| Posted: Sat Oct 26, 2024 3:30 pm Post subject: |
|
|
Potentially missing updates with the official method was the original motivator. Something about that just seemed very wrong.
The find statement needs to be fixed to handle files with spaces.
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 22583
|
| Posted: Sat Oct 26, 2024 4:37 pm Post subject: |
|
|
Piping output from find ought to work. How did it fail for you, and what was the invocation?
You can avoid the sed by directing find to print without the leading .: find -printf '%P\n'. |
|
| Back to top |
|
|
Goverp
Advocate
Joined: 07 Mar 2007
Posts: 2167
|
| Posted: Sat Oct 26, 2024 7:04 pm Post subject: |
|
|
IIUC this is so far just a proposal, in which case may I add a couple of suggestions:
a) allow very simple matching by filename, though permitting more extensive if desired. (This could almost be "grep -f <patternsfile>" against tree listing of the firmware directory, but using GLOB rather than regex...
[a SMOP, perhaps just a sed replace "*" with ".*", "?" with "." and "." with "[.]"]
so valid patterns might be: | Code: | *amd_fam17h.bin
amdgpu/*
amdgpu/*polaris10* |
and the | Code: | | /etc/portage/savedconfig/sys-kernel/linux-firmware.filter |
could be a directory and the contents merged, so I might have
linux-firmware.filter/cpu
and
linux-firmware.filter/gpu
I appreciate making suggestions is so much easier than implementing them!
_________________
Greybeard |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20476
|
| Posted: Sat Oct 26, 2024 8:47 pm Post subject: |
|
|
| Hu wrote: | | Piping output from find ought to work. How did it fail for you, and what was the invocation? |
Good question. I didn't make notes about it. I'll try to reproduce it. In general I'm guessing a pipe would be preferred due to pipefail checking.
| Hu wrote: | | You can avoid the sed by directing find to print without the leading .: find -printf '%P\n'. |
Thanks. I almost never think of find's formatted output. I don't think I've ever tried to combine filename safety with formatted output, so that could be interesting.
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20476
|
| Posted: Sat Oct 26, 2024 9:05 pm Post subject: |
|
|
| Goverp wrote: | | IIUC this is so far just a proposal, in which case may I add a couple of suggestions: |
Correct. Thank you.
| Goverp wrote: | a) allow very simple matching by filename, though permitting more extensive if desired. (This could almost be "grep -f <patternsfile>" against tree listing of the firmware directory, but using GLOB rather than regex...
[a SMOP, perhaps just a sed replace "*" with ".*", "?" with "." and "." with "[.]"] |
For now it's tied to awk pattern matching. I don't know what it uses. I suspect some stuff would just work. I should be able to add some additional tests.
Whether or not awk remains is likely to be influenced by whatever prospective maintainers think is a better solution. At the time, awk was the only way I was able identify a solution.
| Goverp wrote: | so valid patterns might be: | Code: | *amd_fam17h.bin
amdgpu/*
amdgpu/*polaris10* |
|
I'll add those for testing. And I'll try to think of a way to use grep. But I think it would be an even uglier solution. Unless I'm missing something obvious.
| Goverp wrote: | and the | Code: | | /etc/portage/savedconfig/sys-kernel/linux-firmware.filter |
could be a directory and the contents merged, so I might have
linux-firmware.filter/cpu
and
linux-firmware.filter/gpu
I appreciate making suggestions is so much easier than implementing them! |
I don't think that would be too difficult, Using one file as I have it now or all files in a directory _shouldn't_ be that much different. But using a directory seems like a bigger change as that affects stuff outside the ebuild, requiring (I presume) different approval and consideration.
Since I'm "abusing" savedconfig, I expect that to need some rework. At a minimum, the error / warning message would need to change.
I'll try to do some of those tests / changes tonight.
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20476
|
| Posted: Sun Oct 27, 2024 3:55 am Post subject: |
|
|
Regarding find with printf, it doesn't appear to have a suitable formatting option: The "default": | Code: | $ find ./rtl_nic/ \! -type d -name rtl8105e-1.fw
./rtl_nic/rtl8105e-1.fw |
with the leading ./ needing to be trimmed: | Code: | $ find ./rtl_nic/ \! -type d -name rtl8105e-1.fw |sed -e 's:^./::'
rtl_nic/rtl8105e-1.fw |
%P removes the directory as well: | Code: | $ find ./rtl_nic/ \! -type d -name rtl8105e-1.fw -printf '%P\n'
rtl8105e-1.fw |
The only other options that seem related: | Code: | $ find ./rtl_nic/ \! -type d -name rtl8105e-1.fw -printf 'f: %f\nh: %h\nH: %H\np: %p\n'
f: rtl8105e-1.fw
h: ./rtl_nic
H: ./rtl_nic/
p: ./rtl_nic/rtl8105e-1.fw |
As for the find | awk issue, it fails silently: | Code: | | $ find ./rtl_nic/ \! -type d |sed -e 's:^./::' | awk 'NR==FNR{patterns[$1]; next} {for (p in patterns) if ($0 ~ p) print $0}' /etc/portage/savedconfig/sys-kernel/linux-firmware.filter |
Compared to: | Code: | $ awk 'NR==FNR{patterns[$1]; next} {for (p in patterns) if ($0 ~ p) print $0}' /etc/portage/savedconfig/sys-kernel/linux-firmware.filter <(find ./rtl_nic/ \! -type d)
./rtl_nic/rtl8411-2.fw
./rtl_nic/rtl8411-1.fw
./rtl_nic/rtl8168h-2.fw
./rtl_nic/rtl8168h-1.fw
./rtl_nic/rtl8168g-3.fw
./rtl_nic/rtl8168g-2.fw
./rtl_nic/rtl8168g-1.fw
./rtl_nic/rtl8168fp-3.fw
./rtl_nic/rtl8168f-2.fw
./rtl_nic/rtl8168f-1.fw
./rtl_nic/rtl8168e-3.fw
./rtl_nic/rtl8168e-2.fw
./rtl_nic/rtl8168e-1.fw
./rtl_nic/rtl8168d-2.fw
./rtl_nic/rtl8168d-1.fw |
And again with sed: | Code: | $ awk 'NR==FNR{patterns[$1]; next} {for (p in patterns) if ($0 ~ p) print $0}' /etc/portage/savedconfig/sys-kernel/linux-firmware.filter <(find ./rtl_nic/ \! -type d |sed -e 's:^./::') |head -1
rtl_nic/rtl8411-2.fw |
As I discovered that use of awk with a filter some time ago, I don't recall what I read about how it worked. I would normally expect to use find | and if that failed, I don't know if I would have guessed that <() would work, so I presume I adapted it from an example.
Taking a chance, searching for awk FNR==NR turned up a possible explanation: | Quote: | This means that the condition NR==FNR is normally only true for the first file, as FNR resets back to 1 for the first line of each file but NR keeps on increasing.
This pattern is typically used to perform actions on only the first file. It works assuming that the first file is not empty, otherwise the two variables would continue to be equal while Awk was processing the second file. |
https://stackoverflow.com/questions/32481877/what-are-nr-and-fnr-and-what-does-nr-fnr-imply/32482115#32482115
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20476
|
| Posted: Sun Oct 27, 2024 5:13 am Post subject: |
|
|
awk doesn't use shell globbing, so the literal equivalent with an asterisk is dot asterisk (.*). | Code: | $ cat ../../temp/cpu.filter
.*amd_fam17h.bin
$ cat ../../temp/gpu.filter
.*polaris10_.*.bin |
I'll try to find another option, but the key with the awk solution is being able to iterate over each pattern.
EDIT: I missed what you meant with the SMOP. Now I understand (grep -f didn't work with GLOB matching either).
Since neither awk nor grep support GLOB matching, it seems like this feature may be for "advanced" users :grin:
More seriously, I question the fragility of trying to turn a GLOB into a regexp.
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3407
|
| Posted: Sun Oct 27, 2024 6:09 am Post subject: |
|
|
If you have a problem, and think "I'll sove it with regex", it means you have 2 problems.
And if you opt to solve it with java, you get a problem factory 
| Quote: | | More seriously, I question the fragility of trying to turn a GLOB into a regexp. |
Shell's "*" means exactly the same as PCRE's ".*" , what's so fragile about it?
I mean, you do have to escape all dots you want treated as literal dots, but it doesn't seem too bad, so what's the actual problem there?
_________________
Make Computing Fun Again |
|
| Back to top |
|
|
Goverp
Advocate
Joined: 07 Mar 2007
Posts: 2167
|
| Posted: Sun Oct 27, 2024 8:30 am Post subject: |
|
|
| szatox wrote: | | ...Shell's "*" means exactly the same as PCRE's ".*" , what's so fragile about it?... |
Actually, AFAIK it means the same as [^/]*
A fragile bit is not so much the conversion of the limited range of globbing special characters so much as preventing the rest of the initial string being interpreted as a regex. For example, files containing a dollar sign
"foo$bah*" will match nothing if interpreted as the regex "foo$bah[^/]*"
Annoyingly, in the dark ages "glob" was a command-line program, which would be quite useful here. But now the only way to get globbing is through a programming language, which is a pain.
_________________
Greybeard |
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3407
|
| Posted: Sun Oct 27, 2024 9:59 am Post subject: |
|
|
> "foo$bah*" will match nothing if interpreted as the regex
True, but we know that in advance, and if we want to match files with $, we just escape it in the pattern, like we did with the dot to prevent it from matching everything:
'foo\$bah'
'foo.bah'
'fo.{3}ah'
Generating patterns from arbitrary input would certainly be a pain, but I suppose even in this case escaping special characters would be the easy part.
I though we're looking for files based on patterns written manually in advance, and in this scenario it doesn't look that bad.
> Actually, AFAIK it means the same as [^/]*
Maybe, I can't think of a context in which this distinction would matter right now, but when I find one I bet it will be implementation dependent anyway. It's still within the realm of "replace THIS literal string with THAT literal string and the OTHER thing will reliably produce exactly the SAME result".
Shells' globe pattern syntax uses slightly fewer characters, but I don't really see it being less fragile than PCRE
_________________
Make Computing Fun Again |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 22583
|
| Posted: Sun Oct 27, 2024 2:04 pm Post subject: |
|
|
| pjp wrote: | Regarding find with printf, it doesn't appear to have a suitable formatting option: The "default": | Code: | $ find ./rtl_nic/ \! -type d -name rtl8105e-1.fw
./rtl_nic/rtl8105e-1.fw |
with the leading ./ needing to be trimmed: | Code: | $ find ./rtl_nic/ \! -type d -name rtl8105e-1.fw |sed -e 's:^./::'
rtl_nic/rtl8105e-1.fw |
%P removes the directory as well: | Code: | $ find ./rtl_nic/ \! -type d -name rtl8105e-1.fw -printf '%P\n'
rtl8105e-1.fw |
|
%P is documented as: | info find: | '%P'
File's name with the name of the command line argument under which
it was found removed from the beginning. |
In your original ebuild patch, the command line argument was an implicit ., so removing it was fine. I replicated that implicit . in my suggestion. In your tests here, you used an explicit command line argument, so yes, you got an undesirable result in your response post where that explicit subdirectory was removed. I think %P will work for the code you showed in the opening post.
| pjp wrote: | | As for the find | awk issue, it fails silently: |
When given a filename on the command line, awk ignores stdin. If you want to read both a file and stdin, you need to tell awk this explicitly. You can use the placeholder - to refer to stdin. See info gawk 'Naming Standard Input' for details. |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20476
|
| Posted: Mon Oct 28, 2024 2:03 am Post subject: |
|
|
| szatox wrote: | If you have a problem, and think "I'll sove it with regex", it means you have 2 problems.
And if you opt to solve it with java, you get a problem factory :lol: |
Sure ;). I'm not aware of another possibility to the "match things that look like X, allowing for variations" problem.
| szatox wrote: | | Quote: | | More seriously, I question the fragility of trying to turn a GLOB into a regexp. |
Shell's "*" means exactly the same as PCRE's ".*" , what's so fragile about it?
I mean, you do have to escape all dots you want treated as literal dots, but it doesn't seem too bad, so what's the actual problem there? |
Perhaps I misunderstood Goverp. | Goverp wrote: | [...] but using GLOB rather than regex...
[a SMOP, perhaps just a sed replace "*" with ".*", "?" with "." and "." with "[.]"] |
I read that as a suggestion to use sed to "convert" from one matching method to another. First, doing that in an ebuild seems wrong. Second, I don't know the differences between GLOB and regex well enough to write a conversion function..
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20476
|
| Posted: Mon Oct 28, 2024 2:47 am Post subject: |
|
|
@Hu,
Thank you. | Code: | $ find \! -type d -printf '%P\n' | awk 'NR==FNR{patterns[$1]; next} {for (p in patterns) if ($0 ~ p) print $0}' ../../temp/nic.filter -
rtl_nic/rtl8411-2.fw
rtl_nic/rtl8411-1.fw
... |
Although now that I think of it, I should probably leave off the directory exclusion.
Next is dealing with spaces in filenames for the pattern that needs to be matched. As far as I can tell, that would require the pattern to contain [[:space:]] as in: | Code: | $ cat ../../temp/foobar.filter
foobar/foo[[:space:]]bar(1|3) |
Maybe a wiki with a "mini manual" would suffice.
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3407
|
| Posted: Mon Oct 28, 2024 10:00 am Post subject: |
|
|
| pjp wrote: | | I read that as a suggestion to use sed to "convert" from one matching method to another. First, doing that in an ebuild seems wrong. Second, I don't know the differences between GLOB and regex well enough to write a conversion function.. |
Nevermind, I missed that detail and it was quite important.
_________________
Make Computing Fun Again |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20476
|
| Posted: Thu Oct 31, 2024 12:16 am Post subject: |
|
|
Updated with the -r3 version of the ebuild.
awk and grep updated and tested. Since grep is less complicated, I've commented out the awk version, presuming simplicity would be preferred. | diff -u: | --- gentoo/sys-kernel/linux-firmware/linux-firmware-20241017-r3.ebuild 2024-10-26 13:11:01.000000000 -0600
+++ local/sys-kernel/linux-firmware/linux-firmware-20241017-r3.ebuild 2024-10-29 13:10:13.866405222 -0600
@@ -288,6 +288,7 @@
local FW_OPTIONS=( "-v" )
local files_to_keep=
+ local filter_file="/etc/portage/savedconfig/sys-kernel/linux-firmware.filter"
if use savedconfig; then
if [[ -s "${S}/${PN}.conf" ]]; then
@@ -295,6 +296,16 @@
grep -v '^#' "${S}/${PN}.conf" 2>/dev/null > "${files_to_keep}" || die
[[ -s "${files_to_keep}" ]] || die "grep failed, empty config file?"
FW_OPTIONS+=( "--firmware-list" "${files_to_keep}" )
+ elif [[ -s "${filter_file}" ]]; then
+ files_to_keep="${T}/files_to_keep.lst"
+ pushd "${WORKDIR}/${P}" &>/dev/null || die
+ set -o pipefail
+# find -printf "%P\n" \
+# | awk 'NR==FNR{patterns[$1]; next} {for (p in patterns) if ($0 ~ p) print $0}' "${filter_file}" - > "${files_to_keep}"
+ find -printf "%P\n" | grep -E -f "${filter_file}" > "${files_to_keep}"
+ popd &>/dev/null || die
+ [[ -s "${files_to_keep}" ]] || die "filter failed, pattern match error?"
+ FW_OPTIONS+=( "--firmware-list" "${files_to_keep}" )
fi
fi
|
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
