| View previous topic :: View next topic |
| Author |
Message |
brendlefly62
Apprentice
Joined: 19 Dec 2009
Posts: 154
|
 Posted: Wed Jan 08, 2025 3:06 pm Post subject: (Solved) Can't find udev executable - but it's there, linked |
 |
|
Bottom line up front: links to /bin/udevadm are present in /lib/systemd/ and /usr/lib/systemd/ but they don't test as executable the way /etc/init.d/udev is looking at them -- so I had to manually add a link to udevadm in /sbin as a workaround. Is this a known bug? not a bug? Anyone else encounter this and fix it a different way?
I converted a well-running, frequently updated 10 yr old core i7-3537U system from ACCEPT_KEYWORDS="~amd64" to ACCEPT_KEYWORDS="amd64" and ran a global update with several downgrades. After rebooting, my user account got error messages about not having permission to /dev/null -- indicating a possible udev problem.
rc-status indicated the udev service was in the sysinit runlevel, but it was "stopped," as was udev-trigger. I tried to restart udev with | Code: | # /etc/init.d/udev start
Unable to find udev executable.
|
Examining the content of /etc/init.d/udev, I noted that this output comes from this function
| Code: | get_udevd_binary() {
local bins
bins="/sbin/udevd /lib/systemd/systemd-udevd /usr/lib/systemd/systemd-udevd"
for f in ${bins}; do
if [ -x "$f" ]; then
command="$f"
fi
done
if [ -z "$command" ]; then
eerror "Unable to find udev executable."
return 1
fi
}
|
I inspected the associated directories of these "$bins" and found links in the systemd directories --
| Code: | for f in ${bins}; do echo "----[ $f ]----"; ls -al $(dirname $f)/ | grep $(basename $f); done
----[ /sbin/udevd ]----
----[ /lib/systemd/systemd-udevd ]----
lrwxrwxrwx 1 root root 17 Dec 2 20:30 systemd-udevd -> ../../bin/udevadm
----[ /usr/lib/systemd/systemd-udevd ]----
lrwxrwxrwx 1 root root 17 Dec 2 20:30 systemd-udevd -> ../../bin/udevadm |
I ran this same test as the init.d script, but manually, and Igot these results
| Code: | for f in ${bins}; do if [ -x "$f" ]; then echo true; else echo false; fi; done
false
false
false |
I googled and looked in gentoo bugs and forums, and I found this reference --
https://forums.gentoo.org/viewtopic-t-1145996-start-0.html
At first I thought it did not apply to me, because the thread of comments indicates this user's problem stemmed from having systemd in an INSTALL_MASK -- but I do not. Nonetheless, I tried adding a link in /sbin and re-running the test -
| Code: | # cd /sbin/
# ln -s /bin/udevadm udevd
# for f in ${bins}; do if [ -x "$f" ]; then echo true; else echo false; fi; done
true
false
false
|
I rebooted, and everything seems fine.
My question is whether this is a bug -- the systemd links do not evaluate true when tested with [ -x
or whether something about my system's history put me in an unusual situation - any advice?
Last edited by brendlefly62 on Sat Jan 11, 2025 6:17 pm; edited 1 time in total |
|
| Back to top |
|
 |
Hu
Administrator
Joined: 06 Mar 2007
Posts: 22976
|
| Posted: Wed Jan 08, 2025 3:56 pm Post subject: |
|
|
The test seems to do the right thing here on an openrc no-merged-usr system. | Code: | $ bins="/sbin/udevd /lib/systemd/systemd-udevd /usr/lib/systemd/systemd-udevd"
$ for f in ${bins}; do
if [ -x "$f" ]; then echo true; else echo false; fi; done
false
true
false
$ namei -l $bins
f: /sbin/udevd
drwxr-xr-x root root /
drwxr-xr-x root root sbin
udevd - No such file or directory
f: /lib/systemd/systemd-udevd
drwxr-xr-x root root /
drwxr-xr-x root root lib
drwxr-xr-x root root systemd
lrwxrwxrwx root root systemd-udevd -> ../../bin/udevadm
drwxr-xr-x root root ..
drwxr-xr-x root root ..
drwxr-xr-x root root bin
-rwxr-xr-x root root udevadm
f: /usr/lib/systemd/systemd-udevd
drwxr-xr-x root root /
drwxr-xr-x root root usr
drwxr-xr-x root root lib
drwxr-xr-x root root systemd
systemd-udevd - No such file or directory |
I think the udevd name is older. udevadm is the name that systemd-udevd prefers now. I think the links ought to evaluate as true for [ -x ] when the target exists and is executable. They do for me. They clearly did not for you. How does your namei -l output compare to mine? Are you on split-usr or merged-usr? |
|
| Back to top |
|
|
bstaletic
Guru
Joined: 05 Apr 2014
Posts: 453
|
| Posted: Wed Jan 08, 2025 4:37 pm Post subject: |
|
|
| brendlefly62 wrote: | | I ran this same test as the init.d script, but manually, and Igot these results |
Init scripts use openrc-run as the interpreter. I have a feeling you're not using that or bash as your interactive shell.
For me, the test works in bash, but not in zsh. |
|
| Back to top |
|
|
brendlefly62
Apprentice
Joined: 19 Dec 2009
Posts: 154
|
| Posted: Wed Jan 08, 2025 6:06 pm Post subject: |
|
|
Thanks Hu, bstaletic, my initial instinct was that the "reversion" from ~adm64 to amd64 and associated downgrades and subsequent emerge --depclean removed something... now, more specifically, I think the problem is that the systemd links in the /etc/init.d/udev script point to /usr/bin whereas the executable udevadm is actually in /bin -- not important on my newer merged-usr systems, but important on the layout of this older system ... so is this a bug in the init.d/udev script then? or do I have the wrong layout for /lib/systemd because I need to add an explicit "split-usr" USE flag somewhere? (see my note about USE at the bottom of this current post)
bstaletic:
| Quote: | | Init scripts use openrc-run as the interpreter. I have a feeling you're not using that or bash as your interactive shell. |
I'm running bash -
| Code: | echo "shell: $SHELL" && echo "shell process:" && ps -p $$
shell: /bin/bash
shell process:
PID TTY TIME CMD
4490 pts/0 00:00:00 bash
# for f in ${bins}; do echo "----[ $f ]----"; ls -al $(dirname $f) | grep $(basename $f); if [ -x "$f" ]; then echo true; else echo false; fi; done
----[ /sbin/udevd ]----
lrwxrwxrwx 1 root root 12 Jan 8 08:50 udevd -> /bin/udevadm
true
----[ /lib/systemd/systemd-udevd ]----
false
----[ /usr/lib/systemd/systemd-udevd ]----
lrwxrwxrwx 1 root root 17 Dec 2 20:30 systemd-udevd -> ../../bin/udevadm
false
|
Hu:
Here's my name -l output --
| Code: | # namei -l $bins
f: /sbin/udevd
drwxr-xr-x root root /
drwxr-xr-x root root sbin
lrwxrwxrwx root root udevd -> /bin/udevadm
drwxr-xr-x root root /
drwxr-xr-x root root bin
-rwxr-xr-x root root udevadm
f: /lib/systemd/systemd-udevd
drwxr-xr-x root root /
drwxr-xr-x root root lib
lrwxrwxrwx root root systemd -> ../usr/lib/systemd
drwxr-xr-x root root ..
drwxr-xr-x root root usr
drwxr-xr-x root root lib
drwxr-xr-x root root systemd
lrwxrwxrwx root root systemd-udevd -> ../../bin/udevadm
drwxr-xr-x root root ..
drwxr-xr-x root root ..
drwxr-xr-x root root bin
udevadm - No such file or directory
f: /usr/lib/systemd/systemd-udevd
drwxr-xr-x root root /
drwxr-xr-x root root usr
drwxr-xr-x root root lib
drwxr-xr-x root root systemd
lrwxrwxrwx root root systemd-udevd -> ../../bin/udevadm
drwxr-xr-x root root ..
drwxr-xr-x root root ..
drwxr-xr-x root root bin
udevadm - No such file or directory |
I note that name -l thinks the target ( ../../bin/udevadm ) does not exist, but it does (as the output of name -l demonstrates for the /sbin/udevd link that I created)
I also note that, on this system, /lib/systemd is a link to ../usr/lib/systemd (so it doesn't help a non-merged-system find the executable in /bin)
| Code: | # tree -d -L 2 /lib
/lib
|-- dhcpcd
| `-- dhcpcd-hooks
...
|-- sysctl.d
|-- systemd -> ../usr/lib/systemd
`-- udev -> ../usr/lib/udev
|
but /usr/lib/systemd is not --
# tree -d -L 2 /usr/lib
/usr/lib
| Code: | |-- audit
|-- clang
| `-- 18
|-- cmake
| `-- nghttp3
...
|-- systemd
| |-- network
| |-- ntp-units.d
| |-- system
| `-- user
...
|
This shows me the link in /usr/lib/systemd, but it is in blinking, indicating the target is not on the system --
| Code: | # ls -al /usr/lib/systemd/
total 24
drwxr-xr-x 6 root root 4096 Jan 8 08:30 .
drwxr-xr-x 34 root root 4096 Jan 7 20:56 ..
drwxr-xr-x 2 root root 4096 Dec 26 17:57 network
drwxr-xr-x 2 root root 4096 Mar 8 2024 ntp-units.d
drwxr-xr-x 2 root root 4096 Jan 7 22:00 system
lrwxrwxrwx 1 root root 17 Dec 2 20:30 systemd-udevd -> ../../bin/udevadm <--- this line blinkig white text on red background
drwxr-xr-x 2 root root 4096 Jun 5 2023 user |
When I try to follow the link, I find it takes me to /usr/bin, not to /usr where the udevadm executable is actually located
| Code: | # cd /usr/lib/systemd/
# cd ../../bin/
# pwd
/usr/bin
# which udevadm
/bin/udevadm |
When I examine another non-merged-usr system, that was stable (keyword amd64) from the start, I don't have that problem --
| Code: | # for f in ${bins}; do echo "----[ $f ]----"; ls -al $(dirname $f) | grep $(basename $f); if [ -x "$f" ]; then echo true; else echo false; fi; done
----[ /sbin/udevd ]----
false
----[ /lib/systemd/systemd-udevd ]----
lrwxrwxrwx 1 root root 17 Dec 2 17:33 systemd-udevd -> ../../bin/udevadm
true
----[ /usr/lib/systemd/systemd-udevd ]----
false
# cd /lib/systemd
# cd ../../bin
# pwd
/bin |
I notice these packages support a split-usr USE flag
| Code: | # equery h split-usr
* Searching for USE flag split-usr ...
[IP-] [ ] app-alternatives/awk-4:0
[IP-] [ ] app-alternatives/bzip2-1:0
[IP-] [ ] app-alternatives/cpio-0:0
[IP-] [ ] app-alternatives/gzip-1:0
[IP-] [ ] app-alternatives/tar-0:0
[IP-] [ ] dev-libs/lzo-2.10:2
[IP-] [ ] sys-apps/baselayout-2.17:0
[IP-] [ ] sys-apps/coreutils-9.5:0
[IP-] [ ] sys-apps/shadow-4.14.8:0/4
[IP-] [ ] sys-apps/systemd-utils-254.17:0
[IP-] [ ] sys-libs/ncurses-6.4_p20240414:0/6 |
I'm wondering if my "reversion" went astray because I don't have that USE flag set, anywhere. Sould I set it for one or more of these packages, or maybe globally? is it safe to do that? |
|
| Back to top |
|
|
bstaletic
Guru
Joined: 05 Apr 2014
Posts: 453
|
| Posted: Wed Jan 08, 2025 7:25 pm Post subject: |
|
|
| Quote: | | I'm wondering if my "reversion" went astray because I don't have that USE flag set, anywhere. Sould I set it for one or more of these packages, or maybe globally? is it safe to do that? |
Definitely do not touch the split-usr flag. It is set and enforced by your profile.
| Quote: | | I note that name -l thinks the target ( ../../bin/udevadm ) does not exist, but it does (as the output of name -l demonstrates for the /sbin/udevd link that I created) |
../../bin/udevadm is relative to /usr/lib/systemd/systemd-udevd, so the target is at /usr/bin/udevadm. |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 22976
|
| Posted: Wed Jan 08, 2025 7:37 pm Post subject: |
|
|
| What is the output of emerge --info; emerge --pretend --verbose /bin/udevadm? I suspect you somehow installed a merged-usr package onto a split-usr system. This shouldn't be possible by default, and can cause problems if you manage to do it. |
|
| Back to top |
|
|
grknight
Retired Dev
Joined: 20 Feb 2015
Posts: 1978
|
| Posted: Wed Jan 08, 2025 7:38 pm Post subject: |
|
|
| I would be curious to see the output of ls -ld /bin /usr/bin |
|
| Back to top |
|
|
brendlefly62
Apprentice
Joined: 19 Dec 2009
Posts: 154
|
| Posted: Wed Jan 08, 2025 11:36 pm Post subject: |
|
|
Thanks, Hu, grknight. Here is the info you requested (some via wgetpaste) --
Hu:
emerge --info --> https://bpa.st/6RYA
emerge --pretend --verbose /bin/udevadm --> https://bpa.st/O23Q
| Code: | $ emerge --pretend --verbose /bin/udevadm
Local copy of remote index is up-to-date and will be used.
Local copy of remote index is up-to-date and will be used.
Local copy of remote index is up-to-date and will be used.
These are the packages that would be merged, in order:
Calculating dependencies ... done!
Dependency resolution took 4.18 s (backtrack: 0/20).
[binary R ] sys-apps/systemd-utils-254.17::gentoo USE="kmod (split-usr) tmpfiles udev -acl -boot -kernel-install -secureboot (-selinux) -sysusers -test -ukify" ABI_X86="(64) -32 (-x32)" PYTHON_SINGLE_TARGET="python3_12 -python3_10 -python3_11" 0 KiB
Total: 1 package (1 reinstall, 1 binary), Size of downloads: 0 KiB |
(note: I have several -march=ivybridge systems which serve as binhost for each other)
to get a non-binary-package ...
FEATURES=" -getbinpkg -distcc" emerge --pretend --verbose /bin/udevadm --> https://bpa.st/PKPQ
| Code: | $ FEATURES=" -getbinpkg -distcc" emerge --pretend --verbose /bin/udevadm
These are the packages that would be merged, in order:
Calculating dependencies ... done!
Dependency resolution took 4.65 s (backtrack: 0/20).
[ebuild R ] sys-apps/systemd-utils-254.17::gentoo USE="kmod (split-usr) tmpfiles udev -acl -boot -kernel-install -secureboot (-selinux) -sysusers -test -ukify" ABI_X86="(64) -32 (-x32)" PYTHON_SINGLE_TARGET="python3_12 -python3_10 -python3_11" 14,226 KiB
Total: 1 package (1 reinstall), Size of downloads: 14,226 KiB
|
in either case, it wants to emerge systemd-utils (and I note that it picks the split-usr USE flag with parentheses, which I think means it is forced ty my profile, which by the way is
| Code: | $ eselect profile list | grep '\*$'
[58] default/linux/amd64/23.0/split-usr/hardened (stable) * |
I didn't change that in my "reversion" - I just changed ACCEPT_KEYWORDS in my make.conf
grknight: Here is the ls -ld --
| Code: | # ls -ld /bin /usr/bin
drwxr-xr-x 2 root root 4096 Jan 7 21:07 /bin
drwxr-xr-x 2 root root 106496 Jan 8 17:50 /usr/bin |
|
|
| Back to top |
|
|
brendlefly62
Apprentice
Joined: 19 Dec 2009
Posts: 154
|
| Posted: Sat Jan 11, 2025 6:14 pm Post subject: |
|
|
Solved, but I still have some questions near the bottom -- it looks like sys-apps/systemd-utils-255.7-r4 initially did this back in June. My system changed from "replaced dir /lib/systemd: to "replaced sym /lib/systemd" with systemd-utils-255.7-r4 on 20240614 and then it changed to "cfgpro sym /lib/systemd" with systemd-utils-256.9 on20241204 -- then when I did the "reversion" to stable packges across the system, it switched to "cfgpro dir /lib/systemd" with systemd-utils-254.17 on 20250108. After digging through logs and discovering this (documented below), I applied the fix at the bottom, though I still have questions...
This part is OK: /lib/systemd/systemd-udevd is a symlink that points to ../../bin/udevadm.
The problem is that, on this system, /lib/systemd is not a directory; it is a symlink to /usr/lib/systemd (which should not be the case on a split-usr system).
| Code: | # file /lib/systemd
/lib/systemd: symbolic link to ../usr/lib/systemd |
So, when systemd-utils installs the link to systemd-udevd (../../bin/udevadm), it points to a non-existent target in /usr/bin/ | Code: | # ls -al /lib/systemd
lrwxrwxrwx 1 root root 18 Dec 14 13:12 /lib/systemd -> ../usr/lib/systemd |
Lots of packages claim to own /lib/systemd -
| Code: | # equery b /lib/systemd
* Searching for /lib/systemd ...
app-admin/logrotate-3.22.0 (/lib/systemd)
app-crypt/gnupg-2.4.6-r1 (/usr/lib/systemd)
...
sys-apps/systemd-utils-254.17 (/lib/systemd)
...
sys-process/cronie-1.7.2 (/lib/systemd)
www-servers/apache-2.4.62 (/lib/systemd) |
I looked at the logs for all versions installed since the beginning of 2024, since it may not have been the most-recent install that created /lib/systemd as a symlink...
I searched all logfiles for text indicating the merge had touched either "dir /lib/systemd" or "sym /lib/systemd"
| Code: | for owner in $(equery b /lib/systemd);
do logname=$(qatom -F %{CATEGORY}:%{PN} $owner);
echo "----[ $logname ]----------";
for candidate in $(find /var/log/portage/ -name "$logname"'*' | grep '202[45]');
do [[ $(grep -i 'dir /lib/systemd\|sym /lib/systemd' $candidate) ]] && echo -n $candidate && grep -i 'dir
/lib/systemd\|sym /lib/systemd' $candidate;
done;
done > /home/<user>/lib_systemd_makers
#cat /home<user>/lib_systemd_makers
...
/var/log/portage/sys-apps:systemd-utils-254.13:20240603-163236.log--- replaced sym /lib/systemd/systemd-udevd
--- replaced dir /lib/systemd/network
--- replaced dir /lib/systemd
/var/log/portage/sys-apps:systemd-utils-254.12:20240603-163656.log--- replaced sym /lib/systemd/systemd-udevd
--- replaced dir /lib/systemd/network
--- replaced dir /lib/systemd
/var/log/portage/sys-apps:systemd-utils-255.7-r1:20240609-020811.log--- replaced sym /lib/systemd/systemd-udevd
--- replaced dir /lib/systemd/network
--- replaced dir /lib/systemd
/var/log/portage/sys-apps:systemd-utils-254.13:20240609-021223.log--- replaced sym /lib/systemd/systemd-udevd
--- replaced dir /lib/systemd/network
--- replaced dir /lib/systemd
/var/log/portage/sys-apps:systemd-utils-255.7-r4:20240614-203127.log--- replaced sym /lib/systemd
/var/log/portage/sys-apps:systemd-utils-255.7-r1:20240614-203628.log--- replaced sym /lib/systemd
/var/log/portage/sys-apps:systemd-utils-255.8:20240709-142720.log--- replaced sym /lib/systemd
/var/log/portage/sys-apps:systemd-utils-255.7-r4:20240709-143324.log--- replaced sym /lib/systemd
/var/log/portage/sys-apps:systemd-utils-255.9:20240720-212632.log--- replaced sym /lib/systemd
/var/log/portage/sys-apps:systemd-utils-255.8:20240720-213316.log--- replaced sym /lib/systemd
...
/var/log/portage/sys-apps:systemd-utils-254.17:20241226-225720.log--- cfgpro sym /lib/systemd
/var/log/portage/sys-apps:systemd-utils-256.9-r1:20241226-225743.log--- cfgpro sym /lib/systemd
/var/log/portage/sys-apps:systemd-utils-254.17:20250108-021109.log--- replaced sym /lib/systemd/systemd-udevd
--- replaced dir /lib/systemd/network
--- cfgpro dir /lib/systemd
/var/log/portage/sys-apps:systemd-utils-254.17:20250108-021132.log--- replaced sym /lib/systemd/systemd-udevd
--- replaced dir /lib/systemd/network
--- cfgpro dir /lib/systemd
/var/log/portage/sys-apps:systemd-utils-254.17:20250108-133037.log--- replaced sym /lib/systemd/systemd-udevd
--- replaced dir /lib/systemd/network
--- cfgpro dir /lib/systemd
/var/log/portage/sys-apps:systemd-utils-254.17:20250108-133058.log--- replaced sym /lib/systemd/systemd-udevd
--- replaced dir /lib/systemd/network
--- cfgpro dir /lib/systemd |
I pasted the full output here: https://paste.gentoo.zip/OgPhmtY4
I notice that systemd-utils is the only package that built /lib/systemd as a symlink. It evidently started doing so, on this system, on June 14 2024, when I merged systemd-utils-255.7-r4 and it replaced sym /lib/systemd. Previously, on June 9 2024, systemd-utils-254.13 had replaced dir /lib/systemd, as it had done in every update prior to that.--
sys-apps:systemd-utils-255.7-r1:20240609-020811.log-... --- replaced dir /lib/systemd
and
systemd-utils-254.13:20240609-021223.log... --- replaced dir /lib/systemd
Starting June 14 however --
sys-apps:systemd-utils-255.7-r4:20240614-203127.log--- replaced sym /lib/systemd
On Dec 26, it changed from "replaced" sym to "cfgpro sym"
sys-apps:systemd-utils-254.17:20241226-225720.log--- cfgpro sym /lib/systemd
Then, with the "verversion" to all-stable packages on Jan 8, it went to "cfgpro dir"
sys-apps:systemd-utils-254.17:20250108-021109.log... --- cfgpro dir /lib/systemd
I had not noticed that, nor that the logs since December, when I merged systemd-utils-256.9 have been saying "cfgpro sym /lib/systemd" instead of "replaced..."
I don't understand why this did not become a problem untill I rolled back to stable versions of everything on January 8 2025, and I don't really know what the "configpro" indicates, but there does not seem to be any CONFIG_PROTECT assignment which would affect /lib/systemd, and my INSTALL_MASK covers only a few directories in /usr/share"
Solution: I carefully removed systemd-utils and deleted the symlink /lib/systemd and then reinstalled systemd-utils and verified it is now a directory. I also verified that it contains the symlink to udevadm, which now works | Code: | # emerge -avC systemd-utils
# rm /lib/systemd
# FEATURES="${FEATURES} -getbinpkg" emerge -av1 systemd-utils
## Now verify the link
# file /lib/systemd
/lib/systemd: directory
# Now to reverify the test in the /init.d/udev script --
# bins="/sbin/udevd /lib/systemd/systemd-udevd /usr/lib/systemd/systemd-udevd"
# for f in ${bins}; do echo "----[ $f ]----"; if [ -x "$f" ]; then echo true; else echo false; fi; ls -al $(dirname $f)/ | grep $(basename $f); done
----[ /sbin/udevd ]----
true
lrwxrwxrwx 1 root root 12 Jan 8 08:50 udevd -> /bin/udevadm
----[ /lib/systemd/systemd-udevd ]----
true
lrwxrwxrwx 1 root root 17 Jan 11 12:24 systemd-udevd -> ../../bin/udevadm
----[ /usr/lib/systemd/systemd-udevd ]----
false
|
Now the link I installed in /sbin as a workaround is redundant, but I will leave it there |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Installing Gentoo
