View previous topic :: View next topic |
Author |
Message |
-lk- n00b
Joined: 26 May 2015 Posts: 10
|
Posted: Thu Jan 16, 2020 11:39 pm Post subject: systemd assertion => machine won't boot after update |
|
|
Hi,
yesterday I did not obey the basic rule "if it works, don't touch it" and I ran
emerge -uDva world
on my machine to have the system up to date .When all 288 packages were merged, I rebooted the machine and it was the last time I saw it fully working. After reboot, just when systemd was starting, systemd started to complaint about
assertion "u->instance" failed src/core/load-fragment.c and system freezes.
I am able to to edit grub menu to start not /usr/lib/systemd/systemd but /bin/bash which means I am at least able to get in. I was trying to rebuild systemd once again but every reboot ends with frozen system.
I am desperate as I do not know how to recover from this. Google returns nothing. Did anyone else hit this issue? Is there anything I can do to fix this? Thanks |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9697 Location: almost Mile High in the USA
|
Posted: Fri Jan 17, 2020 12:38 am Post subject: |
|
|
My guess would be something to do with a kernel feature that doesn't exist in your current kernel? Try building a new recent gentoo-sources kernel if you didn't do so?
But this is only a guess. _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
Hu Administrator
Joined: 06 Mar 2007 Posts: 21793
|
Posted: Fri Jan 17, 2020 2:17 am Post subject: |
|
|
Please post the output of emerge --info for that system. If this is too difficult to collect under the circumstances, tell us your kernel version, gcc version, and systemd version. If possible, also pastebin your kernel configuration. |
|
Back to top |
|
|
-lk- n00b
Joined: 26 May 2015 Posts: 10
|
Posted: Fri Jan 17, 2020 8:09 am Post subject: |
|
|
It's rather easy to get this information because I am able to use /bin/bash instead of systemd so system is a bit crippled and I have to do many things manually but at least machine is accessible somehow.
Kernel is the latest one and I went through all settings marked as mandatory in Gentoo install guide.
/proc/config.cz - https://pastebin.com/8YMfjfPH
Code: | # emerge --info
Portage 2.3.79 (python 3.6.9-final-0, default/linux/amd64/17.0/no-multilib, gcc-9.2.0, glibc-2.29-r7, 4.19.86-gentoo x86_64)
=================================================================
System uname: Linux-4.19.86-gentoo-x86_64-Intel-R-_Xeon-R-_CPU_5110_@_1.60GHz-with-gentoo-2.6
KiB Mem: 16427092 total, 1556560 free
KiB Swap: 0 total, 0 free
Timestamp of repository gentoo: Wed, 15 Jan 2020 13:00:01 +0000
Head commit of repository gentoo: 689d8da1332fb8ba1094228a70be392789025114
sh bash 4.4_p23-r1
ld GNU ld (Gentoo 2.30 p5) 2.30.0
app-shells/bash: 4.4_p23-r1::gentoo
dev-lang/perl: 5.30.1::gentoo
dev-lang/python: 2.7.17::gentoo, 3.4.8::gentoo, 3.5.5::gentoo, 3.6.9::gentoo
dev-util/cmake: 3.14.6::gentoo
dev-util/pkgconfig: 0.29.2::gentoo
sys-apps/baselayout: 2.6-r1::gentoo
sys-apps/sandbox: 2.13::gentoo
sys-devel/autoconf: 2.69-r4::gentoo
sys-devel/automake: 1.15.1-r2::gentoo, 1.16.1-r1::gentoo
sys-devel/binutils: 2.30-r4::gentoo, 2.32-r1::gentoo
sys-devel/gcc: 7.3.0-r3::gentoo, 8.3.0-r1::gentoo, 9.2.0-r2::gentoo
sys-devel/gcc-config: 2.1::gentoo
sys-devel/libtool: 2.4.6-r6::gentoo
sys-devel/make: 4.2.1-r4::gentoo
sys-kernel/linux-headers: 4.19::gentoo (virtual/os-headers)
sys-libs/glibc: 2.29-r7::gentoo
Repositories:
gentoo
location: /usr/portage
sync-type: rsync
sync-uri: rsync://rsync.gentoo.org/gentoo-portage
priority: -1000
sync-rsync-verify-max-age: 24
sync-rsync-verify-jobs: 1
sync-rsync-verify-metamanifest: yes
sync-rsync-extra-opts:
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -march=nocona -fomit-frame-pointer"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /etc/stunnel/stunnel.conf /usr/share/config /usr/share/easy-rsa /usr/share/gnupg/qualified.txt /var/spool/fax/config /var/spool/fax/etc /var/www/localhost/sqmail/data"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.2/ext-active/ /etc/php/apache2-php7.2/ext-active/ /etc/php/apache2-php7.3/ext-active/ /etc/php/cgi-php5.2/ext-active/ /etc/php/cgi-php7.2/ext-active/ /etc/php/cgi-php7.3/ext-active/ /etc/php/cli-php5.2/ext-active/ /etc/php/cli-php7.2/ext-active/ /etc/php/cli-php7.3/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -pipe -march=nocona -fomit-frame-pointer"
DISTDIR="/usr/portage/distfiles"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="cs en"
MAKEOPTS="-j5"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="3dnow acl amd64 apache2 bash-completion berkdb bind-mysql bzip2 c++ cgi cgroup-hybrid clamav clamd cli crypt ctype curl cxx dba dhcp dlz dri encode exif extras fastbuild fastcgi force-cgi-redirect fpm ftp gd geoip gmp gnutls iconv imap ipv6 libtirpc logrotate maildir memlimit mng mysql mysqli ncurses nls noauthunix nptl openmp openssh openssl pam pcre pop3d posix postfix python3 readline sasl seccomp sftp simplexml sockets socks5 softquota spamassassin split-usr sse ssl symlink systemd sysvipc tokenizer unicode utf8 vhosts xattr xinetd xsl zip zlib" ABI_X86="64" ADA_TARGET="gnat_2018" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="dumpio actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_de fault authn_core authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_ filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp pro xy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias cgid authz_core socache_shmcb" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" NETBEANS_MODULES="apisupport cnd groovy gsf harness ide identity j2ee java mobility nb php profiler soa visualweb webcommon websvccommon xml" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-3 php5-2" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" RUBY_TARGETS="ruby24 ruby25" USERLAND="GNU" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset: CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS |
[Moderator edit: added [code] tags to preserve output layout. -Hu] |
|
Back to top |
|
|
elsandosgrande Tux's lil' helper
Joined: 18 May 2019 Posts: 144 Location: Sarajevo 71000, Bosnia and Herzegovina
|
Posted: Fri Jan 17, 2020 5:44 pm Post subject: |
|
|
-lk- wrote: | Linux-4.19.86-gentoo-x86 |
That is not even 5.x, let alone being the latest kernel.
Realizes that the guy is running stable.
Um... try using the "testing" kernel? I mean, I have never had any issues running everything as testing, so... I have actually gotten so bored with the testing packages that I am throwing my laptop into the flames by trying GCC 10 just to spice things up (and to see what GCC 10 might bring for GentooLTO, but never mind that), so you can tell that testing is not that unstable. If anythings in testing causes an issue for you, it should not be impossible to switch back, though you might need to do it step by step (I am talking about globally setting `~amd64`, but you can also set `~amd64` for the kernel individually in package.keywords).
Realizes that he ended up ranting, but has no better way to put it. Is very sorry. |
|
Back to top |
|
|
-lk- n00b
Joined: 26 May 2015 Posts: 10
|
Posted: Fri Jan 17, 2020 8:37 pm Post subject: |
|
|
I am sorry I did not emphasise I am running mostly stable packages.
However, I installed kernel 5.4.12 and systemd-244. I ended up with the same failure:
https://imgur.com/a/HTnVARA |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9697 Location: almost Mile High in the USA
|
Posted: Fri Jan 17, 2020 8:48 pm Post subject: |
|
|
Okay, I don't think it's kernel anymore. Seems something wrong with the startup units database... unfortunately I'm ready to learn more about how to fix this other than perhaps making sure your etc-update/dispatch-conf on systemd targets, etc. files are updated. _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
-lk- n00b
Joined: 26 May 2015 Posts: 10
|
Posted: Fri Jan 17, 2020 9:43 pm Post subject: |
|
|
Thanks, I do not think this will be an issue:
Code: | # etc-update
Scanning Configuration files...
Exiting: Nothing left to do; exiting. :) |
[Moderator edit: added [code] tags to preserve output layout. -Hu] |
|
Back to top |
|
|
jburns Veteran
Joined: 18 Jan 2007 Posts: 1214 Location: Massachusetts USA
|
Posted: Fri Jan 17, 2020 10:48 pm Post subject: |
|
|
Have you tried using journalctl to look at the systemd log? |
|
Back to top |
|
|
-lk- n00b
Joined: 26 May 2015 Posts: 10
|
Posted: Fri Jan 17, 2020 11:55 pm Post subject: |
|
|
Because systemd stops before mounting disks rw, there is no journal.
I managed to return systemd-239-r2. It was painful as gcc 9.2 refused to compile original sources without modifications but finally I got it there and after reboot, system was ready! Half of the services is not functioning (like network etc.) but at least I can got somewhere... |
|
Back to top |
|
|
elsandosgrande Tux's lil' helper
Joined: 18 May 2019 Posts: 144 Location: Sarajevo 71000, Bosnia and Herzegovina
|
Posted: Sat Jan 18, 2020 12:09 am Post subject: |
|
|
1. It seems that that file has had assertions fail at least once before: https://github.com/systemd/systemd/issues/6596
2. Is there anything more to the error message?
3. What does `journalctl --catalog --boot` say about the failing services now that you are partially fine?
Why does it feel like everybody but me is having some sort of systemd issue, whereas it is just peachy on my laptop?
Edit
Also, you could have tried compiling systemd with GCC 8. |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9697 Location: almost Mile High in the USA
|
Posted: Sat Jan 18, 2020 12:10 am Post subject: |
|
|
I was staring at the code, I think the assert is in a different place than the one in this webpage. There are tons of asserts everywhere in systemd but does not tell what caused it.
Quote: | Also, you could have tried compiling systemd with GCC 8. |
Might be worth it. I'm still building most stuff with gcc8.3... now I'm scared to try gcc-configging to gcc9 ... :D
I am using systemd-243-r2, and it works fine for me... _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
elsandosgrande Tux's lil' helper
Joined: 18 May 2019 Posts: 144 Location: Sarajevo 71000, Bosnia and Herzegovina
|
Posted: Sat Jan 18, 2020 1:03 am Post subject: |
|
|
eccerr0r wrote: | There are tons of asserts everywhere in systemd but does not tell what caused it. |
And that is why I asked for further details on the error message.
eccerr0r wrote: | I'm still building most stuff with gcc8.3... now I'm scared to try gcc-configging to gcc9 ... |
And I am currently rebuilding world with GCC 10 pre-release... To each their own.
eccerr0r wrote: | I am using systemd-243-r2, and it works fine for me... |
Same here with 244. |
|
Back to top |
|
|
GDH-gentoo Veteran
Joined: 20 Jul 2019 Posts: 1569 Location: South America
|
Posted: Sat Jan 18, 2020 1:22 am Post subject: |
|
|
Huh. That's a test in systemd's code for a situation that should not happen in the program's current state. A sanity check. So assertion failure pretty much equals bug. And by definition, a situation systemd can't possibly know how to get out of, so it freezes execution.
As far as I can tell, yes, that's part of the code that loads units. systemd seemingly finds itself with a unit that should be generated from a template (name@.type), the data structure that contains information about the unit should specify the instance name, but somehow doesn't. Unless one is an expert in systemd's internals, it's not easy to say why that happened, and since triggering the bug means ending up with a dead machine, it's probably not easy to collect additional diagnostic information either. |
|
Back to top |
|
|
elsandosgrande Tux's lil' helper
Joined: 18 May 2019 Posts: 144 Location: Sarajevo 71000, Bosnia and Herzegovina
|
Posted: Sat Jan 18, 2020 3:46 am Post subject: |
|
|
So... let's call Poettering and see if this is worthy of a ticket? By the sound of it, it should at least be worth a Gentoo Bugzilla ticket. |
|
Back to top |
|
|
-lk- n00b
Joined: 26 May 2015 Posts: 10
|
Posted: Sun Jan 19, 2020 11:35 am Post subject: |
|
|
I am not an systemd expert so I may miss something.
To put systemd-239-r2 back to the Portage, I added 239-r2 manifest into Portage, made patches for it (because "treat warnings as errors" made this unbuildable) and rebuilt portage manifest. Then I was able to emerge 239-r2 and boot to the system. I found there are a lot of missing services.
The first issue with missing network interfaces was probably caused by outdated configuration. I have no idea how this could had worked but had to enable systemd-networkd.service and remove two exclamation marks in this line:
ExecStart=!!/lib/systemd/systemd-networkd
The other services did not start because they disliked PrivateTmp=true in their files. Please do not ask me how it is possible it worked before with the same files but it did. Please do not ask me why systemd is so mysterious that instead of reporting "Unrecognized option PrivateTmp in /lib64/systemd/system/mysqld.service" it just reports "Failed to enable unit: Invalid Argument" and forces me to play very boring trial and error game. After commenting out PrivateTmp in all broken services everything works OK.
Thanks to all who tried to help me, I appreciate your support.
Last edited by -lk- on Mon Jun 15, 2020 11:49 am; edited 1 time in total |
|
Back to top |
|
|
GDH-gentoo Veteran
Joined: 20 Jul 2019 Posts: 1569 Location: South America
|
Posted: Sun Jan 19, 2020 3:22 pm Post subject: |
|
|
Out of curiosity, did the assertion failure happen with current stable systemd (243-r2), or just 244? It is not clear from this thread if you upgraded to 243, had this problem, and then upgraded to 244 to test again (and fail), or if you went straight to 244. |
|
Back to top |
|
|
-lk- n00b
Joined: 26 May 2015 Posts: 10
|
Posted: Sun Jan 19, 2020 7:10 pm Post subject: |
|
|
Originally I was running 239-r2 and had no problems. Then I ran emerge -uDva world and systemd was updated to the latest stable version 243-r2. I was not able to boot so according to the advice from elsandosgrande here in the thread I tried to use the most recent version which is 244. If failed the same way so I returned back to 239-r2.
Last edited by -lk- on Mon Jun 15, 2020 11:49 am; edited 3 times in total |
|
Back to top |
|
|
-lk- n00b
Joined: 26 May 2015 Posts: 10
|
Posted: Mon Jun 15, 2020 11:47 am Post subject: |
|
|
Just an update - about half year later, after update to 245.5, the nasty bug is still there so my machine freezes during the boot due to systemd assertion. |
|
Back to top |
|
|
-lk- n00b
Joined: 26 May 2015 Posts: 10
|
Posted: Wed Jun 21, 2023 9:47 pm Post subject: |
|
|
Another update - 253.3-r1 finally works!!! |
|
Back to top |
|
|
stefan11111 l33t
Joined: 29 Jan 2023 Posts: 922 Location: Romania
|
Posted: Wed Jun 21, 2023 9:58 pm Post subject: |
|
|
Such a quick fix from the systemd team. _________________ My overlay: https://github.com/stefan11111/stefan_overlay
INSTALL_MASK="/etc/systemd /lib/systemd /usr/lib/systemd /usr/lib/modules-load.d *udev* /usr/lib/tmpfiles.d *tmpfiles* /var/lib/dbus /usr/bin/gdbus /lib/udev" |
|
Back to top |
|
|
Hu Administrator
Joined: 06 Mar 2007 Posts: 21793
|
Posted: Wed Jun 21, 2023 10:51 pm Post subject: |
|
|
In what systemd issue was this failure reported? |
|
Back to top |
|
|
|