systemd assertion => machine won't boot after update

-lk- · n00b Joined: 26 May 2015 Posts: 10

Hi,
yesterday I did not obey the basic rule "if it works, don't touch it" and I ran
emerge -uDva world

on my machine to have the system up to date .When all 288 packages were merged, I rebooted the machine and it was the last time I saw it fully working. After reboot, just when systemd was starting, systemd started to complaint about
assertion "u->instance" failed src/core/load-fragment.c and system freezes.

I am able to to edit grub menu to start not /usr/lib/systemd/systemd but /bin/bash which means I am at least able to get in. I was trying to rebuild systemd once again but every reboot ends with frozen system.

I am desperate as I do not know how to recover from this. Google returns nothing. Did anyone else hit this issue? Is there anything I can do to fix this? Thanks

eccerr0r · Posted: Fri Jan 17, 2020 12:38 am Post subject:

My guess would be something to do with a kernel feature that doesn't exist in your current kernel? Try building a new recent gentoo-sources kernel if you didn't do so?

But this is only a guess.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

Hu · Administrator Joined: 06 Mar 2007 Posts: 21724

Please post the output of emerge --info for that system. If this is too difficult to collect under the circumstances, tell us your kernel version, gcc version, and systemd version. If possible, also pastebin your kernel configuration.

-lk- · n00b Joined: 26 May 2015 Posts: 10

It's rather easy to get this information because I am able to use /bin/bash instead of systemd so system is a bit crippled and I have to do many things manually but at least machine is accessible somehow.

Kernel is the latest one and I went through all settings marked as mandatory in Gentoo install guide.
/proc/config.cz - https://pastebin.com/8YMfjfPH

elsandosgrande · Linux-4.19.86-gentoo-x86

-lk- · n00b Joined: 26 May 2015 Posts: 10

I am sorry I did not emphasise I am running mostly stable packages.

However, I installed kernel 5.4.12 and systemd-244. I ended up with the same failure:

https://imgur.com/a/HTnVARA

eccerr0r · Posted: Fri Jan 17, 2020 8:48 pm Post subject:

Okay, I don't think it's kernel anymore. Seems something wrong with the startup units database... unfortunately I'm ready to learn more about how to fix this other than perhaps making sure your etc-update/dispatch-conf on systemd targets, etc. files are updated.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

-lk- · n00b Joined: 26 May 2015 Posts: 10

Thanks, I do not think this will be an issue:

jburns · Posted: Fri Jan 17, 2020 10:48 pm Post subject:

Have you tried using journalctl to look at the systemd log?

-lk- · n00b Joined: 26 May 2015 Posts: 10

Because systemd stops before mounting disks rw, there is no journal.

I managed to return systemd-239-r2. It was painful as gcc 9.2 refused to compile original sources without modifications but finally I got it there and after reboot, system was ready! Half of the services is not functioning (like network etc.) but at least I can got somewhere...

elsandosgrande · Posted: Sat Jan 18, 2020 12:09 am Post subject:

1. It seems that that file has had assertions fail at least once before: https://github.com/systemd/systemd/issues/6596
2. Is there anything more to the error message?
3. What does `journalctl --catalog --boot` say about the failing services now that you are partially fine?

Why does it feel like everybody but me is having some sort of systemd issue, whereas it is just peachy on my laptop?

Edit

Also, you could have tried compiling systemd with GCC 8.

eccerr0r · Posted: Sat Jan 18, 2020 12:10 am Post subject:

elsandosgrande · Posted: Sat Jan 18, 2020 1:03 am Post subject:

GDH-gentoo · Posted: Sat Jan 18, 2020 1:22 am Post subject:

Huh. That's a test in systemd's code for a situation that should not happen in the program's current state. A sanity check. So assertion failure pretty much equals bug. And by definition, a situation systemd can't possibly know how to get out of, so it freezes execution.

As far as I can tell, yes, that's part of the code that loads units. systemd seemingly finds itself with a unit that should be generated from a template (name@.type), the data structure that contains information about the unit should specify the instance name, but somehow doesn't. Unless one is an expert in systemd's internals, it's not easy to say why that happened, and since triggering the bug means ending up with a dead machine, it's probably not easy to collect additional diagnostic information either.

elsandosgrande · Posted: Sat Jan 18, 2020 3:46 am Post subject:

So... let's call Poettering and see if this is worthy of a ticket? By the sound of it, it should at least be worth a Gentoo Bugzilla ticket.

-lk- · n00b Joined: 26 May 2015 Posts: 10

I am not an systemd expert so I may miss something.

To put systemd-239-r2 back to the Portage, I added 239-r2 manifest into Portage, made patches for it (because "treat warnings as errors" made this unbuildable) and rebuilt portage manifest. Then I was able to emerge 239-r2 and boot to the system. I found there are a lot of missing services.

The first issue with missing network interfaces was probably caused by outdated configuration. I have no idea how this could had worked but had to enable systemd-networkd.service and remove two exclamation marks in this line:
ExecStart=!!/lib/systemd/systemd-networkd

The other services did not start because they disliked PrivateTmp=true in their files. Please do not ask me how it is possible it worked before with the same files but it did. Please do not ask me why systemd is so mysterious that instead of reporting "Unrecognized option PrivateTmp in /lib64/systemd/system/mysqld.service" it just reports "Failed to enable unit: Invalid Argument" and forces me to play very boring trial and error game. After commenting out PrivateTmp in all broken services everything works OK.

Thanks to all who tried to help me, I appreciate your support.

GDH-gentoo · Posted: Sun Jan 19, 2020 3:22 pm Post subject:

Out of curiosity, did the assertion failure happen with current stable systemd (243-r2), or just 244? It is not clear from this thread if you upgraded to 243, had this problem, and then upgraded to 244 to test again (and fail), or if you went straight to 244.

-lk- · n00b Joined: 26 May 2015 Posts: 10

Originally I was running 239-r2 and had no problems. Then I ran emerge -uDva world and systemd was updated to the latest stable version 243-r2. I was not able to boot so according to the advice from elsandosgrande here in the thread I tried to use the most recent version which is 244. If failed the same way so I returned back to 239-r2.

-lk- · n00b Joined: 26 May 2015 Posts: 10

Just an update - about half year later, after update to 245.5, the nasty bug is still there so my machine freezes during the boot due to systemd assertion.

-lk- · n00b Joined: 26 May 2015 Posts: 10

Another update - 253.3-r1 finally works!!!

stefan11111 · l33t Joined: 29 Jan 2023 Posts: 922 Location: Romania

Such a quick fix from the systemd team.
_________________
My overlay: https://github.com/stefan11111/stefan_overlay
INSTALL_MASK="/etc/systemd /lib/systemd /usr/lib/systemd /usr/lib/modules-load.d *udev* /usr/lib/tmpfiles.d *tmpfiles* /var/lib/dbus /usr/bin/gdbus /lib/udev"

Hu · Administrator Joined: 06 Mar 2007 Posts: 21724

In what systemd issue was this failure reported?