Since Kernel+GCC update, MD-RAID refuses to write

[Cygon]

EDIT/UPDATE: I initially thought this was a VirtualBox issue and posted it under "Desktop Environments," but the problem is much broader, so I'm reposting this in "Kernel & Hardware"

Summary: yesterday, I did an `emerge --sync` and `emerge --update` that seems to have conjured up major trouble on my system.

State that worked fine for weeks:

• sys-devel/gcc-13.2.1_p20230826
  
• sys-kernel/gentoo-sources 6.1.53-r1

State where nothing gets written to the disks of my MD-RAID partitions.

• sys-devel/gcc-13.2.1_p20231014
  
• sys-kernel/gentoo-sources-6.1.57

https://forums.gentoo.org/posting.php?mode=editpost&p=8805742
I tried to go back to kernel 6.1.53-r1 to narrow the issue down, but that one doesn't even have the 'amd64' or '~amd64' keywords anymore in the current portage tree. Maybe I'll add it back in, manually, but I'm sure the maintainers removed the keywords for a reason...

I have a SuperMicro motherboard with a "fake RAID" chip (Intel VROC). It's essentially a boot-time-configurable MD-RAID: I have a boot menu where I can create and destroy RAID partitions and when I boot to Windows w/installed Intel VROC driver or to Linux w/installed MD-RAID, it automatically assembles the software RAID.

The problem I'm now having asserts itself like this:

Since the `emerge --update`, nothing is ever written to the drives in this RAID anymore. It mounts, I can read from it and I can even write a few Gigabytes to it (that all end up in the disk cache), but not a single byte is ever committed to disk.

If I type `sync` to flush the disk cache, or reach roughly 5 Gigabytes of cached writes, write accesses on all partitions suddenly hang indefinitely. I assume the kernel is waiting for some write queue that has gotten too full, but I don't know the exact inner workings. At that point, the system is quite unusable, I can't even reboot anymore, I can only press the reset button to bring it back into a usable state.

I've now compiled kernel 6.1.57 with the latest stable GCC (sys-devel/gcc-13.2.1_p20230826) and the issue is still there. Weirdly, the current Gentoo Live USB image runs kernel 6.1.57, also compiled with gcc-13.2.1_p20230826, and it cleanly detects, assembles and writes to the MD-RAID partitions without issue.

I'll now try booting an older kernel that still lingers on my boot partition, but if that doesn't work, I'm at a loss at what might cause it. Could other packages that arrived with yesterday's `emerge --update` be to blame?

[Cygon]

I'm back on

[NeddySeagoon]

Cygon,

What does dmesg show after a small test write?
Put the whole thing onto a pastebin please.

What about

[Cygon]

I already checked `/proc/mdstat` and saw nothing different from how it was during all the weeks (years!) it worked:

[Cygon]

I repeated the test and gave it a few minutes to write or fail or something else (but the `sync` is still stuck even as I'm writing this).

Here's what KSysGuard monitoring my disk cache and both drives shows:
https://imgur.com/a/fSGXnRp

As you can see, the file copy even reads from both drives, but doesn't ever write. It all collects in the disk cache and stays there.

Here's the output of `dmesg`, captured after the write was hanging for several minutes already. The last line with the `nfs` mount is shortly after the write started (I mounted a network drive so I could save the screenshot to make sure I would be able to save it).
https://pastebin.com/t4ZePCSG

[NeddySeagoon]

Cygon,

[Cygon]

I'm using the fuse version of the NTFS-3G driver.

Until the `emerge --update` yesterday evening, I have been using those NTFS RAID partitions quite thoroughly for development, video editing and games, 99.9% of the data stored on those drives was written through that NTFS implementation.

Kernel config:

[NeddySeagoon]

Cygon,

[Cygon]

I believe `ntfs` has been an alias for `ntfs-3g` nearly since the ancient ntfs kernel module was given up, but just to be sure, here's the test:

Okay, first, working mounts from the Gentoo Live USB image:

[Cygon]

I searched through my `.bash_history` and found that a few attempts to set up a "write-intent bitmap" for my RAID array intersected with the system update:

[grknight]

[Cygon]

Yes, I checked dmesg, /proc/mdstat and even the related processes each time. The array just remains in the 'normal' and 'clean' state.

I've had a few resyncs in the past, the file system performance is tolerable throughout and I was still able to cleanly shut down the system.

In this case, even the shutdown (both via KDE or via 'reboot' or 'poweroff' in a root shell) hangs forever, so I have no choice but to use the reset button. Thereafter (even if I waited 10-30 minutes before resetting), all changes on the RAID drives are gone and the arrays remain clean. Deleted files come back. File copies I did never happened, and so on. All changes I do seem to wait in the disk cache forever.

When I used the physical reset button in the past, it was a surefire method of having to sit through another RAID resync

[Cygon]

Success! I finally got it to write again:
https://imgur.com/a/kk3VRGF

Unluckily, I did a few changes at once this time, without any testing inbetween:

• Extracted config.gz out of my month-old kernel and used that as a base
  
• Compiled gentoo-source-.6.1.57 from scratch (directory deleted, re-emerged, .config copied in)
  
• Built the NTFS3 code into the kernel image
  
• Explicitly used `ntfs3` in my `/etc/fstab`
  
• Rebuilt all packages that were emerged after the `~amd64` GCC release was emerged

Weirdly, the kernel now has md124+md127 for the SSD-based RAID-1 and md125+md126 for the old degraded HDD-based RAID-1.

Also, if I change back to `ntfs-3g` or even just `ntfs`, it still works.

dmesg when RAID drives were not writing:

[NeddySeagoon]

Cygon,


Compare

[Cygon]

I tried adding a write intent bitmap (see a few post up), it's not supported by mdadm for Intel's weird kind of raid arrays with needless sub-arrays, unfortunately.


The problem just returned. While the system was running and with no configuration changes of any kind.

It's still present after a reboot. So now I'm back at square one (and I'm using `ntfs3` for all NTFS partitions since this morning).

UPDATE: I also had a manually triggered RAID resync running when it happened. The resync went to 100% but `mdadm --examine` still said the array was "dirty.".

After booting into Windows, running the "Check Disk" function on all partitions w/"Repair" where it was offered, then rebooting back into Linux... NTFS partitions write again.

This is either just a coincidence and I've got only sporadic failures now - or - the NTFS3/NTFS-3G code has some kind of problem (version of the NTFS file system?) and is handling it in a very, very confusing way (if completely not processing a write and leaving it hanging is even something a Linux file system driver can do).

[Cygon]

After being able to cleanly shut down my system yesterday, upon booting today, it started a resync (which did write to the underlying SSDs) but all changes on the file system level accumulated in the disk cache until I once more had to use the reset button.

Dual-booting into Windows shows no errors on the NTFS partitions.

It's beyond frustrating at this point.

What might be the best way of getting past this?

• Kill the Linux partition and start over from scratch with a clean install?
  
• Forget about the RAID and set up an rsync job
  
• Use my surplus proprietary hardware RAID controller (I think it's a MegaRAID SAS 9361-4i) - no idea about driver situation, worried that it may AES-encrypt drives, single (expensive) point of failure and so on)
  
• Switch the partitions to ext4 or another well-established file system (assuming the file system driver is even to blame)

If only that machine code bastard that is the cause of this would write the issue it's having into some log... somewhere.