Massive slowdowns when handling many small files

[Jimini]

Dear all,

when handling many small files with gentoo-sources-6.1.*, everything works fine.
With gentoo-sources-6.6.*, the transfer gets slower after a while. A reboot helps for a few moments, until the performance problem occurs again. I have to add, that a normal reboot does not work then, since one of the processes cannot be stopped or killed. Thus, I have to hard reset the system, e.g. with a magic SysRQ.

It does not make any difference how the files are transferred - for testing purposes, I transfer the extracted kernel sources (/usr/src/linux/) from A to B:
- local rsync from SSD to HDD RAID6
- rsync from a NFS share to a local folder
- cp within the same RAID6
- rsync within the same RAID6
- extracting a tarball on the RAID6
For now, a file transfer within the same local SSD seems to work fine.

I have tested a few kernel versions, with the following results:
gentoo-sources-6.1.19 -> OK
gentoo-sources-6.1.28 -> OK
gentoo-sources-6.1.67 -> OK
gentoo-sources-6.6.13 -> problems
gentoo-sources-6.6.62 -> problems
(I did not configure and compile every new kernel from scratch. For new kernel versions, I reuse the old config and, after a "make syncconfig", I compile the new kernel.)

The CPU load looks fine, so does the memory consumption.I assume, that some I/O stuff may be the root cause here, but I honestly have no idea how to verify this.

Since I have a workaround by simply using an older kernel, it does not look like a hardware (memory, CPU, RAID6) problem to me.

I would be thankful for any input for further testing.

Kind regards
Jimini
_________________
"The most merciful thing in the world, I think, is the inability of the human mind to correlate all its contents." (H.P. Lovecraft: The Call of Cthulhu)

[pietinger]

[pingtoo]

So is only destination RAID6 show this symptom? does reverse destination to SSD show same symptom?

It is worthy to know it is write bound or read bound.

[wjb]

[Jimini]

pietinger, pingtoo & wjb, thank you for your replies.

Regarding syncconfig, it seems like I did not get the memo ;)
I have adapted my scripts and rebuilt the kernel on this particular machine, thank you.

I ran a few tests:
- unpacking a tarball on the SSD took 1m43s
- unpacking the same tarball on the RAID6 was canceled by me after there was no progress for 75min
- unpacking the same tarball from the RAID6 to the SSD took 1min10s
- unpacking the same tarball from the SSD to the RAID6 was canceled by me after 38mins

Writing to the RAID6 seems to lead to problems.
This is not the case in general - I can copy a 25G file to the RAID6, which takes about 7min (about 60MB/s).

In addition, I had a look at the processes on the system, while I was trying to reboot. "kworker/u8:0+flush-253:0" was at about 100% CPU load.

Maybe I should add share some information on my setup:
- Intel Core i3-9100 CPU @ 3.60GHz, 4 GB memory
- software RAID6, consisting of 7 HDDs (Seagate Exos X16, Toshiba MG08ACA16TE)
- on it is a LUKS2 container with aes-xts-plain64 as cipher
- ext4 is the filesystem:
dumpe2fs 1.47.1 (20-May-2024)
Filesystem volume name: <none>
Last mounted on: /home/share
Filesystem UUID: 94c5bf33-25cb-4eb0-9081-6e638dfabe67
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean
[...]

Please bear in mind that with gentoo-sources-6.1.*, it works without problems :)
Any idea what I could test yet?

Kind regards
Jimini
_________________
"The most merciful thing in the world, I think, is the inability of the human mind to correlate all its contents." (H.P. Lovecraft: The Call of Cthulhu)

[pietinger]

I could take a look at your kernel configuration ... preferably together with all three data mentioned here:
https://wiki.gentoo.org/wiki/User:Pietinger/Overview_of_System_Information
_________________
https://wiki.gentoo.org/wiki/User:Pietinger

[Jimini]

pietinger, thanks a lot for your offer.

Here you are:
emerge --info: https://paste.gentoo.zip/BhPUxQ13
dmesg: https://paste.gentoo.zip/0e1IKBml
lspci -nnk: https://paste.gentoo.zip/fZ9hw8oR (the command throws "lspci: Unable to load libkmod resources: error -2", but I guess this should not be the root cause for my problem)
kernel config: https://paste.gentoo.zip/q2mJlIVk

Again, I just tried to unpack the kernel files right on the RAID6, just for having a look at the dmesg output. But the only new line is the following:
[ 525.705770] EXT4-fs (dm-0): mounted filesystem 94c5bf33-25cb-4eb0-9081-6e638dfabe67 r/w with ordered data mode. Quota mode: none.


From today on, I will be on a trip until next sunday, so I will not be able to respond until then. So please take your time for having a look into the shared information.

Best regards and thanks in advance
Jimini
_________________
"The most merciful thing in the world, I think, is the inability of the human mind to correlate all its contents." (H.P. Lovecraft: The Call of Cthulhu)

[pietinger]

Jimini,

you have as profile selected: default/linux/amd64/23.0/split-usr/no-multilib/hardened
... but you dont have a hardened kernel, you know that?

Your dmesg is not complete: I am missing the beginning; it starts at timestamp [ 1.477459]

I see you have not much RAM: 3.685.320 byte

It is an (old) Intel machine: Intel-R-_Core-TM-_i3-9100 and CONFIG_MCORE2=y

You have disabled module support (like me) and get the harmless error from lspci (I also have) because the package sys-apps/pciutils (which contains lspci) was emerged with Use-flag "kmod". If you want to get rid of the error message, you have to switch off this use flag for this package (but as I said, this is absolutely harmless).

I am missing some important options in your kernel, e.g. CONFIG_X86_X2APIC=y because you miss CONFIG_IRQ_REMAP ... because you switched it off completely:

[Jimini]

Good morning altogether,

first of all, thanks a lot @pietinger for your helpful input. Unfortunately, the system had some hardware issues and I had to replace the core components. Since then, I could not reproduce the problem anymore - but I will test it from time to time.

I will definitely have a look and your hints regarding the kernel config! Some of my configs are more than 15 years old, it is definitely possible that there is come old "misconfig heritage" ;)

Best regards
Jimini
_________________
"The most merciful thing in the world, I think, is the inability of the human mind to correlate all its contents." (H.P. Lovecraft: The Call of Cthulhu)

[pietinger]

[Ralphred]

[pietinger]

[Jimini]

Good morning,

as written last week, I have moved the system to newer hardware (AMD Ryzen 3 PRO 4350G, 16G RAM).

pietinger, I have now reconfigured my kernel (6.6.74) according to your hints - except for the Intel CPU related options, of course.
EXTRA_FIRMWARE is set to "amd-ucode/microcode_amd_fam17h.bin" (according to https://wiki.gentoo.org/wiki/Ryzen#Firmware).

I have now run a few tests again. Hereby I unpacked a tar.gz archive of /usr/src/linux/ - once on the SSD and once on the RAID6.
SSD: 0m13.652s
RAID6: 0m15.606s

Looks good so far, so I repeated the test:
SSD: 0m11.276s
RAID6: the process slows down drastically after a few seconds. While unpacking the archive on the RAID6 takes about one second for one single file to unpack, the result on the SSD is unchanged with 11.282s.

Looking into "top", the output is interesting:

[pietinger]

[Jimini]