Massive slowdowns when handling many small files

[Jimini]

Dear all,

when handling many small files with gentoo-sources-6.1.*, everything works fine.
With gentoo-sources-6.6.*, the transfer gets slower after a while. A reboot helps for a few moments, until the performance problem occurs again. I have to add, that a normal reboot does not work then, since one of the processes cannot be stopped or killed. Thus, I have to hard reset the system, e.g. with a magic SysRQ.

It does not make any difference how the files are transferred - for testing purposes, I transfer the extracted kernel sources (/usr/src/linux/) from A to B:
- local rsync from SSD to HDD RAID6
- rsync from a NFS share to a local folder
- cp within the same RAID6
- rsync within the same RAID6
- extracting a tarball on the RAID6
For now, a file transfer within the same local SSD seems to work fine.

I have tested a few kernel versions, with the following results:
gentoo-sources-6.1.19 -> OK
gentoo-sources-6.1.28 -> OK
gentoo-sources-6.1.67 -> OK
gentoo-sources-6.6.13 -> problems
gentoo-sources-6.6.62 -> problems
(I did not configure and compile every new kernel from scratch. For new kernel versions, I reuse the old config and, after a "make syncconfig", I compile the new kernel.)

The CPU load looks fine, so does the memory consumption.I assume, that some I/O stuff may be the root cause here, but I honestly have no idea how to verify this.

Since I have a workaround by simply using an older kernel, it does not look like a hardware (memory, CPU, RAID6) problem to me.

I would be thankful for any input for further testing.

Kind regards
Jimini
_________________
"The most merciful thing in the world, I think, is the inability of the human mind to correlate all its contents." (H.P. Lovecraft: The Call of Cthulhu)

[pietinger]

[pingtoo]

So is only destination RAID6 show this symptom? does reverse destination to SSD show same symptom?

It is worthy to know it is write bound or read bound.

[wjb]

[Jimini]

pietinger, pingtoo & wjb, thank you for your replies.

Regarding syncconfig, it seems like I did not get the memo ;)
I have adapted my scripts and rebuilt the kernel on this particular machine, thank you.

I ran a few tests:
- unpacking a tarball on the SSD took 1m43s
- unpacking the same tarball on the RAID6 was canceled by me after there was no progress for 75min
- unpacking the same tarball from the RAID6 to the SSD took 1min10s
- unpacking the same tarball from the SSD to the RAID6 was canceled by me after 38mins

Writing to the RAID6 seems to lead to problems.
This is not the case in general - I can copy a 25G file to the RAID6, which takes about 7min (about 60MB/s).

In addition, I had a look at the processes on the system, while I was trying to reboot. "kworker/u8:0+flush-253:0" was at about 100% CPU load.

Maybe I should add share some information on my setup:
- Intel Core i3-9100 CPU @ 3.60GHz, 4 GB memory
- software RAID6, consisting of 7 HDDs (Seagate Exos X16, Toshiba MG08ACA16TE)
- on it is a LUKS2 container with aes-xts-plain64 as cipher
- ext4 is the filesystem:
dumpe2fs 1.47.1 (20-May-2024)
Filesystem volume name: <none>
Last mounted on: /home/share
Filesystem UUID: 94c5bf33-25cb-4eb0-9081-6e638dfabe67
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean
[...]

Please bear in mind that with gentoo-sources-6.1.*, it works without problems :)
Any idea what I could test yet?

Kind regards
Jimini
_________________
"The most merciful thing in the world, I think, is the inability of the human mind to correlate all its contents." (H.P. Lovecraft: The Call of Cthulhu)