Massive slowdowns when handling many small files

[Jimini]

Dear all,

when handling many small files with gentoo-sources-6.1.*, everything works fine.
With gentoo-sources-6.6.*, the transfer gets slower after a while. A reboot helps for a few moments, until the performance problem occurs again. I have to add, that a normal reboot does not work then, since one of the processes cannot be stopped or killed. Thus, I have to hard reset the system, e.g. with a magic SysRQ.

It does not make any difference how the files are transferred - for testing purposes, I transfer the extracted kernel sources (/usr/src/linux/) from A to B:
- local rsync from SSD to HDD RAID6
- rsync from a NFS share to a local folder
- cp within the same RAID6
- rsync within the same RAID6
- extracting a tarball on the RAID6
For now, a file transfer within the same local SSD seems to work fine.

I have tested a few kernel versions, with the following results:
gentoo-sources-6.1.19 -> OK
gentoo-sources-6.1.28 -> OK
gentoo-sources-6.1.67 -> OK
gentoo-sources-6.6.13 -> problems
gentoo-sources-6.6.62 -> problems
(I did not configure and compile every new kernel from scratch. For new kernel versions, I reuse the old config and, after a "make syncconfig", I compile the new kernel.)

The CPU load looks fine, so does the memory consumption.I assume, that some I/O stuff may be the root cause here, but I honestly have no idea how to verify this.

Since I have a workaround by simply using an older kernel, it does not look like a hardware (memory, CPU, RAID6) problem to me.

I would be thankful for any input for further testing.

Kind regards
Jimini
_________________
"The most merciful thing in the world, I think, is the inability of the human mind to correlate all its contents." (H.P. Lovecraft: The Call of Cthulhu)

[pietinger]

[pingtoo]

So is only destination RAID6 show this symptom? does reverse destination to SSD show same symptom?

It is worthy to know it is write bound or read bound.

[wjb]