glusterd drives 14 of 16 cpus to 100% active wait ...

linuxwoody · n00b Joined: 01 Aug 2023 Posts: 4

Hello,

please can anybody help me? I have a gluster filesystem running fine on two gentoo nodes for more than 1,5 years now using gluster v8.6. Last Weekend I made an complete update of one node mainly to get the latest samba version. The other node is still left untouched. Everything worked fine, and gluster was updated to v10.2-r2.

The problem is that now the updated node runs with 100% active cpu waiting as soon as the gluster daemon starts. When killing the gluster processes everything is fine again. The overall performance is more or less ok, but i can't figure out why the gluster daemon is behaving in such a way.

To test it I setup another node and completely resetted all gluster configurations to a clean start. The same here: Even when there is no configuration done in any way - as soon as glusterd starts 14 of 16 cpus are at 100% io-load.

Now I'm searching for a solution for more than 3 days and it's driving me nuts. :evil:

Has anybody experienced this strange behaviour or has an explanation for it?

Any help would be welcome.

Best regards,
Woody
_________________
Linux - life is too short for a reboot

alamahant · Advocate Joined: 23 Mar 2019 Posts: 3950

Welcome to Gentoo!
Maybe update the other Gentoo node to same version of glusterfs.
_________________

Goverp · Advocate Joined: 07 Mar 2007 Posts: 2204

Woody, what kernel are you running? There's a minor bug in 6.4.3 or later (or maybe 6.4.4 or later) where io_uring causes a CPU to be marked 100% busy despite being in IOWait. IIUC it's minor because the CPU is not doing anything, it's just being incorrectly marked busy. There should be a fix in 6.4.8 coming.

Of course, if your system is showing symptoms of being overloaded then it's not this bug, which is only "cosmetic". AFAIK this bug should only mark one cpu busy, not 14 out of 16, so it might be something about gluster, or it might be a different problem.
_________________
Greybeard

linuxwoody · n00b Joined: 01 Aug 2023 Posts: 4

Hi alamahant

linuxwoody · n00b Joined: 01 Aug 2023 Posts: 4

Hi Goverp

Hu · Administrator Joined: 06 Mar 2007 Posts: 23103

If you are satisfied that your problem is solved, please edit your opening post to include the text [SOLVED] in its subject line, so that the index will show this thread is resolved. If you want to be especially helpful to future readers (although this problem seems unlikely to show up again in the future with the same solution), also edit the body of your first post to briefly summarize the solution, or point readers to the post where you described the solution. This way, anyone who finds this thread can quickly apply your solution.