[Solved?] MuQSS + TLB Shootdowns Issue?

[DigitalCorpus]

Foreword
Alright, so please bear with me for a moment as I give a little background. I've been absent from this community for several years due to lack of time. I was last on a self-rolled kernel using Zen Sources for BFS, BFQ, and Reiser4 for a spell. Last kernel I rolled for this was v4.1.5, but that was a spurt of activity after bowing out around kernel version 3.1-ish. The server/computer was a Q9400 on a P965 Gigabyte motherboard with 8 GB of DDR2, though the Q9400 replaced the Q6700 simply for power and heat savings, which is did deliver.

Fast forward to about 3 months ago and I'm finally putting together my replacement server based around an i5-8500T on ASrock's top end Z390 micro ITX motherboard due to connectivity and being able to drop in a used 9th gen K series (or somesuch) in 5 or so years. I'm running 16 GB of DDR4 @ 2133 MHz atm as I'm more concerned about idle power.

I do thank the Gentoo developers because, after some fiddling to understand how to use Portage again, and with some usage tweaks over the years, I was able to upgrade that Q9400 to all modern software without having to wipe and reinstall. The Q9400 box is called "Atlas" and I'll call it "A" for the conversation. The 8500T is dubbed "Sexy Prime" (go google the term, I'm running a hexa-core CPU) and I'll call it "SP" henceforth. Now, on to the issue I'm experiencing.

The Actual Problem
So this issue crops up randomly. I've had it happen more frequently on SP because I'm getting acquainted with gong from kernel version 4.1.5 all of the way up to 5.2.3/5.4.6. Lots of new stuff, lots of reorganization, new firmware with ACPI fleshed out on SP, not [comparatively] fleshed out on A. Excluding chipset, storage, and peripheral drivers, along with some minior differences due to architectural support, the kernels are the same. between SP & A. I noticed that when compiling the kernels, occasionally I have a "core" drop out. Even with a "-j4" A drops to 3-cores utilized and SP w/ "-j6" will drop to 5-5.5 cores utilized. I noticed this with netdata and was able to track down what was happening during this time.

Also, note that this utilization drop out would last anywhere from 1-2 seconds to the entire duration of the build process. Sometimes it wouldn't happen at all.

Thanks to netdata's history, I was able to regularly and consistently identify sudden sporadic TLB IRQs happening from what I believe to be TLB shootdowns happening. From my primitive understanding of TLBs, this was happening more regularly on SP than it was on A not just because SP had more RAM and cached files, but instead of dealing with the high latency of a standard HDD which is all that A has, SP was compiling off of a high(er) end NVMe SSD.

Overall CPU Usage Screenshot
Simple Interrupts Screenshot
All CPU Cores - Utilization
All CPU Cores - TLB IRQs

What I've Tried & Why
Some of what the kernel config options say they do is ambiguous or cryptic. I mean that in the manner in such that trying to figure if a specific memory management option affects TLBs or not is kind of obscure. Granted, a technical breakdown is beyond my understanding and largely unnecessary, but it makes it difficult to tell if an option will have an affect or not. Since TLB shootdowns are one CPU [core] invalidating its cache and telling the others to ignore this bit of data, only one core gets affected at a time, though the triggering & handling of these interrupts are bounced around between the cores so the effect is a noted drop of effectively 1 core at a time. I used the compilation of a kernel as my regular test to

CONFIG_SLUB - I've always used it because it has provided more consistent responsiveness on A. Turns out that its the default now too.
CONFIG_BOUNCE - I've toggled this with no effect. Left enabled.
CONFIG_KSM - I've toggled this, also with no effect*. Left enabled, controlled through /sys/kernel/mm/
CONFIG_TRANSPARENT_HUGEPAGE - Turning this off created a higher probability of the TLB shootdowns. The again, some online searching notes that Hugepages are supposed to reduce TLB overhead under most circumstances. I'm aware that it is recommended to disable for systems where database access is the primary function. Left enabled, controlled through /sys/kernel/mm/
CONFIG_CLEANCACHE - Toggled with minimal impact when in isolation.
CONFIG_ZPOOL - Turned on as it seems related to or replaces zcache?
CONFIG_ZBUD - Seems necessary for the former to function. Turned on reduced probability of TLB shootdowns occurring.
CONFIG_Z3FOLD - See above.
CONFIG_SLUB_CPU_PARTIAL - Seemed to have an "immediate" effect. Cut the frequency of TLB shootdowns in about half, though their occurrence was still regular enough.
CONFIG_HZ_PERIODIC - Not what MuQSS is intended to be used with. CPU Usage in htop is reported to be ~50% of actual, TLB Shootdowns affected performance twice as bad on SP, but not on A.
CONFIG_NO_HZ_IDLE - Performance impact of the TLB Shootdowns was dramatically less, but I've still seen 2 cores affected, albeit, a rare situation. Used by default.
CONFIG_CPU_ISOLATION - I've toggled this with no effect. Left disabled.
CONFIG_RCU_FANOUT_LEAF - Left at the default of 16, lowered to 12 & 6 with reduced CPU of rcu_sched.
CONFIG_RCU_FANOUT - Left at the default of 64, lowered to 48 & 24 with reduced CPU of rcu_sched.
CONFIG_RCU_NOCB_CPU - I've toggled this with no effect. Left enabled.
CONFIG_RCU_FAST_NO_HZ - I've toggled this with no effect. Left disabled.
CONFIG_MNATIVE/CONFIG_GENERIC_CPU/CONFIG_MSKYLAKE - No effect.
CONFIG_SCHED_MC - I've toggled this with no effect. Left enabled.
CONFIG_SCHED_MC_PRIO - I've toggled this with no effect. Left enabled.
CONFIG_RQ_* - I've toggled this with no effect. Left set to SMP.

I have played around inside /sys/kernel/mm/transparent_hugepage. Sometimes switching from having them disabled to always enabled would trigger the TLB Shootdowns. While watching the stats inside ./khugepaged, I noticed that more pages were merged and shortly after the dips went away (+/- 5 seconds), /sys/kernel/mm/transparent_hugepage/khugepaged/full_scans would increment by 1, indicating at least a correlation.

What Else Can I Do?
I'm not sure if this is a MuQSS issue or it'll appear with CFS too. I do know that on A, on kernel 4.1.5 w/ BFS that I can get the exact same behavior to occur, albeit randomly as well. This, to me, says that there is an inherent issue with BFS that was transferred to MuQSS, or this is a kernel issue in and over itself that can present and has transferred over the years. Is it s bug or normal behavior? Please note that CFS's normal behavior used to require people compiling their kernels with 1.5 * Core Count and that wasn't considered a bug until BFS showed up.

I admit that I don't know enough as to what is happening here so I'm not confident enough to submit a bug report. I can say that 3 CONFIG options combined reduced the frequency, but do not eliminate the deficit incurred: CONFIG_SCHED_MUQSS, CONFIG_SLUB_CPU_PARTIAL, CONFIG_TRANSPARENT_HUGEPAGE

-Thanks for reading this far. I'm going to get some sleep now.

[DigitalCorpus]

Unfortunately I have not been able to find the solution, but while syncing up the two kernels, the problem seems to have gone away on both. The following are enabled in my config:

CONFIG_SLAB_MERGE_DEFAULT
CONFIG_SLUB_CPU_PARTIAL
CONFIG_KSM
CONFIG_TRANSPARENT_HUGEPAGE
CONFIG_TRANSPARENT_HUGEPAGE_MADVISE
CONFIG_COMPACTION

I'll mark this as solved even though I'm not sure what the problem was. Gotta move forward, despite how much I wish I could identify the one feature or combination thereof that cause the increased frequency of TLB shootdowns and/or their interference with CPU performance.