Vieri
l33t
Joined: 18 Dec 2005
Posts: 907
|
 Posted: Wed Oct 09, 2019 7:53 am Post subject: [SOLVED] ksoftirqd high cpu usage and network performance |
 |
|
Hi,
I'm seeing high cpu usage with severe network performance issues:
| Code: | root 20 0 0 0 0 S 41.4 0.0 0:56.50 ksoftirqd/0
root 20 0 0 0 0 S 16.9 0.0 1:42.80 ksoftirqd/7
root 20 0 0 0 0 R 10.9 0.0 0:39.93 ksoftirqd/5
root 20 0 0 0 0 S 18.5 0.0 0:58.69 ksoftirqd/0
root 20 0 0 0 0 S 16.2 0.0 0:31.31 ksoftirqd/1
root 20 0 0 0 0 S 15.9 0.0 0:47.83 ksoftirqd/2
etc.
|
This is probably due to me using the iptables TEE module to send duplicated ethernet traffic to an IDS.
I'd like to know if I can improve the situation somehow, or if port mirroring compared to TEE is less IRQ intensive.
The amount of duplicated traffic is not that much (way below the Gbps limit of my NIC), but there are a lot of packets.
Would changing the NIC to another brand/model improve anything, or is it a motherboard IRQ limitation?
Can I tweak the Linux kernel somehow? What about the BIOS/UEFI?
Thanks
[EDIT]
I see this BEFORE I notice my performance issue (before I increase network traffic with TEE):
| Code: | # cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7
0: 38 0 0 0 0 0 0 0 IO-APIC 2-edge timer
1: 40 47 43 41 36 49 50 80 IO-APIC 1-edge i8042
9: 0 0 0 0 0 0 0 0 IO-APIC 9-fasteoi acpi
16: 111 110 108 109 110 109 111 111 IO-APIC 16-fasteoi snd_hda_intel:card0
17: 0 0 0 0 0 0 0 0 IO-APIC 17-fasteoi ehci_hcd:usb1
18: 0 0 0 0 0 0 0 0 IO-APIC 18-fasteoi ohci_hcd:usb4, ohci_hcd:usb6
19: 2307018 2238456 2264493 2231795 2268760 2263044 2541531 3714083 IO-APIC 19-fasteoi ahci[0000:00:11.0]
20: 8493729 8402322 8000792 7931677 7996297 8181521 12043638 19403788 IO-APIC 20-fasteoi ohci_hcd:usb5, enp8s5
21: 628379599 609967917 621870084 602488342 619379630 605727135 676766982 749352758 IO-APIC 21-fasteoi ehci_hcd:usb2, enp10s0
22: 0 0 0 0 0 0 0 0 IO-APIC 22-fasteoi ohci_hcd:usb7
23: 0 0 0 0 0 0 0 0 IO-APIC 23-fasteoi ehci_hcd:usb3
27: 65207999 64645316 64644203 64729704 64847960 65953503 73183369 125964523 IO-APIC 23-fasteoi enp5s0
28: 216181372 212555649 213274025 213086365 215086398 213615937 234532323 365482617 IO-APIC 8-fasteoi enp6s0
31: 60110398 59858880 59359634 60369961 59158403 61144056 66332706 123869848 IO-APIC 18-fasteoi enp7s0f2
32: 2009259 2002781 1977972 2010848 1978242 2050958 2264027 4195056 IO-APIC 19-fasteoi enp7s0f3
33: 0 0 0 0 0 0 0 0 IO-APIC 20-fasteoi ahci[0000:02:00.0]
34: 0 0 0 0 0 0 0 0 IO-APIC 22-fasteoi ahci[0000:03:00.0]
35: 4 4 4 4 2 2 2 6 IO-APIC 1-fasteoi snd_hda_intel:card1
NMI: 0 0 0 0 0 0 0 0 Non-maskable interrupts
LOC: 53209683 53209657 53209642 53209627 53209612 53209597 53209582 53209567 Local timer interrupts
SPU: 0 0 0 0 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 0 0 0 0 IRQ work interrupts
RTR: 0 0 0 0 0 0 0 0 APIC ICR read retries
RES: 70451768 41585671 68296116 39997277 67553405 40218487 66215272 40613791 Rescheduling interrupts
CAL: 4053482 2106077 3992761 2209668 4100902 2159050 4052725 2139843 Function call interrupts
TLB: 4056560 2107665 3995265 2211311 4103650 2160431 4055474 2140972 TLB shootdowns
TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts
DFR: 0 0 0 0 0 0 0 0 Deferred Error APIC interrupts
MCE: 0 0 0 0 0 0 0 0 Machine check exceptions
MCP: 1762 1762 1762 1762 1762 1762 1762 1762 Machine check polls
ERR: 0
MIS: 0
PIN: 0 0 0 0 0 0 0 0 Posted-interrupt notification event
PIW: 0 0 0 0 0 0 0 0 Posted-interrupt wakeup event
|
I'm guessing enp10s0 is the culprit.
Last edited by Vieri on Tue Oct 22, 2019 7:23 am; edited 1 time in total |
|
Kernel & Hardware
