network is slooooooooooooow

[slaterson]

i discovered some slowness in some part of my network today. it's a bit baffling.

I have a red network - internet connection - and a green network - everything in the house on my lan.

my setup is internet (red) -> firewall (red network and green network) -> 2.5gb switch -> server and my client machines. server and clients are all on the same network (10.0.0.x). i've had this network in place for many years, long enough that if make any adjustments i need to refer to notes from years past. it's the house network and it "just works". i decided to make some upgrades, specifically to a 2.5gb switch and 2.5gb nic in the server + clients. i decided to test the network speed between the server and a client, which i had never done before.

i did an iperf3. first, i did this between my server and client. i got an appalling 90mbit/s rate. so i ran it again, same rate. and again, same rate. i then decided to run the same test between my firewall and client, and it was nearly the full 2.5gbit/s speed of the nics.

i'm not sure where to begin to figure this one out. i checked the negotiated connection and it's reported at 2500 mbit/s (2.5gb/s).

the server has 5 nics installed and is running libvirt, with 4 of the nics setup with a bridge for native use with qemu.

edit: tested the speed from the server to the firewall, as well as firewall to server, and both are very slow as well (90mbit/s). (not sure if iperf3 results matter based on which host is the server vs client).

[slaterson]

little bit more info. the "5 nics" are actually 2 cards in total. one is a single 2.5gb interface, and the other is a quad 1gb intefrace card. the quad card is setup for use with libvirt and qemu, and it's interfaces enp2s0f0 through enp2s0f3. the 2.5gb interface is enp4s0.

i checked the speed on all cards, and enp2s0f0 is connected at 100 mbit/s (this shouldn't be connecting at such a slow speed, separate issue). this interface is physically plugged into a 1gb switch port, however it's not actually being used a vm. the other enp2s0 interfaces are connected at 1gb, and enp4s0 is connected at 2.5gb.

my guess is enp2s0f0 is somehow included in the traffic flow to/from server. is there a way to verify this other than shutting that interface down?

when i start the server, it brings up all of enp2s0 first, which has always been odd to me. enp4s0 is the primary interface for the OS, with enp2 there for the vms.

[slaterson]

got this fixed, at least temporarily. not sure if it's a proper, permanent solution.

i add the following to /etc/conf.d/net: