network latency

Vieri · l33t Joined: 18 Dec 2005 Posts: 887

Hi,

I have 3 Gentoo VoIP servers, 1 Gentoo inner firewall and 1 Gentoo ISP gateway.
The switching backbone is a mix of Cisco, D-Link and Ubiquiti Unifi.

Some clients (eg. VoIP phones) have networking issues in that a ping from any of the Gentoo servers shows high latency values (300-800ms). There can also be packet loss (40%).

For some reason, it seems that these issues occur only when the clients are connected to 8-port or 16-port "mini" switches. These unmanaged 1Gbps switches (with very little traffic and hosts connected) are unfortunately necessary until the wired network is extended correctly.

The only way I can get these clients to work well (no latency, no audio drops when on a VoIP call, etc.) is to force their NIC eth negotiation to 100Mbps.

On the other hand, these devices work fine with auto-negotiation or 1Gbps full-duplex when connected directly to ports on managed access switches.

I tried tweaking the clients (firmware config in case of VoIP phones or printers), but I'm still facing the same problem.

This behavior has been noticed on devices of all brands and models, so it seems to be a network problem, not a client firmware or hardware issue.
Also, the "mini" unmanaged switches are also of different brands and models.

What should I be looking for? What can I try?
It's really puzzling because I don't see why I get these results.
I even tried connecting just one single device to an unmanaged switch, but I still saw the same behavior.

The access switch I connected the unmanaged switch to is a D-Link with no special config (no bpdu-guard features or the likes).

Any ideas are greatly appreciated.

Regards,

Vieri

[EDIT]

Actually, it seems that there's something "poisoning" the network (but it only affects a few devices connected as I said) because I just noticed that after 3:30 pm the latency issues suddenly went away. I'm guessing that it could be because a device or group of devices shut down. I'm also guessing that the problem will surface again tomorrow morning as soon as "something" comes back on-line.
If I were to try to capture all the traffic that reaches, say, the Gentoo firewall which is the default gateway for all devices, what kind of traffic should I be searching for to try to find the source that may be causing this trouble?

[EDIT 2]

It seems that I could have found the culprit. There's a Linux Mint desktop in my default vlan that seems to have something to do with it all. IF I halt it or disconnect its NIC there are no more network issues. Rebooting it or reconnecting the NIC doesn't have necessarily an immediate effect on all the "failing" devices, but after a while they eventually start failing. Stopping or disconnecting the NIC of this Linux Mint host immediately solves my network woes.
The OS is updated.
What can I try next?
I could grab a tcpdump trace on the Gentoo default gateway, but even if I did, what would I be looking for?

alamahant · Advocate Joined: 23 Mar 2019 Posts: 3879

It seems a QoS issue.
Plz see
https://wiki.archlinux.org/title/Advanced_traffic_control
and
https://www.howtogeek.com/75660/the-beginners-guide-to-qos-on-your-router/
Maybe enable it in your router?
_________________

pietinger · Posted: Thu May 26, 2022 7:12 pm Post subject: Re: network latency

pingtoo · Posted: Thu May 26, 2022 8:55 pm Post subject:

My wild guess

Your Linux Mint desktop have duplicated IP with something on you network.

Vieri · l33t Joined: 18 Dec 2005 Posts: 887

For now I can say that the Linux Mint system is more and more suspicious because I haven't had one single network issue since it was shut down. It's a standard up-to-date single-NIC DHCP desktop client (standalone, not joined to an AD domain). No special services installed.
When the network woes occur the Linux Mint host is not necessarily doing anything (at least user-triggered). All I need to do is boot the system to get into trouble (after several minutes running).

I don't think QoS is the solution, or at least it wouldn't solve the underlying issue.

I also don't think it's a duplicate IP addr. issue because:

1) it's in DHCP so it is "unlikely" going to be served a duplicate IP addr. There are no other static IP addr. in that DHCP range.

2) As soon as the Linux Mint host went off-line I checked the network for another host with the same IP addr. and found none (nmap). I did it again today -- still nothing.

3) If there were an IP addr. conflict the booting host would not get network access, and it would not poison the rest of the network. The Linux Mint machine connects fine, can browse the network just fine, runs sshd to which I can connect fine and stay connected hours on end.

A broadcast storm is definitely something I'd like to look into.

Next time I boot the system I'd like to:

a) ps aux
to see all the running processes

b) netstat -na
to see all the connections

c) tcpdump -n -i any (actually, there's only one interface)
Now, in this case how do I filter for broadcast storms?
I don't think I should be looking for Layer 3 broadcasts, but then again I can't rule out anything.
Are Layer 2 broadcasts simply ARP requests?

I think I can run something like this directly on the Linux Mint system:

pietinger · Posted: Fri May 27, 2022 8:41 am Post subject:

Vieri · l33t Joined: 18 Dec 2005 Posts: 887

pietinger · Posted: Fri May 27, 2022 8:51 am Post subject:

pietinger · Posted: Fri May 27, 2022 8:59 am Post subject:

P.S.: Vieri, please see my edit of my post before my last post.