Network hangs for ~60 seconds, randomly

[mv]

Hello,

I have a difficult to reproduce problem:

I am connected over some router with some DSL network. Occassionally (all 5-120 minutes) the network "hangs" in the sense that all running connections (be it ssh, wget, streaming a movie in a browser, trying to attempt a new browser page, pinging a known ip, opening a new ssh connection) hang for 20-300 second (usually ~60 seconds). No error message or sign what should be wrong. After this time, everything is working again as it should.

The strange thing is that during that time, I can connect with android phone over the same (WLAN) router without any problems.

So I guess that it must be something on my gentoo host which is falsely configured...

Is there an "obvious" candidate for e.g. a missing kernel or dhcpcd option? How could I debug this issue?

[NeddySeagoon]

mv,

An r8169 network card with a missing firmware patch?
dmesg will tell you all about it. It may even add a 60 sec delay to your boot while the driver looks for firmware.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[mv]

[NeddySeagoon]

mv,

Firmware loading is only attempted at boot. In theory, its not needed but it exists to fix bugs.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[mv]

Apparently in my case it really fixes a bug: So far, I had no hangs for several hours.

It is still strange that the problem did not occur in my previous place (with another router): Apparenty the triggering of the bug depends on the router hard- or software.

As so often: Thank you very much for your help.

[mv]

I posted too early: The hang occured again, actually now rather often.

It was apprently just by accident that it worked for some hours.

[mv]

Bump... Nobody?

[NeddySeagoon]

mv,

Try running wireshark or tumpdump on the interface. I don't know what to tell you to look for and you need to see a pattern.
If it is traffic related, the traffic causing the problem may not even get onto the wire.

Try a plug in NIC if thats possible. Replace the ethernet cable, try a different router port ...
Just change one thing at a time unless you plan a binary search.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[Cyker]

I reckon it's the NIC.

I have had extreme variability with later gen RealTek NICs in Linux; Some work great but with others I've just had no end of problems. The later revision 8111's are especially flakey.

On my old Asus Socket 939 mobo I had to completely disable MSI/MSI-X capabilities in the kernel to make it work. I never got the one on my A88XM-E45 to work stably, it would freeze every now and then and even hard-lock the whole system under sustained high load.

Ended up buying a PCIe NIC based on an Intel chipset and haven't had any trouble since!

[mv]

Thanks for the input.

Since I am loggging with tcpdump -i lan0 >logfile, the problem did not re-occur. Maybe logging itself has some effect on it? But it might be just an accident.

Yesterday, I removed CONFIG_PCI_MSI and CONFIG_PCI_MMCONFIG from the kernel options - I had them in only because the kernel help text strongly recommended it. I do not really understand what they do and hope that removing them will not influence any USB devices, because my USB controller uses PCI. My other PCI devices are

[NeddySeagoon]

mv,

CONFIG_PCI_MSI is a good thing to have. in the dim and distant past, when an IRQ was acknowledged, the IRQ service routine had to search all the devices that could raise that particular IRQ before it could run the right interrupt service routine.

MSI means Message Signalled Interrupt. There is some initial setup. Devices that can raise a MSI are programmed with the address of the service routine.
When their IRQ is acknowledged, they place the address of the service routine on the bus, so the CPU can jump straight to it.
Real systems have always done it this way, its just come late to the PC.
Both ways work. MSI reduces the overhead. MSI capable devices will cooperate with the old way.

CONFIG_PCI_MMCONFIG memory maps the PCI configuration space for those devices that support it.
This is faster that using I/O ports to read the configuration space, which is the old way.
Again, both ways work.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[mv]

Thanks for the explanation. So it seems I should disable both only it as a fallback.

I will continue logging and see how it works out with both enabled. As mentioned, I cannot test for a week now (or maybe even longer), so do not expect a report any soon.

[mv]

So far no fix. The hangs continue in random intervals; tcpdump did not provide anything useful.

Edit: Currently trying with the net-misc/r8168 module.

[mv]

Now I found a way to "remove" the hang:

When the system hangs, executing

[khayyam]

[mv]

[khayyam]

[mv]

Sorry for my late reply: Currently, it takes many hours until the problem occurs, and then the network is back very quickly, so testing is in the moment very slow.
(There have been other phases were the problem occured every 1-2 minutes, and the network did not come back at all until I did /etc/init.d/dhcpcd restart).

[krinn]

Here's a real good candidate for you mv: https://bugs.gentoo.org/show_bug.cgi?id=477356
as the note may not be in all ebuild, you might have not seen it, or maybe you've been a bad boy

[khayyam]

[UberLord]

[mv]

The problem is not solved. I can now confirm that

[mv]

As excpected, the change from the previous Edit 1 did not help.

Instead of redirecting the tcpdump output, I looked at it now on screen, and it continued during the hang. The hang started somewhere during the following output (probably near the end).
I already observed earlier that the ARP messages (with "oui unknown") always appear in the neighborhood of the hang. However, these messages often apear also when there is no hang, so it might be a red herring.
I edited the output manually to shorten it and to avoid publishing local hardware, IP adresses, or hostname.
The hang occured while I was connected with ssh to machine xxx.xxx.xxx.xx.
When asking for my external ip with whatsmyip after the hang, it was not yyyy.yyy.yyy.yy, so maybe the problem occured in the moment when the provider wanted to associate me a new externel IP? (On the other hand, I cannot imagine this, because sometimes the hang is consecutively every few minutes).

[khayyam]

mv ...

I think I've just blown a fuse trying to make sense of your obfuscation :) ... are you saying, wrt 'whatsmyip', 'yyy.yyy.yyy.yy' is the IP of the external interface? If so then "who-has yyy.yyy.yyy.yy [...] tell yyy.yyy.yyy.yy" is gratuitous and I'm fairly sure you shouldn't see this locally.

best ... khay

[Logicien]