Linux Servers packets wont route, but Windows Server's will!

[njcwotx]

Hello all, I had a very strange problem today with some production Linux systems...very strange....

We have a large WAN and it runs mostly Winblows. However we do have a few Linux based boxes. In particular a Gentoo workstation and a Red Hat Enterprise 4 Server. The Red Hat box runs a Linux based database and application that users connect to via an ssh client. My Gentoo box is pretty much an admin's box that I use to resolve various issues, perform admin functions and scripting, and annoy my Linux hating boss.

Earlier today users of the linux based application started complaining of random disconnects. They were for brief durations about a minute or two in length, go away for a several minutes then return again. Typically when these disconnects occured in the past we lost a WAN link or some connectivity problem occured affecting all traffic and very easy to isolate. However, this time it became apperent after a while that all other services were unaffected except my Gentoo box and this Red Hat server. Windows servers were not being affected.

Doing a little tcpdump and wireshark and ping tests I could determine a packet would leave the remote site, hit the linux box and get answered by the linux box, only to get lost on the return trip. If i initiated a ping from a linux box, the packets would never arrive at the other end. We narrowed the search to a single Cisco router. When the intermittent problem would occur the packets would die there just as if there was some kind of routing glitch. Then it just clears up and works fine for a while. The issue had stopped after hours and some reboots, but they might very well return in the morning when users begin to use the system again.

The only problem was that this glitch only affects packets travelling from the LAN to the WAN and Linux servers only. We dont do any fancy routing, its all subnet based routes with no acls or any filtering to get in the way. The servers are on the subnet connected directly to the WAN interface. There are no ip tables on the servers. There are no changes made withing the last few weeks. It just started out of the blue. There is an MPLS WAN that does use a dynamic routing protocol like BGP or EIGRP (think its eigrp but the ATT engineer called it bgp for some reason). Anyway they claim innocence and told us to go fish. I somewhat suspect there may be something there, but I have no proof.

????????BIG QUESTION???????????????
Has anyone in their experiences came across a situtation where packets from a linux server be rejected but a windows server allowed?!?! This consumed most of my day and I dont have an answer.
_________________
Drinking from the fountain of knowldege.
Sometimes sipping.
Sometimes gulping.
Always thirsting.

[njcwotx]

An ip address conflict with the gateway was discovered. Appearently the Windows servers were remembering the mac address of the gateway and vice versa, while the linux boxes were switching between the true gateway and the conflicting address. Once we thought the issue cleared on its own we change the address on one linux box and the issue returned. Since we could not ping a linux host from the router on the directly connected network I thought to try clearing the arp cache on the router. After this the traffic would pass for a bit then stop. Shutting down the gateway and pinging helped us find the problem device. proxy arp settings on the cisco router may have assisted the windows servers staying up but I am not 100% sure. A little research on the web showed this was a possibility. Anyway, I post this in case someone else gets this issue.
_________________
Drinking from the fountain of knowldege.
Sometimes sipping.
Sometimes gulping.
Always thirsting.