Multiple NIC problems

DeathCount · n00b Joined: 18 Jun 2004 Posts: 8

Hello everybody,

Having a few issues with my multi NIC server. I have three gigabit network ports (1x onboard and 2x on a dual port PCIe) and for some reason all three initialize like they should but only one ever gets used.
For example, my onboard nic always has all traffic go through it regardless of intended ip address. So no matter what ip I ping (192.168.1.25, 192.168.1.26, 192.168.1.27) it all goes through the onboard port.
I know this because when I unplug the cable from it, I am unable to ping any address even though the other two still have cables and lights on the router. I am using gentoo-sources-2.6.36-r5 with
udev-151-r4. Thanks for your help!

My onboard NIC: (r8169)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01)

My PCIe NICs: (e1000e)
02:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
02:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)

ifconfig

Jaglover · Posted: Sat Feb 19, 2011 1:00 am Post subject:

You can't have multiple NICs on same subnet just like that.

http://www.google.com/linux?hl=en&safe=off&q=nic+bonding&btnG=Search
_________________
My Gentoo installation notes.
Please learn how to denote units correctly!

your_WooDness · Tux's lil' helper Joined: 25 Oct 2007 Posts: 77

Hi DeathCount,

this is because of the so called "arp flux". If you have several NICs in one subnet, the first NIC will answer on the MAC layer address for all addresses on the subnet. So all arp tables of switches and other servers will have only one MAC address for, in your case three IP addresses. You can check this by using arping from a host pingin the host with the three NICs. You will get one and the same MAC address for each IP address.

There are different ways to work around this. Not sure, but I think there is a option somewhere in /proc you can alter. Search for "arp flux". Best would be to configure a bonding on the NICs if you only use one subnet.

WooD

TJNII · l33t Joined: 09 Nov 2003 Posts: 648 Location: for(;;);

That config will never work the way you expect, regardless of ARP. Ask yourself these questions: What is my IP? What are my routes? What ethernet device is used for which subnet? What IP do other machines use to connect to me?

Ethernet bonding isn't going to work unless you have something on the other end which supports bonding. If you don't have a bonding capable switch, you can't share traffic across all 3 nics transparently. If you have a home switch, you're SOL. Buy a managed switch.

You can hack around this with routing, by having all local traffic go through eth0 and all the outside traffic go through eth1. That is just a suboptimal hack using 2 Nics, though.

Buy a bridge capable switch and set up link bonding. That will do what you want.

DeathCount · n00b Joined: 18 Jun 2004 Posts: 8

Thanks for the replies everybody. What about setting up bonding with mode 6? I hear that has the benefits of 802.3ad without special hardware. Am I incorrect?

TJNII · l33t Joined: 09 Nov 2003 Posts: 648 Location: for(;;);

AFAIK that isn't true bonding, then. With true bonding all the ports function as one with one MAC address. That is why you need a bonding capable switch, as a regular switch will be very confused by having 3 pots share a MAC address.

However, that is just semantics. If it will do what you want, go for it. Keep in mind that that won't load balance across multiple NICs on a per connection basis using that methodology, though. Traffic to one endpoint will go through one NIC, not all 3. However, traffic to 3 endpoints should use all 3 NICs.

TJNII · l33t Joined: 09 Nov 2003 Posts: 648 Location: for(;;);

Also, out of curiosity, what are you doing that requires so much chutzpah from your network? I've never been able to choke a 1 gig connection, I've always hit other throttles first.

DeathCount · n00b Joined: 18 Jun 2004 Posts: 8

I'm not actually hitting a bottleneck really but it's more for multi-tasking. I originally set this up like this for LAN parties. The onboard nic would dedicate to serving all game server traffic while the other 2 nics would serve patches, files, isos, etc. full speed to multiple users without affecting game server traffic. When I'm not having a LAN party, I still would want the onboard to continue to serve game server traffic for my online gaming community and other "server" stuff such as SSH while the other two would do file sharing and 1080p video streams to multiple devices throughout the house at the same time. So in reality, no real reason besides multi-tasking.
Same reason this server has a quad-core processor and 2 hard drives setup in raid0 for sharing.... No real reason besides multi-tasking, and the fact I can do it.

Thanks again for all your help everybody.