GBit LAN and the PCI Bus?

NullDevice · Tux's lil' helper Joined: 15 Jul 2005 Posts: 84

Hi,

Actually i don't know if this is the correct sub-forum for this... its not really a linux-specific question. So ill just post it into Networking here.

Im about to upgrade my LAN to GBit, and buy 4 Gigabit NICs, with Intel Pro 1000 chipset, for my 4 Gentoo Machines.

Although only one of those has a PCIe Bus on the mainboard. The rest of them has like... the common PCI-bus or whatever it is called.
Might this slow down the GBit connection between them?

The common PCI slot can transfer 133 MByte/s, if i can trust Wikipedia, and if i did not misunderstand anything.
But GBit would mean like 125 MByte/s (in both directions or just one??)
If you consider, that the PCI bus will share this whole bandwith with all devices, i guess it will slow down the Gigabit connection. It would be slower than it could be, right?

PCIe 1x can transfer 250 MByte/s per slot, well i guess this is a theoretical value though.

Do you think i also should upgrade to PCIe hardware, to gain maximum network performence?
Or did i misunderstand something and there is no performance loss?
What are your expieriences with that?

Thx in advance,
Ndev
_________________
MfG,
NullDevice

Galahad · Tux's lil' helper Joined: 12 Feb 2003 Posts: 126

Standard PCI seems to limit network performance to about 550MBit.

The rest is your call.

NullDevice · Tux's lil' helper Joined: 15 Jul 2005 Posts: 84

nobspangle · Posted: Fri May 04, 2007 6:53 pm Post subject:

I doubt you'll get near 1Gbit for most applications anyhow.

I never see 5 minute averages on our Gbit network clear more than 130Mbit. Obviously this is a 5 minute average so there may be bursts that get much higher. This is with Nortel 5520 switches and all decent Gbit network cards on either PCI-X or PCI-e bus

NeddySeagoon · Posted: Fri May 04, 2007 7:26 pm Post subject:

NullDevice,

You need to consider end to end performance to determine where the bottlneck is.
You have the HDD to main memory, about 50MB/sec sustained
main memory to network card, 133MB/sec unidirectional theoretical max, you need packets both ways.
Then the same at the other end of the link.

In short, upgrading to a 1Gb network is a waste of money.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

NullDevice · Tux's lil' helper Joined: 15 Jul 2005 Posts: 84

NeddySeagoon · Posted: Fri May 04, 2007 8:30 pm Post subject:

NullDevice,

Yes. Feel free to run

wyv3rn · Apprentice Joined: 18 Aug 2005 Posts: 154 Location: USA

I wouldn't say upgrading to gigabit networking is worthless. You won't get full gigabit speed for the reasons stated above, but it still beats the pants off the typical 10-11MB/s you get with 100mb ethernet.

Oh, and the 33Mhz/32-bit PCI bus 133MB/s is a theoretical and doesn't account for overhead. In the real world about the best you can expect is ~120MB/s data throughput. BTW, spotting a 64-bit PCI bus slot is obvious but some motherboards have 66Mhz/32-bit PCI bus which looks identical to your typical PCI slot but achieves approximately double the throughput. In this case lspci -vv can help to determine if your motherboard is 66Mhz PCI bus capable. Of course the expansion card must also be capable to take advantage.