| View previous topic :: View next topic |
| Author |
Message |
number_nine
Tux's lil' helper
Joined: 05 May 2005
Posts: 136
|
 Posted: Mon Jul 02, 2007 3:11 am Post subject: Gigabit network performance |
 |
|
I've got a fileserver (running gentoo) where I house all my media. Transfers from the fileserver to the media PC (also running gentoo) are extremely slow. However, transfers from the fileserver to other computers on the home LAN are at least 2x as fast.
Example: transferring a 2.4 GB file via NFS from the fileserver to the media PC takes 2:49 (about 15 MB/sec). The same transfer on other machines took 1:20 or less. I tried to do the same transfer via CIFS (samba mount), and it only transferred 17 MB in 5 minutes!
Any hint as to what could cause the severe network performance problem on the media PC?
Also, all computers on the LAN have their MTU set to 1500 (the default). For an experiment, I tried using "jumbo frames", ala:
| Code: | | ifconfig eth0 mtu 9000 |
All my machines seem to support this, except the media PC, which says:
| Code: | | SIOCSIFMTU: Invalid argument |
The machine is using the Abit NF-M2 nView (alt link) motherboard. This has the nVidia nForce 430 south bridge. Does anyone know if this integrated NIC supports jumbo frames? I have another machine using an integrated nvidia ethernet device, and it does support jumbo frames.
The only other thing that's unusual about the media PC is that it's connected by the longest cable. But it's a purchased (not homemade) cable, and it's only 100 ft long.
Relevant info:
| Code: |
$ ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:50:8D:98:D0:3C
inet addr:192.168.1.34 Bcast:192.168.255.255 Mask:255.255.0.0
UP BROADCAST NOTRAILERS RUNNING MULTICAST MTU:1500 Metric:1
RX packets:13886363 errors:121 dropped:0 overruns:0 frame:0
TX packets:9391786 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:19843540531 (18924.2 Mb) TX bytes:11549050130 (11014.0 Mb)
Interrupt:23 Base address:0x2000
$ ethtool eth0
Settings for eth0:
Supported ports: [ MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 1
Transceiver: external
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Link detected: yes
$ dmesg
...
forcedeth.c: Reverse Engineered nForce ethernet driver. Version 0.57.
ACPI: PCI Interrupt Link [APCH] enabled at IRQ 23
ACPI: PCI Interrupt 0000:00:14.0[A] -> Link [APCH] -> GSI 23 (level, low) -> IRQ 23
PCI: Setting latency timer of device 0000:00:14.0 to 64
forcedeth: using HIGHDMA
eth0: forcedeth.c: subsystem: 0147b:1c26 bound to 0000:00:14.0
dmfe: Davicom DM9xxx net driver, version 1.36.4 (2002-01-17)
...
$ lspci -vx
...
00:14.0 Bridge: nVidia Corporation MCP51 Ethernet Controller (rev a1)
Subsystem: ABIT Computer Corp. Unknown device 1c26
Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 23
Memory at fe02b000 (32-bit, non-prefetchable) [size=4K]
I/O ports at c800 [size=8]
Capabilities: [44] Power Management version 2
00: de 10 69 02 07 00 b0 00 a1 00 80 06 00 00 00 00
10: 00 b0 02 fe 01 c8 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 7b 14 26 1c
30: 00 00 00 00 44 00 00 00 00 00 00 00 05 01 01 14
...
$ cat /proc/interrupts
CPU0 CPU1
0: 544743919 406279734 XT-PIC-XT timer
1: 0 2 IO-APIC-edge i8042
9: 0 26 IO-APIC-fasteoi acpi
12: 0 4 IO-APIC-edge i8042
14: 644 66411 IO-APIC-edge ide0
16: 49468 26872505 IO-APIC-fasteoi nvidia
18: 6364 2585281 IO-APIC-fasteoi cx88[0], cx88[0], cx88[0]
19: 0 3 IO-APIC-fasteoi ohci1394
20: 0 3 IO-APIC-fasteoi ehci_hcd:usb1
21: 0 0 IO-APIC-fasteoi libata
22: 17200 4233341 IO-APIC-fasteoi libata, HDA Intel
23: 143928 159405842 IO-APIC-fasteoi ohci_hcd:usb2, eth0
NMI: 949175 949078
LOC: 950874179 950874175
ERR: 0
|
Any hints or ideas would be much appreciated.
Thanks! |
|
| Back to top |
|
 |
poly_poly-man
Advocate
Joined: 06 Dec 2006
Posts: 2477
Location: RIT, NY, US
|
| Posted: Mon Jul 02, 2007 4:21 am Post subject: |
|
|
What's the cpu/memory load on the media PC?
It's unlikely that it is network-related, as everything appears okay. 100ft is well under the 384ft (or is it m? - either way...) limit.
hih,
poly-p man
_________________
iVBORw0KGgoAAAANSUhEUgAAA
avatar: new version of logo - see topic 838248. Potentially still a WiP. |
|
| Back to top |
|
|
pdr
l33t
Joined: 20 Mar 2004
Posts: 618
|
| Posted: Mon Jul 02, 2007 11:03 am Post subject: |
|
|
I use cat-6 on the Gb part of my network..
Do your ports have a light indicator that tells if they are operating in 100Mb vs 1Gb mode? Mine have a light that is green for Gb, yellow for 100Mb, off for 10Mb..
If going through a router/switch - is the port used by the media pc Gb? Some have a mix of Gb and 100Mb ports.. |
|
| Back to top |
|
|
number_nine
Tux's lil' helper
Joined: 05 May 2005
Posts: 136
|
| Posted: Mon Jul 02, 2007 12:30 pm Post subject: |
|
|
| poly_poly-man wrote: | What's the cpu/memory load on the media PC?
It's unlikely that it is network-related, as everything appears okay. 100ft is well under the 384ft (or is it m? - either way...) limit. |
The machine is an X2 3800, running two instances of seti@home. But the seti process is running at nice 19. Also, all my other machines (file server included) are running seti@home, and it doesn't hurt network performance. Machine is otherwise unloaded.
Here's something that did catch my eye though:
| Code: |
$ cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 75
model name : AMD Athlon(tm) 64 X2 Dual Core Processor 3800+
stepping : 2
cpu MHz : 2200.020
cache size : 512 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow pni cx16 lahf_lm cmp_legacy svm cr8_legacy
bogomips : 4402.23
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc
processor : 1
vendor_id : AuthenticAMD
cpu family : 15
model : 75
model name : AMD Athlon(tm) 64 X2 Dual Core Processor 3800+
stepping : 2
cpu MHz : 2200.020
cache size : 512 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow pni cx16 lahf_lm cmp_legacy svm cr8_legacy
bogomips : 972.38
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc
|
Notice that the bogomips value is different for the two processors? By a factor of four! I know that bogomips are "bogus", but I'd expect the value to be exactly the same for each core. Also, if you look above at the output of "cat /proc/interrupts", you can see that CPU1 services the majority of interrupts. In the case of eth0, it's by a factor of 1000. But going by cpuinfo, the kernel detects CPU1 as having the lower BogoMIPS value.
Is it possible that I have a bum CPU? |
|
| Back to top |
|
|
number_nine
Tux's lil' helper
Joined: 05 May 2005
Posts: 136
|
| Posted: Mon Jul 02, 2007 12:34 pm Post subject: |
|
|
| pdr wrote: | I use cat-6 on the Gb part of my network..
Do your ports have a light indicator that tells if they are operating in 100Mb vs 1Gb mode? Mine have a light that is green for Gb, yellow for 100Mb, off for 10Mb..
If going through a router/switch - is the port used by the media pc Gb? Some have a mix of Gb and 100Mb ports.. |
Good idea. I'll double check when I get home, but I'm 99% sure that the media PC is plugged into a gigabit port (this is my switch, FWIW). Also, I did check the lights on the switch last night (all ports are running at Gb speed).
I suppose I should still check the port on the media PC itself.
Thanks! |
|
| Back to top |
|
|
markp2000
n00b
Joined: 30 Oct 2004
Posts: 41
|
| Posted: Tue Jul 03, 2007 3:21 am Post subject: |
|
|
I ran this around for awhile myself. I found that I had a bad NIC. I was using the onboard NIC, but once I got a GB nic everything went back to normal. Do you have a spare nic to through in there?
Mark |
|
| Back to top |
|
|
number_nine
Tux's lil' helper
Joined: 05 May 2005
Posts: 136
|
| Posted: Tue Jul 03, 2007 4:02 am Post subject: |
|
|
| markp2000 wrote: | | I ran this around for awhile myself. I found that I had a bad NIC. I was using the onboard NIC, but once I got a GB nic everything went back to normal. Do you have a spare nic to through in there? |
You had that weird bogomips problem as well, or just the crummy transfer speeds?
I actually just ordered a nice Intel PCIe gigabit NIC, but I was going to put it in the fileserver. Looks like I'll be using it for the media PC now. |
|
| Back to top |
|
|
number_nine
Tux's lil' helper
Joined: 05 May 2005
Posts: 136
|
| Posted: Tue Jul 03, 2007 2:29 pm Post subject: |
|
|
Update: I used the net-analyzer/nttcp program to do some more testing. This program sends data from memory (rather than disk), hopefully eliminating disk I/O bottlenecks (i.e. allows you to isolate what is actually being tested, which is just the network).
Using this tool, raw TCP and UDP throughput were fairly consistent between the file server and all other machines (including the media PC).
So it appears that the problem actually lies with NFS and/or disk I/O on the media PC.
Thanks!
By the way, that nttcp program is really easy to use. |
|
| Back to top |
|
|
markp2000
n00b
Joined: 30 Oct 2004
Posts: 41
|
| Posted: Thu Jul 05, 2007 9:59 pm Post subject: |
|
|
| It is the slow transfer speed of files. I have the same setup of a gentoo fileserver and mythbox media pc. It was a bad integrated NIC. |
|
| Back to top |
|
|
number_nine
Tux's lil' helper
Joined: 05 May 2005
Posts: 136
|
| Posted: Sat Jul 07, 2007 10:54 pm Post subject: |
|
|
| markp2000 wrote: | | It is the slow transfer speed of files. I have the same setup of a gentoo fileserver and mythbox media pc. It was a bad integrated NIC. |
I don't think it's the NIC. See my follow-up post to this one. I went into more detail about using the nttcp program, as well as other info. At this point, I'm pretty sure the problem has to do with slow disk I/O on the media PC. But whether that is due to the hardware, filesystem, or some other factor, I'm not sure. |
|
| Back to top |
|
|
|
Networking & Security
