unusual network speed issues when using rsync...

[rishi]

Hi there,

Hoping for some ideas towards solving this odd behavior.

I am copying about 1000GB of data from three servers to a portable FireWire/1394a drive on my local gentoo workstation using rsync (archiving obsolete job data). The arrangement is like this:

- Server A (SME Server i686 box).
- Server B (SuSE 10.2).
- Server C (Intel ReadyNAS).

- Minitar gigabit switch, all gigabit networking.

- Local workstation running gentoo. This is where the rsync command is being run.

- Western Digital MyBook 2TB 1394a portable hard disk drive.

If I run one rsync command copying data from any of servers A, B or C to the 1394 disk I get about 500kB/s total transfer speed, on average, as reported by rsync --progress option. Unsatisfactory performance, will take days longer.

If I run two or three rsync commands at once (on separate terminals obviously), each from separate server sources, EACH rsync transfer runs at around 5,000kB/s, giving me total data writing throughput to the 1394 disk of either 10,000kB/s or 15,000kB/s. Satisfactory performance, will only take a couple of days to complete.

In this second case, once two servers have finished their work and only one rsync command is left with work to do, it drops back down to 500kB/s transfer speed. Frustrating, takes a long time to get to the finish!

What could be causing this? It is some sort of QoS issue? Happy to provide any information that might help identify the problem here.

Any help appreciated. Cheers,

EDIT: Changed subject as this seems to be rsync specific.
_________________
--
Rishi
Melbourne, Australia.

[spOOwn]

Did you try to transfer a file with, by instance, a simple scp ? I means without using rsync !

Without rsync did you get also bad performance on transfering a file ?

Is there any firewall rule in one host to limit transfer perhaps ?
_________________
###########################
# Web site : http://sys-admin.wikidot.com #
###########################

[rishi]

Hi sp00wn, thanks for the ideas,

I have the network drives mounted locally two using cifs and one using smbfs. So to test another way I'm using cp to copy a bunch of data across. This goes straight to 5,000kB/s+.

So yes, perhaps the problem seems to relate to the use of rsync. The rsync command I'm using is something like this (run from within a batch script):

[spOOwn]

I known that rsync compute a checksum on each block of your file or something like that, perhaps your computer isn't faster enough to do this, and hence the transfer is only !!

I never did a transfer so much, so I don't really known !

Check your cpu usage when you are doing the rsync !
_________________
###########################
# Web site : http://sys-admin.wikidot.com #
###########################

[rishi]

Hi sp00wn,

I don't think it is a CPU bottleneck as when you work it harder (two transfers at once) it actually runs faster.

The workstation running the rsync is P4 3Ghz 1GB RAM so it should be up to the task anyway.

Here is my emerge info:

[rishi]

I've edited the subject line as this seems to be rsync specific.

To summarize:

Copying data from a few different file servers on the local network, all mounted locally. Copying to an external FireWire 1394a HDD (mounted locally). (Same result when plugged in via USB).

If I run one instance of rsync the transfer rate is about 500kB/s. (Much slower than it should run).

If I run two or more instances of rsync, the transfer rate is about 5,000kB/s for EACH instance. (i.e. 10,000kB/s+ total).

Using other programs like cp runs at full speed even if only one instance is run at once.

So rsync is capable of doing the job nice and fast, but for some reason it is not... why would this be?
_________________
--
Rishi
Melbourne, Australia.

[SeaTiger]

Try to do the multiple rsync test again and use iftop to monitor the transfer speed.

I am guessing (pure guessing, no real experience) that rsync is reporting fake transfer rate when you run multiple copies at the same time (from the same account?).

Anyway, give iftop a try.

[rishi]

Hi junksiu,

I tried iftop to verify the rsync reported speeds, and they look about right / verified. I also have a superkaramba monitor running which is reporting the same speeds too.

So it looks like the speeds being reported by rsync are about right. It really is running slow when there is one instance and it really is running fast when there are two instances running...

New Info: I have tried this on a SuSE linux workstation we have here. I used the same external drive, the same network shares and the same script running rsync. The rsync transfers run at full speed when there is only one instance of rsync running. Also, as you would expect, when you run multiple instances of rsync they all run a bit slower as they are sharing the available bandwidth.

So this problem seems to be confined to the gentoo workstation!

What could be different on the gentoo box? I'll check for:
Different versions of rsync.
Different kernel versions.
Different firewalls / QoS?
_________________
--
Rishi
Melbourne, Australia.

[rishi]

More testing:

I have a single rsync running and it is plugging away at 500kB/s. Has been running like this steadily for 30mins or more. Total network use is reported at around 500kB/s too.

I use Konqueror to drag and drop about 1Gb of data from a network share onto a USB stick. Konqueror reports this transfer going through at around 4.5MB/s. I notice total network use has jumped up to over 10MB/s (even getting up to 15).

I look back at the rsync while this Konqueror data copy is running and it has jumped up to 5MB/s+.

As soon as the Konqueror copy finishes, the rsync rate drops back down to 500kB/s. All confirmed against total network usage monitored with iftop.

Why won't it stay at full speed..?
_________________
--
Rishi
Melbourne, Australia.

[SeaTiger]

What is your rsync nice level?