Problem with large file transfers

hurgh · Last edited by hurgh on Wed Oct 30, 2002 4:18 am; edited 1 time in total

Hey all.
This is my first post on the forums (although i am sure that i had some before but i could not login so i created a new user

)
Anyway, what i am here for is maybe some insite into a small problem that i have had, now, this problem has appeared on other distro's but i am using gentoo now so i am asking here incase there is some sort of trick that gentoo has to help deal with this problem.

Here goes.

I have a PC (PIII) setup with 2 HDD's 1x 10 gig for the system and users dir's, and 1x 40 gig for storage and serving of files to others on my network.

I have also installed a FTP server (pure-FTPd) and Apache as a webserver and Samba (all the latest package versions). The problem arises when ever i try to copy a large amount of data across the network, this can be in the form of 1 large file (Over 200Mb) or lots of smaller files. And it does not matter what i use to copy them, if i use a smb share it happens, if i use ftp, it happens, etc...

What is 'it' i hear you ask? Well 'it' is simply this. The gentoo box freezes. Yep, that's correct, the HDD light is stuck on, and i cant do anything with the box. i have to do a hard reset before i can do anything again. Now i have been told by many people to use hdparm -d1 (DMA On) which i have done, i have enabled DMA in my kernel image and still the problem happens. Oh, and it does not matter which way i do the transfer eaither, if i try to send the files to the gentoo box, it freezes (not straight away but after doing about half of the transfer, depending on the file size, or the ammount of files), and the same happens if i try and grab files from the gentoo box.

I have tried changing all the hardware, one by one, including putting all the files onto another hdd, i have tried a new network card, different switches, different hubs, different speeds, (10 / 100 m), and i have even tried different distro's.

If anyone has ever come across the same sort of problem and knows a fix or knows of something that i might try please let me know.

I seem to think it has something to do with hdparm or some software on the box, cause as i said it happens with different hardware also...

Any help at all with this matter would be much appreciated.

Thanks for taking the time to read this.
Also if i left any info out that you need to know, or you think that it might help someone else work out what the problem is, please let me know and i will add all the info that i can.

Thanks again

-|- Hurgh -|-

securiteaze · Posted: Wed Oct 30, 2002 3:33 am Post subject:

I have experienced similar problems to this using ftp (linux-ftp --> w2k-client) with files over 2GB, the transfer just hangs after a while. I got around it by using smbclient, the linux box only runs ssh and proftpd. Somehow by logging into the linux box and connecting to the windows share it works.

YMMV
_________________
Blah..

rac · Posted: Wed Oct 30, 2002 3:38 am Post subject:

What kernel sources are you using? If the answer is not vanilla, it might be worth trying a vanilla kernel to see if the problem persists.
_________________
For every higher wall, there is a taller ladder

hurgh · Posted: Wed Oct 30, 2002 4:16 am Post subject:

securiteaze:
Thanks for the info, it seems not to be related to FTP thought cause it happens for smb also, and the filesize is not a problem cause it happens when i try to transfer lots of little files at the same time (ie, select the files then let it work out which order it will copy them itself), and also with 1 big file.

rac:
I am using the canilla kernel and nearly excatly how it comes too, i dont think i made many changes at all (only the network module and maybe the ext3 FS). as i mentioned though the problem does happen with different distro's.

Oh and one thing that i forgot to mention b4 was the fact that i have tried it on both ext2 and ext3 filesystems, should i try it with another one or not?? I dont realy want to use another one....

Thanks

-|- Hurgh -|-

maxmc · Posted: Wed Oct 30, 2002 5:02 am Post subject:

This sounds an awfull lot like a problem my friend has.
But he's using windows...

He came to the conclution that his HDD wasn't compatible with the motherboard.
He has a Western Digital HDD and it seems they have a problem with
VIA KT133A and 686B chipsets.

Maybe that's your problem?
The solution is to buy a new HDD or a new Motherboard

hurgh · Posted: Wed Oct 30, 2002 5:38 am Post subject:

maxmc:

Hmmm, well i thought that it could be something to do with the hdd so i tried another hdd, the one i have in the atm is a 40gig WD, so i tried an 80gig Segate and still same thing

...

The chipset on this motherboard is a intel one ne way but anything is worth a go....

Thanks for the reply.

-|- Hurgh -|-

PS My guess is that it is a combination fo a few things, IE. the hdd not compatible with MB, or DMA does not work on current MB with that hdd, or something like that so i will keep swapping and stuff till i find a combo that works....

Sargon · Tux's lil' helper Joined: 23 Jan 2003 Posts: 90

I experience the same problem as you describe, hurgh.

It happens when I ftp big files over eth0 (100mbit/s) to my Mac, but not when I copy it over wlan0 (10mbit/s). It's a pain to backup several GBs with 10mbit/s... :o) But at least it works this way. I don't think that the interface (eth0/wlan0) is the problem, the speed is. I'm using the gentoo kernel sources.

Sargon

hurgh · Posted: Thu Oct 30, 2003 1:15 pm Post subject:

sargon:

didnt even remember this post existed

You are about 1 year too late, infact 1 year exactly, what are the odds...

ne way, i found the cause of this problem, and after fixing it, i have never experienced this again.

I found as others had suggested, that turning DMA on fixed all my problems, although i did have to get a new HDD before this worked as the HDD that i had did not support DMA mode at the time.

But whatever the case DMA fixed the problems, and i can d/l and u/l as much as i want at what ever speed i want too now

Hope that helps.

-Hurgh-
_________________
Website: http://www.hurgh.org/

Sargon · Tux's lil' helper Joined: 23 Jan 2003 Posts: 90

Dinini · Apprentice Joined: 11 Jun 2003 Posts: 236

This is the sending gentoo box that freezes? Wild guessing since I don't have any more information. (skipping asking about the harddrive since this is the sending machine and unless the drive is dying, it's transfer rate should at worst slow down the file transfer not lock the machine)

1) You have a buggy network card driver (what kernel version and nic?)

2) Network is congested, lots of collisions (lowering the send and receive windows might help here)

3) You have an interrupt timing problem. Try booting with acpi=off assuming you have it enabled. Enabling DMA on the harddrive lowers interrupt rate so it's also a good thing to try even if the transfer rate is fine without it.

4) (unlikely in the extreme but i did see it happen several years ago) your disk cache is for some reason paging into the swap file.

Course since I haven't slept yet, these could be lousy guesses