| View previous topic :: View next topic |
| Author |
Message |
chix4mat
Apprentice
Joined: 11 Dec 2005
Posts: 190
Location: Canada
|
 Posted: Mon Mar 31, 2008 5:00 am Post subject: Solved: "File too large" errors when using rsync.. |
 |
|
Hi all,
I have network-attached storage (Maxtor Shared Storage II) that I keep backups and other things on, but have run into an issue where rsync cannot copy a file over 2GB in size. This is bad for me since I have nightly backups that weigh in at around 5GB to be copied there, but something is preventing it from happening. I have searched the web far and wide, but have been unable to find hard evidence of what would cause this, or how to work around it.
At first, I thought it was a Samba issue, but I don't think it is. I can copy the file using Konqueror, with absolutely no issue.
| Code: | username Desktop # rsync -av archive.tar.gz /mnt/nas/Backups/Server/
sending incremental file list
archive.tar.gz
rsync: writefd_unbuffered failed to write 4 bytes [sender]: Broken pipe (32)
rsync: write failed on "/mnt/nas/Backups/Server/archive.tar.gz": File too large (27)
rsync error: error in file IO (code 11) at receiver.c(298) [receiver=3.0.0]
rsync: connection unexpectedly closed (28 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.0] |
Is there a way to work around this issue? Is there an alternative to rsync or cp that would allow me to make such a large copy? I need to have something that works with a script.
I appreciate any help!
_________________
emerge -NuDe world
Last edited by chix4mat on Sun Apr 27, 2008 8:32 pm; edited 1 time in total |
|
| Back to top |
|
 |
Keruskerfuerst
Advocate
Joined: 01 Feb 2006
Posts: 2289
Location: near Augsburg, Germany
|
| Posted: Mon Mar 31, 2008 5:57 am Post subject: |
|
|
emerge --info?
Can you copy a compressed file with size over 2GB? (not with rsync, tar.bz2).
If yes, I think this is a bug in rsync. |
|
| Back to top |
|
|
hirakendu
Guru
Joined: 24 Jan 2007
Posts: 386
Location: san diego
|
| Posted: Mon Mar 31, 2008 6:32 am Post subject: |
|
|
also see if its a vfat (fat32) partition. although the limit of fat32 is 4gb. had a bad time because of that few days back  .
_________________
Helium Sources || Gentoo Minimal Livecd |
|
| Back to top |
|
|
chix4mat
Apprentice
Joined: 11 Dec 2005
Posts: 190
Location: Canada
|
| Posted: Mon Mar 31, 2008 8:01 am Post subject: |
|
|
| Keruskerfuerst wrote: | emerge --info?
Can you copy a compressed file with size over 2GB? (not with rsync, tar.bz2).
If yes, I think this is a bug in rsync. |
Yes, I mentioned I could copy the file fine using Konqueror. The archive size is 5.3GB. I cannot copy it with "cp" and rsync.
| hirakendu wrote: | | also see if its a vfat (fat32) partition. although the limit of fat32 is 4gb. had a bad time because of that few days back . |
I am not sure how I can find this out without physically hauling the hard drives out and hooking them to the PC. It doesn't say anywhere in the admin system. I'll e-mail the company and see if I can find out.
I should note that I've never been able to copy a large file like this with any NAS box, so this particular one is not unique. It is the first one, however, that has allowed me to create and copy symlinks, which is helpful.
_________________
emerge -NuDe world |
|
| Back to top |
|
|
Keruskerfuerst
Advocate
Joined: 01 Feb 2006
Posts: 2289
Location: near Augsburg, Germany
|
| Posted: Mon Mar 31, 2008 10:33 am Post subject: |
|
|
You can check out the filesystem type with
or
Are these commands (cp, mv, rsync) working internally (between two folders or partitions)? |
|
| Back to top |
|
|
chix4mat
Apprentice
Joined: 11 Dec 2005
Posts: 190
Location: Canada
|
| Posted: Mon Mar 31, 2008 9:35 pm Post subject: |
|
|
| Keruskerfuerst wrote: | You can check out the filesystem type with
or
Are these commands (cp, mv, rsync) working internally (between two folders or partitions)? |
As it's a network drive, it's mounted as smbfs, but as for the hard drives themselves, I'm unsure. I am in the process of finding an answer. I am unsure how much smbfs differs from Samba as an application, but I am able to copy via Samba through Konqueror without issue, so I'm unsure if Samba or smbfs is at fault here.
And yes, the commands all work anywhere else... just not with the NAS. It's becoming a bit frustrating though. If it was a 4GB limit, I would understand the issue a bit more (despite being on a 64-Bit OS), but for 2GB... I can't understand it.
Edit: Just heard back from Maxtor... the hard drives in the NAS box are indeed ext3.
_________________
emerge -NuDe world |
|
| Back to top |
|
|
hirakendu
Guru
Joined: 24 Jan 2007
Posts: 386
Location: san diego
|
|
| Back to top |
|
|
kernelOfTruth
Watchman
Joined: 20 Dec 2005
Posts: 6111
Location: Vienna, Austria; Germany; hello world :)
|
|
| Back to top |
|
|
chix4mat
Apprentice
Joined: 11 Dec 2005
Posts: 190
Location: Canada
|
| Posted: Thu Apr 03, 2008 4:21 am Post subject: |
|
|
| hirakendu wrote: | i hope by now you are done with some sort of file splittng software  . |
File splitting would be fine for a normal file, but the reason for these backups are in case something goes wrong with a website. Putting a file back together and then extracting it takes precious time. I could just rsync entire folders, but I won't be able to retain permissions or usergroups that way.
| kernelOfTruth wrote: | you tried re-compiling rsync / ... with | Code: | | -D_FILE_OFFSET_BITS=64 |
?
| Quote: | | On 32-bit systems, enables file sizes over 2Gigabytes. Useful for ftp of iso images works with lftp gftp (Opera probably) |
source |
Thanks for the advice... but that didn't make any difference either 
_________________
emerge -NuDe world |
|
| Back to top |
|
|
chix4mat
Apprentice
Joined: 11 Dec 2005
Posts: 190
Location: Canada
|
| Posted: Sun Apr 27, 2008 8:31 pm Post subject: |
|
|
Hi all,
In case anyone references this thread in the future, the solution to my problem turned out to be cifs. I noticed after upgrading to the latest kernel that smbfs would be removed entirely in 2.6.27, and recommended cifs (which I didn't even know about until now). I moved on over and things are running a lot smoother. I successfully rsync'd 6GB files without issue, and notice an overall performance increase as well.
Here's an example of the mounting scheme, and fstab entry:
Mounting from command-line:
mount.cifs //192.168.1.5/Public /mnt/nas -o user=admin%password
fstab:
//192.168.1.5/Public /mnt/nas cifs auto,owner,rw,username=admin,password=password,uid=1000,gid=100 0 0
You'll also want to emerge net-fs/mount-cifs to read through the man entry.
_________________
emerge -NuDe world |
|
| Back to top |
|
|
|
Unsupported Software
