| View previous topic :: View next topic |
| Author |
Message |
lukmih
n00b
Joined: 16 Nov 2005
Posts: 13
|
 Posted: Tue Sep 12, 2006 2:28 am Post subject: Large file corrupted when copied |
 |
|
This has been a general annoyance for me. But when I copy a large file inside kde it tends to not end up the same thing. I mostly copy video files around and notice keyframes missing and other strange distortions. I checked with md5sum and got two different results as well on any given screwed up file set.
Most of the time, I just need to copy them again from the command line and I end up with a good file.
I am just wondering if this is a known issue or if it is not. I also never really did any bug reporting before, so I don't want to waste anyones time.
Both file systems are reiserfs and on sata disk drives with no lvm though those modules are built into the 2.6.17-gentoo-r7 kernel.
I bring this up because I don't want anyone to run into this and not notice until its too late. If required I'll gladly do more testing or whatever if someone can tell me how to recursivly run md5sums on a bunch of files in one folder and compare to files in another mountpoint that is completely full so I can move some data around and have the space cause I have "written myself into a corner here".
/dev/sda2 93G 93G 0 100% /mnt/downloads
/dev/sdb1 373G 190G 183G 51% /mnt/store
Last edited by lukmih on Thu Sep 14, 2006 3:46 am; edited 1 time in total |
|
| Back to top |
|
 |
desultory
Bodhisattva
Joined: 04 Nov 2005
Posts: 9410
|
| Posted: Tue Sep 12, 2006 5:08 am Post subject: Re: kde large file copy |
|
|
| lukmih wrote: | | Most of the time, I just need to copy them again from the command line and I end up with a good file. |
To verify, files are consistently copied properly with cp but are corrupted when copied via KDE, correct?
Is this effect restricted to some subset of file types?
| lukmih wrote: | | I am just wondering if this is a known issue or if it is not. |
It would appear not.
| lukmih wrote: | | I also never really did any bug reporting before, so I don't want to waste anyones time. |
Basic rules of bug reporting, in no particular order:- Try to reproduce the effect, document how to reproduce the effect.
- Document what is effected.
- Document what, if anything, is not effected when treated in a similar fashion.
- Document what software exactly is being run and if possible what options it was built with.
- When unsure whether to include some data in the bug report, put it in the bug report.
- If you have found a reproducible bug which has not yet been reported, it is not a waste of time to report it.
| lukmih wrote: | | If required I'll gladly do more testing or whatever if someone can tell me how to recursivly run md5sums on a bunch of files in one folder and compare to files in another mountpoint that is completely full so I can move some data around and have the space cause I have "written myself into a corner here". |
To run md5sum on a tree of files:
| Code: | cd $root_of_tree
find . -type f -exec md5sum {} \; > md5sums |
To verify the checksums present in the file md5sums match the files present in the locations listed:
Note that when contructed as above the paths listed in the file md5sums will be relative paths, not absolute paths, which allows for checking of identical trees, even if the root of the tree has a different name, without needing to edit the checksums file to correct the paths for the tree to be verified. |
|
| Back to top |
|
|
lukmih
n00b
Joined: 16 Nov 2005
Posts: 13
|
| Posted: Thu Sep 14, 2006 2:26 am Post subject: large file copy on sata drives |
|
|
This is probably not related to kde. When copies are made from one drive to the other in a console ssh session things act funny too. Here is what I am noticing:
| Code: |
luke@evil /mnt/downloads $ cp Cheng* /mnt/store/test/
luke@evil /mnt/downloads $ md5sum Cheng*
42104854dc2036cc4d21578080cf9bbc Chenga 1-06.wmv
b1ecafb95acc1ca59f4a6aefaed6a75e Chenga 1-06low.wmv
02b87915a0dfbd8b15f9c85ed158d846 Chenga 1-06med.wmv
luke@evil /mnt/downloads $ md5sum /mnt/store/test/Cheng*
832a65d9166fd5f3fcfab86ece33ea64 /mnt/store/test/Chenga 1-06.wmv
b1ecafb95acc1ca59f4a6aefaed6a75e /mnt/store/test/Chenga 1-06low.wmv
02b87915a0dfbd8b15f9c85ed158d846 /mnt/store/test/Chenga 1-06med.wmv
luke@evil /mnt/downloads $ cp Cheng* /mnt/store/test/
luke@evil /mnt/downloads $ md5sum /mnt/store/test/Cheng*
42104854dc2036cc4d21578080cf9bbc /mnt/store/test/Chenga 1-06.wmv
b1ecafb95acc1ca59f4a6aefaed6a75e /mnt/store/test/Chenga 1-06low.wmv
02b87915a0dfbd8b15f9c85ed158d846 /mnt/store/test/Chenga 1-06med.wmv
|
I am now starting to wonder what the difference is with the second copy.
I can replicate the problem after I copy different larger files until that copy doesn't pass the md5 check, copy it again, and then it does pass the md5 check, then go and copy the file I want to have it happen again on.
The hash on the bad copy on the same file does not end up the same as previous bad copies.
I have not seen anything like this before, I'm going to try from a live cd and see what happens there. I'm open for ideas again, I can't move things around easily since the moves are not correctly done. |
|
| Back to top |
|
|
desultory
Bodhisattva
Joined: 04 Nov 2005
Posts: 9410
|
| Posted: Thu Sep 14, 2006 2:53 am Post subject: |
|
|
| Check the RAM in the system. |
|
| Back to top |
|
|
lukmih
n00b
Joined: 16 Nov 2005
Posts: 13
|
| Posted: Thu Sep 14, 2006 2:59 am Post subject: large file copy on sata drives while in live cd |
|
|
So, I basically can not reliably copy large files. The larger I go the more unreliable. This backup.tar (1.2GB) does not copy correctly ever. The Chenga file that didnt copy in the previous post was 200MB What should I do next?
This part tested with this live cd, I believe 2006.1:
Linux livecd 2.6.12-gentoo-r6 #1 SMP Tue Aug 2 16:32:16 UTC 2005 i686 AMD Athlon(tm) XP 2000+ AuthenticAMD GNU/Linux
| Code: |
livecd store # cp /mnt/downloads/music/backup.tar /mnt/store/music/
cp: overwrite `/mnt/store/music/backup.tar'? y
livecd store # md5sum /mnt/downloads/music/backup.tar
e23c39fe7326a07df0f0816d936cdb20 /mnt/downloads/music/backup.tar
livecd store # md5sum /mnt/store/music/backup.tar
4051067adf129b76ecc6108b42267e0c /mnt/store/music/backup.tar
livecd store # cp /mnt/downloads/music/backup.tar /mnt/store/music/
cp: overwrite `/mnt/store/music/backup.tar'? y
livecd store # md5sum /mnt/store/music/backup.tar
c23a05812e5c452c301dbe752321bcda /mnt/store/music/backup.tar
|
These drives were bought brand new within the past 4 months. Should this post be moved to a different category? |
|
| Back to top |
|
|
desultory
Bodhisattva
Joined: 04 Nov 2005
Posts: 9410
|
| Posted: Thu Sep 14, 2006 3:12 am Post subject: |
|
|
| lukmih wrote: | | These drives were bought brand new within the past 4 months. |
They should still be checked.
| lukmih wrote: | | Should this post be moved to a different category? |
It would appear so. I would suggest Kernel & Hardware along with changing the subject to something like "Large file corrupted when copied".
The first thing to do now though, is to check the RAM in that system. |
|
| Back to top |
|
|
tSp
Apprentice
Joined: 19 Jan 2004
Posts: 214
Location: Maysville, KY
|
| Posted: Thu Sep 14, 2006 3:21 am Post subject: |
|
|
I don't user reiserfs, but I have seen this happen on ext2/ext3 with bad blocks on the hard drive - and trust me, new hard drive or not...there are bad blocks on it. The thing about it is, if you delete the bad file and try to copy the same file back again, it will generally write the file starting at the same block as before and again be bad. Any drive write error messages in /var/log or dmesg?
Can't explain the ssh weirdness
Does this happen irregardless of which drive contains the source file being copied?? (i.e. you copy from drive a to b and it is almost always different, but if you copy the same file from drive b to drive a it is usually always right) This would maybe isolate a specific drive being faulty, but could also be memory related.
_________________
tSp
http://www.rhpstudios.com
http://www.xtremewebhosts.com
http://www.maysville-linux-users-group.org
http://www.edwiget.name
Registered Linux User #162711 |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA
|
| Posted: Thu Sep 14, 2006 4:54 am Post subject: Re: large file copy on sata drives while in live cd |
|
|
| lukmih wrote: |
| Code: |
livecd store # cp /mnt/downloads/music/backup.tar /mnt/store/music/
cp: overwrite `/mnt/store/music/backup.tar'? y
livecd store # md5sum /mnt/downloads/music/backup.tar
e23c39fe7326a07df0f0816d936cdb20 /mnt/downloads/music/backup.tar
livecd store # md5sum /mnt/store/music/backup.tar
4051067adf129b76ecc6108b42267e0c /mnt/store/music/backup.tar
livecd store # cp /mnt/downloads/music/backup.tar /mnt/store/music/
cp: overwrite `/mnt/store/music/backup.tar'? y
livecd store # md5sum /mnt/store/music/backup.tar
c23a05812e5c452c301dbe752321bcda /mnt/store/music/backup.tar
|
These drives were bought brand new within the past 4 months. Should this post be moved to a different category? |
Only two things can screw things this bad:
1. RAM problem.
2. Bad blocks on disk.
Check both thoroughly. Use memtest86++ and let it run overnight. If all is well with RAM, boot into livecd and run 'badblocks' to see if you have bad blocks on the disk. |
|
| Back to top |
|
|
|
Desktop Environments
