View previous topic :: View next topic |
Author |
Message |
nxe9 Tux's lil' helper
Joined: 05 Jun 2021 Posts: 107
|
Posted: Sun Aug 25, 2024 10:48 pm Post subject: How to tell for sure that a USB stick is broken? |
|
|
Hi, I have a 32 GB USB stick and after copying some files to the stick sometimes some files are corrupted. I compared the crc32 checksums of the files on the host with the files on the stick and the data were not consistent. The usb stick contains only one ext4 partition. In some cases the files on the stick are good but after copying another file some of the previous files become corrupt. I can reproduce the problem very often. At the latest after I move a few larger files.
My attempts so far
- Tests on two difference gentoo hosts. On both machines the usb stick corrupts files.
- Check the host with memtest. OK.
- Another stick. OK, all checksums all fine.
- Test with badblock multiple times on two hosts, „badblocks -w -s -o error.log /dev/sdX“. OK, no error.
- The problem is not because I unmounted the device incorrectly or removed the pendrive too quickly.
It seems that my usb stick is corrupt but how to make a sufficiently informative test? The badblock test doesn’t produce any error…. |
|
Back to top |
|
|
dmpogo Advocate
Joined: 02 Sep 2004 Posts: 3414 Location: Canada
|
Posted: Mon Aug 26, 2024 12:10 am Post subject: |
|
|
Check the logs /var/log/syslog or whatever logging system you are using. If there are write errors and logging is set to capture error level message, something maybe there
And
Reformat it, try again copying,
reformat with vfat, copy again
But it is probably dying |
|
Back to top |
|
|
jpsollie Guru
Joined: 17 Aug 2013 Posts: 322
|
|
Back to top |
|
|
Banana Moderator
Joined: 21 May 2004 Posts: 1716 Location: Germany
|
|
Back to top |
|
|
nxe9 Tux's lil' helper
Joined: 05 Jun 2021 Posts: 107
|
Posted: Mon Aug 26, 2024 11:44 am Post subject: |
|
|
Thank you for your suggestions!
@dmpogo
I reformatted it many times, however, it didn't help.
/var/log/syslog, sdb is my usb stick:
Code: | ...
kernel: sd 6:0:0:0: [sdb] Attached SCSI removable disk
kernel: EXT4-fs (sdb1): warning: mounting fs with errors, running e2fsck is recommended
localhost kernel: EXT4-fs (sdb1): recovery complete
kernel: EXT4-fs (sdb1): mounted filesystem 4a... r/w with ordered data mode. Quota mode: none.
kernel: EXT4-fs error (device sdb1): ext4_validate_block_bitmap:421: comm ext4lazyinit: bg 160: bad block bitmap checksum
kernel: kworker/dying (62) used greatest stack depth: 11752 bytes left
... |
@jpsollie
Not sure, but I tried different usb slots and two hosts (laptop and pc). In all cases there is a problem all the time only with this one specific stick.
@Banana
Two attemps to write with f3
1.
Code: | #f3write /mnt/usb/
...
Creating file 13.h2w ... OK!
Creating file 14.h2w ... Write failure: Structure needs cleaning
Creating file 15.h2w ... 81.17% -- 22.64 MB/s -- 7:55f3write: libflow.h:89: get_rem_chunk_size: Assertion `fw->blocks_per_delay > fw->processed_blocks' failed.
Aborted |
2.
Code: | #f3write /mnt/usb/
...
Free space: 28.57 GB
Creating file 1.h2w ... Write failure: Structure needs cleaning
Creating file 2.h2w ... 0.94% -- 23.84 MB/s -- 57:24f3write: libflow.h:89: get_rem_chunk_size: Assertion `fw->blocks_per_delay > fw->processed_blocks' failed.
Aborted |
Everything indicates that the pendrive is broken. Thank you for your help. However, what is a bit strange and dangerous is that the system knows that the pendrive is broken and yet allows you to mount and write data to it. Also strange that badblocks with default parameters does not show any error. The conclusion for me is that you cannot rely solely on badblocks. |
|
Back to top |
|
|
dmpogo Advocate
Joined: 02 Sep 2004 Posts: 3414 Location: Canada
|
Posted: Mon Aug 26, 2024 10:58 pm Post subject: |
|
|
I am unsure about badblocks program, but overall filesystem tries to use the disk if it is responsive. In the age of rotating disks the failure sometimes was taking time, bad blocks will develop, filesystem will remap them so that they are not used, and disk could still be used for quite a while. |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54577 Location: 56N 3W
|
Posted: Tue Aug 27, 2024 2:44 am Post subject: |
|
|
Turn off UAS if its in use.
The kernel knows about a lot of devices with broken UAS suppor and will refuse to use it.
Equally, thre are lots of devices the kernel does not know about with broken UAS support.
Test on USB 2 if you can. That does not support UAS by design.
Should that not be possible, you need to de it the hard way with a kernel parameter.
Testing needs to start with partitioning, since nothing can be trusted on the drive. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
nxe9 Tux's lil' helper
Joined: 05 Jun 2021 Posts: 107
|
Posted: Tue Aug 27, 2024 9:39 am Post subject: |
|
|
It is a bit strange for me. I ran
he asked me about fixing something there and I approved. Then I recreated the partition with fdisk.
After that f3 doesn’t generate any error. Dmesg / syslog shows no erros. There is sometimes only this line
Code: | usb 2-1.2: reset high-speed USB device number 3 using ehci-pci |
badblocks ist still ok. But the stick still corrupts my files. It seems to be still broken. However, I don't understand why I can't prove it with any standardized test any more.
Edit:
Actually I inserted the pendrive into USB 2.0 (yellow instead of blue), recreated the partition with fdisk and it is running
Code: | f3write /mnt/usb/ && f3read /mnt/usb/ && rm /mnt/usb/*.h2w && cp my_folder/* /mnt/usb/ && crc32 /mnt/usb/* && umount /mnt/usb/ |
It will take a few hours.
However, I think that at the very beginning I tried with USB 2.0 and still errors appeared. I don't understand why it didn't generate errors for me now. However, I'll report back later and let you know what the result is with USB 2.0 |
|
Back to top |
|
|
logrusx Advocate
Joined: 22 Feb 2018 Posts: 2397
|
Posted: Tue Aug 27, 2024 11:22 am Post subject: |
|
|
I have an Adata USB 3 stick which behaved very bad on my old desktop computer as well as when plugged into my smart TV. However with my newer laptop I haven't noticed any problems for 3 years now. I don't use it that often, but it hasn't failed a single time.
On my old computer and I'm not sure but I think my old ThinkPad laptop too, I was forced to use an older USB 2 flash drive to get reliable data transfer.
What are the hosts you tried it on?
Also there are those fake drives which report bigger size than the flash chip really is. I can easily imagine how the controller ignores a more significant bit and overrides addresses that have already been written.
Best Regards,
Georgi |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54577 Location: 56N 3W
|
Posted: Tue Aug 27, 2024 11:47 am Post subject: |
|
|
ehci-pci is the PCI end of USB2.
One reset is harmless, when a device is added.
A stream is a problem.
badblocks has been useless for many years.
Even USB flash drives do wear levelling and sector sparing.
Badblocks could trigger either or both mechanisms. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
nxe9 Tux's lil' helper
Joined: 05 Jun 2021 Posts: 107
|
Posted: Tue Aug 27, 2024 6:52 pm Post subject: |
|
|
logrusx wrote: | What are the hosts you tried it on?
Also there are those fake drives which report bigger size than the flash chip really is. |
Host 1: 10 years old PC, LGA 1150, Z97 chipset, 6.6.38-gentoo kernel
Host 2: 12 years old Thinkpad, 6.6.37-gentoo kernel.
The usb pendrive is Intenso 32 GB, no fake from China.
Same result with USB 2.0. No error during testing with f3, no error in dmesg/syslog but corrupt data with own files. Strange and incomprehensible to me. This means for me that I don't have any tools that would allow me to assess the health of the stick. The only possibility is to notice that the files get corrupted during use... |
|
Back to top |
|
|
logrusx Advocate
Joined: 22 Feb 2018 Posts: 2397
|
Posted: Tue Aug 27, 2024 7:30 pm Post subject: |
|
|
nxe9 wrote: | This means for me that I don't have any tools that would allow me to assess the health of the stick. |
Except maybe warranty... I hope you have it.
If you don't, at least spare yourself the trouble and buy a new one.
Best Regards,
Georgi |
|
Back to top |
|
|
pingtoo Veteran
Joined: 10 Sep 2021 Posts: 1232 Location: Richmond Hill, Canada
|
Posted: Tue Aug 27, 2024 8:06 pm Post subject: |
|
|
nxe9 wrote: | logrusx wrote: | What are the hosts you tried it on?
Also there are those fake drives which report bigger size than the flash chip really is. |
Host 1: 10 years old PC, LGA 1150, Z97 chipset, 6.6.38-gentoo kernel
Host 2: 12 years old Thinkpad, 6.6.37-gentoo kernel.
The usb pendrive is Intenso 32 GB, no fake from China.
Same result with USB 2.0. No error during testing with f3, no error in dmesg/syslog but corrupt data with own files. Strange and incomprehensible to me. This means for me that I don't have any tools that would allow me to assess the health of the stick. The only possibility is to notice that the files get corrupted during use... |
If I ignore any of previous post in the thread, I would conclude that the pendrive is a USB 2 device. It may be just have fake USB3 connector.
Last edited by pingtoo on Tue Aug 27, 2024 8:39 pm; edited 1 time in total |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54577 Location: 56N 3W
|
Posted: Tue Aug 27, 2024 8:21 pm Post subject: |
|
|
when you say the data on the drive is corrup,
what size files have you been testing with?
how do you know that data is being read from the drive?
To be sure that the data is written, you need to issue a sync command.
it will not return while you have 'dirty' write caches.
Now you can drop caches to force the kernel to read the device, and not compare with the in cache copy of the file.
I think you have a USB 2 pendrive so UAS cannot be an issue.
Big files can take several minuets to sync over USB 2, even if the flash write speed is not the limiting factor _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
nxe9 Tux's lil' helper
Joined: 05 Jun 2021 Posts: 107
|
Posted: Tue Aug 27, 2024 8:46 pm Post subject: |
|
|
@logrusx:
This pendrive is not important to me. It is a $5 device. This is just a technical issue that interests me.
@pingtoo:
I don't know exactly. I can only provide the name of the stick and information you request. After inserting the stick into USB 3.0 port dmesg shows me among other things
Code: | new high-speed USB device number 4 using ehci-pci |
ehci-pci sounds like USB 2.0 as was said before. Honestly, I don't know much about it.
The item Intenso Speed Line, 32GB, USB-Stick 3.2 Gen 1x1
https://www.amazon.de/Intenso-Speed-Line-Speicherstick-schwarz/dp/B00YMZBMM4?th=1&language=en_GB
@NeddySeagoon:
For example, I transferred three files of size 4.8 GB, 2 GB and 11 GB.
I don't know if I understand the rest of your post correctly. I can say that after copying, I tried, for example, unmounting and remounting the partition and to verify the checksum after that. In this case, sometimes I have to wait a few minutes for unmount and I always wait until the end. Unmounting sometimes takes a long time because then it probably synchronizes? Or maybe it doesn't make sense?
In short, what am I doing wrong? |
|
Back to top |
|
|
logrusx Advocate
Joined: 22 Feb 2018 Posts: 2397
|
Posted: Wed Aug 28, 2024 4:30 am Post subject: |
|
|
nxe9 wrote: | @logrusx:
This pendrive is not important to me. It is a $5 device. This is just a technical issue that interests me. |
I think in this case you can't learn much by yourself with this type of products. Even without the tricks they played on the consumer, things are already too complicated. Better look at the 1 star reviews in the link you've shared.
BTW:
nxe9 wrote: |
The usb pendrive is Intenso 32 GB, no fake from China. |
I've never heard of that brand before. It sounds pretty much like a Chinese piece of garbage to me. And the price suggests it. Amazon is notorious for allowing such and worse kind of crap on their store:
https://www.youtube.com/watch?v=qZCMislL6_I
https://www.youtube.com/watch?v=DeUNC7z5MM0
Best Regards,
Georgi |
|
Back to top |
|
|
nxe9 Tux's lil' helper
Joined: 05 Jun 2021 Posts: 107
|
Posted: Wed Aug 28, 2024 4:02 pm Post subject: |
|
|
The brand is German.
https://de.wikipedia.org/wiki/Intenso
This is a company that has been around for many years, but produces quite cheap products. Well, that's not the point here. In conclusion, it's a pity that I don't have the ability to detect errors with any specific tool. And regarding this pendrive, it's true, I have no choice but to throw it into the trash. Because everything indicates that there is something wrong with this stick.
Regarding the f3 and badblocks program, I think the problem is that they work on the principle of standardized data write. If the packages created by these programs do not generate an error, it will not detect them. It would be useful to have a tool that generates some random data on the host and on the pendrive and compares them with each other. I think that writing such a program yourself would not be difficult.
One more thing, I wonder how this pendrive will behave on Windows. I think I can check it later and let you know. |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54577 Location: 56N 3W
|
Posted: Wed Aug 28, 2024 6:28 pm Post subject: |
|
|
nex9,
umount gives you a free sync, which is why it takes so long.
The prompt does not return until the data is committed to the USB stick and its safeto remove it.
in short, there is nothing wrong with your process. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
nxe9 Tux's lil' helper
Joined: 05 Jun 2021 Posts: 107
|
Posted: Fri Aug 30, 2024 9:21 am Post subject: |
|
|
Further results:
- My second, new pendrive, same model, creates the same problems on my gentoo systems.
- Files do not get corrupted on Windows. However, I used ntfs there instead of ext4.
It seems as if the system had problems managing this pendrive correctly.
I will perform more tests soon to try to narrow down the problem:
- ntfs instead of ext4 on gentoo
- Another linux distro like ubuntu live cd.
I'll let you know later. |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54577 Location: 56N 3W
|
Posted: Fri Aug 30, 2024 12:13 pm Post subject: |
|
|
nxe9,
ntfs and ext4 are both journalled filesystems.
The journal requires extra writes.
With ext4, the journal can be turned off to save the these writes.
That may be worth trying.
add -O ^has_journal to the command when you make the filesystem or play with tune2fs _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
pingtoo Veteran
Joined: 10 Sep 2021 Posts: 1232 Location: Richmond Hill, Canada
|
Posted: Fri Aug 30, 2024 1:27 pm Post subject: |
|
|
nxe9 wrote: | Further results:
- My second, new pendrive, same model, creates the same problems on my gentoo systems.
- Files do not get corrupted on Windows. However, I used ntfs there instead of ext4.
It seems as if the system had problems managing this pendrive correctly.
I will perform more tests soon to try to narrow down the problem:
- ntfs instead of ext4 on gentoo
- Another linux distro like ubuntu live cd.
I'll let you know later. | In you posted context, window and gentoo is same computer hardware? if not than what kind of file system may not relevant.
And how is Windows identify the pandrive? as USB3 device or USB2 device? |
|
Back to top |
|
|
nxe9 Tux's lil' helper
Joined: 05 Jun 2021 Posts: 107
|
Posted: Fri Aug 30, 2024 8:55 pm Post subject: |
|
|
Thank you for your help so far.
@NeddySeagoon
When invoking
Code: | mkfs.ext4 /dev/sd... |
the output shows "Creating journal: ... [done]".
Code: | # debugfs -R features /dev/sd... | contains "has_journal".
@pingtoo
Yes, the same hardware.
I haven't checked what type of USB on Windows yet, but in the context of further information, it's probably not important?
I have some news. I created ntfs on gentoo and all the copied data was correct.I did some counter-tests and switched between ext4 and ntfs on gentoo several times. In short, the problem occurs constantly on ext4, but on ntfs it has never occurred.
So there is some problem between ext4 and my pendrive or perhaps some problem related to the operating system? What's going on here? How to explain this? What else can I do? |
|
Back to top |
|
|
pingtoo Veteran
Joined: 10 Sep 2021 Posts: 1232 Location: Richmond Hill, Canada
|
Posted: Fri Aug 30, 2024 9:16 pm Post subject: |
|
|
nxe9 wrote: | I have some news. I created ntfs on gentoo and all the copied data was correct.I did some counter-tests and switched between ext4 and ntfs on gentoo several times. In short, the problem occurs constantly on ext4, but on ntfs it has never occurred. |
I wonder if this mean the pandrive have special block size. May be ext4 default block size different from ntfs and the pandrive is picky about this.
If you windows can identify the pandrive as USB3 device, that could mean on Linux there something need to done in order to make it work proberly, this could also affect the ext4 vs ntfs.
Also I think in Linux the ntfs is user space. so that could also make some different. |
|
Back to top |
|
|
nxe9 Tux's lil' helper
Joined: 05 Jun 2021 Posts: 107
|
|
Back to top |
|
|
dmpogo Advocate
Joined: 02 Sep 2004 Posts: 3414 Location: Canada
|
Posted: Sat Aug 31, 2024 5:39 am Post subject: |
|
|
pingtoo wrote: |
Also I think in Linux the ntfs is user space. so that could also make some different. |
There is new ntfs support in kernel, at least since 5.x.x kernels (the old one was limited). So it is a question, what is used, kernel ntfs or user-space ntfs-3g.
There should be something in the log regarding ntfs when drive is intersted |
|
Back to top |
|
|
|