Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
How to tell for sure that a USB stick is broken?
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
nxe9
Tux's lil' helper
Tux's lil' helper


Joined: 05 Jun 2021
Posts: 133

PostPosted: Sun Aug 25, 2024 10:48 pm    Post subject: How to tell for sure that a USB stick is broken? Reply with quote

Hi, I have a 32 GB USB stick and after copying some files to the stick sometimes some files are corrupted. I compared the crc32 checksums of the files on the host with the files on the stick and the data were not consistent. The usb stick contains only one ext4 partition. In some cases the files on the stick are good but after copying another file some of the previous files become corrupt. I can reproduce the problem very often. At the latest after I move a few larger files.

My attempts so far

- Tests on two difference gentoo hosts. On both machines the usb stick corrupts files.
- Check the host with memtest. OK.
- Another stick. OK, all checksums all fine.
- Test with badblock multiple times on two hosts, „badblocks -w -s -o error.log /dev/sdX“. OK, no error.
- The problem is not because I unmounted the device incorrectly or removed the pendrive too quickly.

It seems that my usb stick is corrupt but how to make a sufficiently informative test? The badblock test doesn’t produce any error….
Back to top
View user's profile Send private message
dmpogo
Advocate
Advocate


Joined: 02 Sep 2004
Posts: 3446
Location: Canada

PostPosted: Mon Aug 26, 2024 12:10 am    Post subject: Reply with quote

Check the logs /var/log/syslog or whatever logging system you are using. If there are write errors and logging is set to capture error level message, something maybe there

And

Reformat it, try again copying,
reformat with vfat, copy again

But it is probably dying
Back to top
View user's profile Send private message
jpsollie
Guru
Guru


Joined: 17 Aug 2013
Posts: 323

PostPosted: Mon Aug 26, 2024 5:56 am    Post subject: Reply with quote

it may also be an issue with USB link power management ... though I do not see why that would make badblocks succeed
_________________
The power of Gentoo optimization (not overclocked): [img]https://www.passmark.com/baselines/V10/images/503714802842.png[/img]
Back to top
View user's profile Send private message
Banana
Moderator
Moderator


Joined: 21 May 2004
Posts: 1783
Location: Germany

PostPosted: Mon Aug 26, 2024 8:00 am    Post subject: Reply with quote

There is https://www.cyberciti.biz/faq/linux-check-the-physical-health-of-a-usb-stick-flash-drive/ which lists some possibilities. No guarantee how good they are, but some sound reasonable
_________________
Forum Guidelines

PFL - Portage file list - find which package a file or command belongs to.
My delta-labs.org snippets do expire
Back to top
View user's profile Send private message
nxe9
Tux's lil' helper
Tux's lil' helper


Joined: 05 Jun 2021
Posts: 133

PostPosted: Mon Aug 26, 2024 11:44 am    Post subject: Reply with quote

Thank you for your suggestions!

@dmpogo
I reformatted it many times, however, it didn't help.

/var/log/syslog, sdb is my usb stick:

Code:
...
kernel: sd 6:0:0:0: [sdb] Attached SCSI removable disk
kernel: EXT4-fs (sdb1): warning: mounting fs with errors, running e2fsck is recommended
localhost kernel: EXT4-fs (sdb1): recovery complete
kernel: EXT4-fs (sdb1): mounted filesystem 4a... r/w with ordered data mode. Quota mode: none.
kernel: EXT4-fs error (device sdb1): ext4_validate_block_bitmap:421: comm ext4lazyinit: bg 160: bad block bitmap checksum
kernel: kworker/dying (62) used greatest stack depth: 11752 bytes left
...

@jpsollie
Not sure, but I tried different usb slots and two hosts (laptop and pc). In all cases there is a problem all the time only with this one specific stick.

@Banana
Two attemps to write with f3

1.
Code:
#f3write /mnt/usb/
...
Creating file 13.h2w ... OK!                           
Creating file 14.h2w ... Write failure: Structure needs cleaning
Creating file 15.h2w ... 81.17% -- 22.64 MB/s -- 7:55f3write: libflow.h:89: get_rem_chunk_size: Assertion `fw->blocks_per_delay > fw->processed_blocks' failed.
Aborted

2.
Code:
#f3write /mnt/usb/
...
Free space: 28.57 GB
Creating file 1.h2w ... Write failure: Structure needs cleaning
Creating file 2.h2w ... 0.94% -- 23.84 MB/s -- 57:24f3write: libflow.h:89: get_rem_chunk_size: Assertion `fw->blocks_per_delay > fw->processed_blocks' failed.
Aborted


Everything indicates that the pendrive is broken. Thank you for your help. However, what is a bit strange and dangerous is that the system knows that the pendrive is broken and yet allows you to mount and write data to it. Also strange that badblocks with default parameters does not show any error. The conclusion for me is that you cannot rely solely on badblocks.
Back to top
View user's profile Send private message
dmpogo
Advocate
Advocate


Joined: 02 Sep 2004
Posts: 3446
Location: Canada

PostPosted: Mon Aug 26, 2024 10:58 pm    Post subject: Reply with quote

I am unsure about badblocks program, but overall filesystem tries to use the disk if it is responsive. In the age of rotating disks the failure sometimes was taking time, bad blocks will develop, filesystem will remap them so that they are not used, and disk could still be used for quite a while.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54615
Location: 56N 3W

PostPosted: Tue Aug 27, 2024 2:44 am    Post subject: Reply with quote

Turn off UAS if its in use.

The kernel knows about a lot of devices with broken UAS suppor and will refuse to use it.
Equally, thre are lots of devices the kernel does not know about with broken UAS support.

Test on USB 2 if you can. That does not support UAS by design.
Should that not be possible, you need to de it the hard way with a kernel parameter.

Testing needs to start with partitioning, since nothing can be trusted on the drive.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
nxe9
Tux's lil' helper
Tux's lil' helper


Joined: 05 Jun 2021
Posts: 133

PostPosted: Tue Aug 27, 2024 9:39 am    Post subject: Reply with quote

It is a bit strange for me. I ran
Code:
fsck -f /dev/sdb1

he asked me about fixing something there and I approved. Then I recreated the partition with fdisk.

After that f3 doesn’t generate any error. Dmesg / syslog shows no erros. There is sometimes only this line
Code:
usb 2-1.2: reset high-speed USB device number 3 using ehci-pci


badblocks ist still ok. But the stick still corrupts my files. It seems to be still broken. However, I don't understand why I can't prove it with any standardized test any more.

Edit:
Actually I inserted the pendrive into USB 2.0 (yellow instead of blue), recreated the partition with fdisk and it is running
Code:
f3write /mnt/usb/ && f3read /mnt/usb/ && rm /mnt/usb/*.h2w && cp my_folder/* /mnt/usb/ && crc32 /mnt/usb/* && umount /mnt/usb/

It will take a few hours.

However, I think that at the very beginning I tried with USB 2.0 and still errors appeared. I don't understand why it didn't generate errors for me now. However, I'll report back later and let you know what the result is with USB 2.0
Back to top
View user's profile Send private message
logrusx
Advocate
Advocate


Joined: 22 Feb 2018
Posts: 2495

PostPosted: Tue Aug 27, 2024 11:22 am    Post subject: Reply with quote

I have an Adata USB 3 stick which behaved very bad on my old desktop computer as well as when plugged into my smart TV. However with my newer laptop I haven't noticed any problems for 3 years now. I don't use it that often, but it hasn't failed a single time.

On my old computer and I'm not sure but I think my old ThinkPad laptop too, I was forced to use an older USB 2 flash drive to get reliable data transfer.

What are the hosts you tried it on?

Also there are those fake drives which report bigger size than the flash chip really is. I can easily imagine how the controller ignores a more significant bit and overrides addresses that have already been written.

Best Regards,
Georgi
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54615
Location: 56N 3W

PostPosted: Tue Aug 27, 2024 11:47 am    Post subject: Reply with quote

ehci-pci is the PCI end of USB2.
One reset is harmless, when a device is added.
A stream is a problem.

badblocks has been useless for many years.
Even USB flash drives do wear levelling and sector sparing.
Badblocks could trigger either or both mechanisms.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
nxe9
Tux's lil' helper
Tux's lil' helper


Joined: 05 Jun 2021
Posts: 133

PostPosted: Tue Aug 27, 2024 6:52 pm    Post subject: Reply with quote

logrusx wrote:
What are the hosts you tried it on?

Also there are those fake drives which report bigger size than the flash chip really is.


Host 1: 10 years old PC, LGA 1150, Z97 chipset, 6.6.38-gentoo kernel

Host 2: 12 years old Thinkpad, 6.6.37-gentoo kernel.

The usb pendrive is Intenso 32 GB, no fake from China.

Same result with USB 2.0. No error during testing with f3, no error in dmesg/syslog but corrupt data with own files. Strange and incomprehensible to me. This means for me that I don't have any tools that would allow me to assess the health of the stick. The only possibility is to notice that the files get corrupted during use...
Back to top
View user's profile Send private message
logrusx
Advocate
Advocate


Joined: 22 Feb 2018
Posts: 2495

PostPosted: Tue Aug 27, 2024 7:30 pm    Post subject: Reply with quote

nxe9 wrote:
This means for me that I don't have any tools that would allow me to assess the health of the stick.


Except maybe warranty... I hope you have it.

If you don't, at least spare yourself the trouble and buy a new one.

Best Regards,
Georgi
Back to top
View user's profile Send private message
pingtoo
Veteran
Veteran


Joined: 10 Sep 2021
Posts: 1314
Location: Richmond Hill, Canada

PostPosted: Tue Aug 27, 2024 8:06 pm    Post subject: Reply with quote

nxe9 wrote:
logrusx wrote:
What are the hosts you tried it on?

Also there are those fake drives which report bigger size than the flash chip really is.


Host 1: 10 years old PC, LGA 1150, Z97 chipset, 6.6.38-gentoo kernel

Host 2: 12 years old Thinkpad, 6.6.37-gentoo kernel.

The usb pendrive is Intenso 32 GB, no fake from China.

Same result with USB 2.0. No error during testing with f3, no error in dmesg/syslog but corrupt data with own files. Strange and incomprehensible to me. This means for me that I don't have any tools that would allow me to assess the health of the stick. The only possibility is to notice that the files get corrupted during use...


If I ignore any of previous post in the thread, I would conclude that the pendrive is a USB 2 device. It may be just have fake USB3 connector.


Last edited by pingtoo on Tue Aug 27, 2024 8:39 pm; edited 1 time in total
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54615
Location: 56N 3W

PostPosted: Tue Aug 27, 2024 8:21 pm    Post subject: Reply with quote

when you say the data on the drive is corrup,
what size files have you been testing with?
how do you know that data is being read from the drive?

To be sure that the data is written, you need to issue a sync command.
it will not return while you have 'dirty' write caches.

Now you can drop caches to force the kernel to read the device, and not compare with the in cache copy of the file.

I think you have a USB 2 pendrive so UAS cannot be an issue.

Big files can take several minuets to sync over USB 2, even if the flash write speed is not the limiting factor
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
nxe9
Tux's lil' helper
Tux's lil' helper


Joined: 05 Jun 2021
Posts: 133

PostPosted: Tue Aug 27, 2024 8:46 pm    Post subject: Reply with quote

@logrusx:
This pendrive is not important to me. It is a $5 device. This is just a technical issue that interests me.

@pingtoo:
I don't know exactly. I can only provide the name of the stick and information you request. After inserting the stick into USB 3.0 port dmesg shows me among other things
Code:
new high-speed USB device number 4 using ehci-pci


ehci-pci sounds like USB 2.0 as was said before. Honestly, I don't know much about it.

The item Intenso Speed Line, 32GB, USB-Stick 3.2 Gen 1x1
https://www.amazon.de/Intenso-Speed-Line-Speicherstick-schwarz/dp/B00YMZBMM4?th=1&language=en_GB

@NeddySeagoon:
For example, I transferred three files of size 4.8 GB, 2 GB and 11 GB.

I don't know if I understand the rest of your post correctly. I can say that after copying, I tried, for example, unmounting and remounting the partition and to verify the checksum after that. In this case, sometimes I have to wait a few minutes for unmount and I always wait until the end. Unmounting sometimes takes a long time because then it probably synchronizes? Or maybe it doesn't make sense?

In short, what am I doing wrong?
Back to top
View user's profile Send private message
logrusx
Advocate
Advocate


Joined: 22 Feb 2018
Posts: 2495

PostPosted: Wed Aug 28, 2024 4:30 am    Post subject: Reply with quote

nxe9 wrote:
@logrusx:
This pendrive is not important to me. It is a $5 device. This is just a technical issue that interests me.


I think in this case you can't learn much by yourself with this type of products. Even without the tricks they played on the consumer, things are already too complicated. Better look at the 1 star reviews in the link you've shared.

BTW:

nxe9 wrote:

The usb pendrive is Intenso 32 GB, no fake from China.


I've never heard of that brand before. It sounds pretty much like a Chinese piece of garbage to me. And the price suggests it. Amazon is notorious for allowing such and worse kind of crap on their store:
https://www.youtube.com/watch?v=qZCMislL6_I
https://www.youtube.com/watch?v=DeUNC7z5MM0

Best Regards,
Georgi
Back to top
View user's profile Send private message
nxe9
Tux's lil' helper
Tux's lil' helper


Joined: 05 Jun 2021
Posts: 133

PostPosted: Wed Aug 28, 2024 4:02 pm    Post subject: Reply with quote

The brand is German.
https://de.wikipedia.org/wiki/Intenso

This is a company that has been around for many years, but produces quite cheap products. Well, that's not the point here. In conclusion, it's a pity that I don't have the ability to detect errors with any specific tool. And regarding this pendrive, it's true, I have no choice but to throw it into the trash. Because everything indicates that there is something wrong with this stick.

Regarding the f3 and badblocks program, I think the problem is that they work on the principle of standardized data write. If the packages created by these programs do not generate an error, it will not detect them. It would be useful to have a tool that generates some random data on the host and on the pendrive and compares them with each other. I think that writing such a program yourself would not be difficult.

One more thing, I wonder how this pendrive will behave on Windows. I think I can check it later and let you know.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54615
Location: 56N 3W

PostPosted: Wed Aug 28, 2024 6:28 pm    Post subject: Reply with quote

nex9,

umount gives you a free sync, which is why it takes so long.
The prompt does not return until the data is committed to the USB stick and its safeto remove it.

in short, there is nothing wrong with your process.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
nxe9
Tux's lil' helper
Tux's lil' helper


Joined: 05 Jun 2021
Posts: 133

PostPosted: Fri Aug 30, 2024 9:21 am    Post subject: Reply with quote

Further results:
- My second, new pendrive, same model, creates the same problems on my gentoo systems.
- Files do not get corrupted on Windows. However, I used ntfs there instead of ext4.

It seems as if the system had problems managing this pendrive correctly.

I will perform more tests soon to try to narrow down the problem:
- ntfs instead of ext4 on gentoo
- Another linux distro like ubuntu live cd.

I'll let you know later.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54615
Location: 56N 3W

PostPosted: Fri Aug 30, 2024 12:13 pm    Post subject: Reply with quote

nxe9,

ntfs and ext4 are both journalled filesystems.
The journal requires extra writes.

With ext4, the journal can be turned off to save the these writes.
That may be worth trying.

add -O ^has_journal to the command when you make the filesystem or play with tune2fs
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
pingtoo
Veteran
Veteran


Joined: 10 Sep 2021
Posts: 1314
Location: Richmond Hill, Canada

PostPosted: Fri Aug 30, 2024 1:27 pm    Post subject: Reply with quote

nxe9 wrote:
Further results:
- My second, new pendrive, same model, creates the same problems on my gentoo systems.
- Files do not get corrupted on Windows. However, I used ntfs there instead of ext4.

It seems as if the system had problems managing this pendrive correctly.

I will perform more tests soon to try to narrow down the problem:
- ntfs instead of ext4 on gentoo
- Another linux distro like ubuntu live cd.

I'll let you know later.
In you posted context, window and gentoo is same computer hardware? if not than what kind of file system may not relevant.

And how is Windows identify the pandrive? as USB3 device or USB2 device?
Back to top
View user's profile Send private message
nxe9
Tux's lil' helper
Tux's lil' helper


Joined: 05 Jun 2021
Posts: 133

PostPosted: Fri Aug 30, 2024 8:55 pm    Post subject: Reply with quote

Thank you for your help so far.

@NeddySeagoon
When invoking
Code:
mkfs.ext4 /dev/sd...

the output shows "Creating journal: ... [done]".

Code:
# debugfs -R features /dev/sd...
contains "has_journal".

@pingtoo
Yes, the same hardware.

I haven't checked what type of USB on Windows yet, but in the context of further information, it's probably not important?

I have some news. I created ntfs on gentoo and all the copied data was correct.I did some counter-tests and switched between ext4 and ntfs on gentoo several times. In short, the problem occurs constantly on ext4, but on ntfs it has never occurred.

So there is some problem between ext4 and my pendrive or perhaps some problem related to the operating system? What's going on here? How to explain this? What else can I do?
Back to top
View user's profile Send private message
pingtoo
Veteran
Veteran


Joined: 10 Sep 2021
Posts: 1314
Location: Richmond Hill, Canada

PostPosted: Fri Aug 30, 2024 9:16 pm    Post subject: Reply with quote

nxe9 wrote:
I have some news. I created ntfs on gentoo and all the copied data was correct.I did some counter-tests and switched between ext4 and ntfs on gentoo several times. In short, the problem occurs constantly on ext4, but on ntfs it has never occurred.

I wonder if this mean the pandrive have special block size. May be ext4 default block size different from ntfs and the pandrive is picky about this.

If you windows can identify the pandrive as USB3 device, that could mean on Linux there something need to done in order to make it work proberly, this could also affect the ext4 vs ntfs.

Also I think in Linux the ntfs is user space. so that could also make some different.
Back to top
View user's profile Send private message
nxe9
Tux's lil' helper
Tux's lil' helper


Joined: 05 Jun 2021
Posts: 133

PostPosted: Fri Aug 30, 2024 10:10 pm    Post subject: Reply with quote

On windows I tested the data transfer only on the blue (usb 3.0) port.

When the pendrive is inserted into the blue or black slot the result in the device manager is the same.

https://i.postimg.cc/28mWg1Cr/devman.png

However, a change is visible in the Windows settings.

Blue: https://i.postimg.cc/tRFPr3QB/winset.png

Black: https://i.postimg.cc/dVVytg7f/winset2.png

So I conclude that Windows operates on USB 3.0.
Back to top
View user's profile Send private message
dmpogo
Advocate
Advocate


Joined: 02 Sep 2004
Posts: 3446
Location: Canada

PostPosted: Sat Aug 31, 2024 5:39 am    Post subject: Reply with quote

pingtoo wrote:


Also I think in Linux the ntfs is user space. so that could also make some different.



There is new ntfs support in kernel, at least since 5.x.x kernels (the old one was limited). So it is a question, what is used, kernel ntfs or user-space ntfs-3g.
There should be something in the log regarding ntfs when drive is intersted
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum