How to tell for sure that a USB stick is broken?

[nxe9]

I don't quite understand what the significance of this is since ntfs works fine on all systems.

I can say that both the "native support" and "FUSE" options are activated in my kernel according to https://wiki.gentoo.org/wiki/NTFS. Also the ntfs3g package is installed on both systems.

dmesg, nothing about ntfs:

[logrusx]

Do you use a custom or a distribution kernel? If it's custom, try with gentoo-kernel-bin. If it works, the problem is in your custom kernel configuration.

Best Regards,
Georgi

[nxe9]

I use the kernel from gentoo-sources.

I tried my old arch live usb, Arch Linux 5.18-16, kernel 5.18.16-arch1-1 and it also corrupts files on ext4 on the same pendrive. In my opinion, it's even worse on arch, because not a single file was correct, Maybe it's just a coincidence.

Later I can try the latest version of ubuntu live usb. I'll let you know the result later.

[dmpogo]

[pingtoo]

nxe9,

It seems strange that sometime you pendrive report using "ehci-pci" which is USB2 related then sometime it show as :SuperSpeed USB device using xhci_hcd" wich is USB3 related.

can you do a lsusb -t when your pendrive plug in to "blue" port? I wan to see under USB3 which usb mas storage driver used. I suspect it is the "uas" driver under USB3 causing problem.

the idVendor=346d, idProduct=5678 search on internet is all over place, some report USB2, some report USB3 so I am not sure how to identify this. but hopefully it will be clear that under fixed condition on your machine we will know how kernel identify it.

the "LPM exit latency is zeroed, disabling LPM" is kernel saying it CANNOT manage power for the pandrive

[NeddySeagoon]

I suspect the Windows/Linux difference is Windows knows that UAS is broken, so only ever users 'bulk mode' for USB3 but Linux does not.
It uses UAS by default but it fails.

Disable UAS for your vendor and Device ID to test.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[nxe9]

[pingtoo]

nxe9,

[nxe9]

@pingtoo:
Did you see my changes in the last post?

ext4+usb-storage+usb2 seems to be bad
ext4+usb-storage+usb3 seems to be bad (not tested on laptop yet but I believe this is the config on my pc).

UAS was not activated in the kernel. I activated UAS and this is the new output (Host 2 - laptop)

[pingtoo]

nxe9,

I just look you update, now it is USB3 mode.

I wonder.

I wonder, if the pendrive need to force on uas, I thought uas is default mode for USB3. may be our guess is upside down. may be it need uas and it is not getting it?

I will do some research.

[NeddySeagoon]

See https://wiki.gentoo.org/wiki/Raspberry_Pi_Install_Guide#Unreliable_USB_Attached_SCSI for disabling UAS.

The method is general, no just for Raspberry Pi.
You must use the Vendor and Device IDs of your USB stick, not the ones giver there.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[pingtoo]

nxe9,

I check hardware for linux, it show the 346d:5678 should be able use uas and usb-storage. and I check linux mainline, 346d:5678 is not known. (can find reference in source code)

so if your kernel have CONFIG_USB_UAS=m? may be you need to load the module before plug in?

[nxe9]

@NeddySeagoon:
The UAS option was not activated in the kernel at all. After activating the UAS driver, it is not loaded as seen in the log above. So I guess the problem isn't that UAS breaks something. Unless I don't understand something.

@pingtoo:
CONFIG_USB_UAS=y is my current config.

I tested data transfer again on my laptop, this time on xhci_hcd (usb 3), both ntfs and ext4. The result is the same. Ntfs works fine and ext4 still corrupts data. I noticed one more thing. ntfs runs extremely fast. In turn, on ext4, data is copied very quickly to a size of about 2 GB. After that, the transfer slows down extremely. Copying several GB takes many minutes. There is no such problem on ntfs.

Edit:
One more thing, when copying to ext4, this information appears in dmesg, as far as I remember, it didn't appear on USB 2.

[NeddySeagoon]

nxe9,

Lets do a sanity check on your times.
A USB 3 interface operates at at theoretical 5000 Gbits/sec.
Say 600Mbytes/sec. there is overhead too, so user data cannot get all the bandwidth.
All the data could pass over the interface in your reported NTFS time.
Good but that's the easy bit.

The FLASH memory cannot write it at anything like that speed.
We need to ask if your memory stick supports discard.
If not, you need to wait for very slow FLASH erase cycles.
FLASH cannot just be overwritten like a HDD.

What does fstrim -v /USB fs mount point tell?

The NTFS time looks too good to be true, so it probably is.
The ext4 time indicates that something is wrong.
Lots of USB bus resets in dmesg?

The reason discard matters, is that mkfs will do discard if its supported, so that the FLASH starts out erased, all ready for writes.
No slow erase cycles required until the fs has been filled once.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[nxe9]

@NeddySeagoon: However, I didn't say anything about the size of my source (myfiles/*). It was rather little data. There it was just about showing the proportion in the time difference.

[dmpogo]

What about any other filesystem on the pendrive ? VFAT ? xfs ? Do they work ?

[nxe9]

@dmpogo:
Good question. I'll test it later and let you know.

[nxe9]

Ntfs vs ext4, 24G of data

[Goverp]

I think totally irrelevant, but it might be fun (for certain values of "fun") to try this with F2FS.
_________________
Greybeard

[nxe9]

@Goverp: I will test it today. Another interesting thing, I will check whether ext4 generates the same errors every time or maybe checksums are random. Although I had the impression that it was random, but I need to verify it.

[nxe9]

I have some news. Ext4 corrupts data „randomly“. The second time I managed to observe that the entire ext4 file system was corrupted. When copying to f2fs the file system also crashed. I am providing some logs below. Ntfs still works flawlessly and very quickly every time.

I will summarize all my observations:
- Ext4 corrupts data every time, or sometimes the entire file system corrupts. This was tested on several Linux distributions and on two machines. Moreover, copying data to ext4 takes a very long time.
- Ntfs worked correctly and quickly every time on both Linux and Windows.
- With f2fs the entire file system broke (only one test).
- exFat worked correctly but very slowly (only one test).
- The problem with ext4 occurs also on my second pendrive on exactly the same model. However, on a model from the same series, but of a different size, the problem does not occur.
- These two series have different idVendor and idProduct.

Models that cause the problem:
Intenso Speed Line, idVendor=346d, idProduct=5678, 31.5 GB/29.3GiB

A model that works well:
Intenso Speed Line, idVendor=1f75, idProduct=0903, 126 GB/117GiB

What could be causing the problems? Perhaps some bug in the system's file implementation or a hardware bug that cannot be detected by other file systems. This is just speculation.

What further options do I have? I think that either contact the manufacturer or report the bug on the kernel mailing list in the hope that some developer will be willing to debug it. I don't know when I will do it, because I treat it as a curiosity and this problem is not a high priority for me.

Thank you for your help, and if anyone has any other ideas, I can try.

I will just add that the errors below are not the same as the situation when files contain incorrect data in the case of ext4. In such a situation, there are no logs in dmesg. However, here all the copying went wrong and was interrupted.

f2fs error while copying