| View previous topic :: View next topic |
| Author |
Message |
tkzv
Tux's lil' helper
Joined: 22 Aug 2014
Posts: 88
|
 Posted: Sun Mar 17, 2024 9:53 am Post subject: Files disappear |
 |
|
It happened 2 or 3 times. I download a file to the default download directory, then several weeks later cannot find it. I am absolutely sure it was there, but "find ~" does not find it anywhere. 2 of those files were *.webm audio tracks downloaded with yt-dlp, and I listened to them several times. The third, about which I'm not sure, was a track, which I converted to *.ogg container and tried to cut down, removing unwanted parts with ffmpeg — I did not like the result and may have not kept the OGG.
Earlier on the same system KWallet file with list of passwords suddenly disappeared 2 times.
e2fsck and smartctl did not show any errors. There's nothing in /lost+found. The second WebM was saved after I got an UPS, and there were no power failures.
My questions are:
1. What may cause such disappearances? Other than accidental deletion.
2. Is there a way to create some kind of "Trash bin" that would track down deletions from ~ by any program, not just GUI file managers? Something that automatically creates hardlinks, maybe, or catches system calls to unlink() and such. |
|
| Back to top |
|
 |
flexibeast
Guru
Joined: 04 Apr 2022
Posts: 425
Location: Naarm/Melbourne, Australia
|
| Posted: Sun Mar 17, 2024 11:22 am Post subject: |
|
|
i presume your downloads directory, and indeed your home directory aren't on a volatile filesystem (i.e. one not preserved by reboots), e.g. a tmpfs system, so i'd be interested to hear any suggestions people might have as to what might be going on here ....
As for a general, non-DE/WM/GUI-specific way of tracking deletions from any program, i'm not immediately aware of pre-existing software that does that (although would be interested to know of any). But one could use the Linux inotify API to build something that basically does what you want: refer to the inotify(7) man page for a general intro, and the man page sfor inotifywatch(1) and inotifywait(1), as binaries you could wrap. There's also fanotify(7), fsnotifywatch(1), and fsnotifywait(1). All of the preceding binaries are available via the sys-fs/inotify-tools package. |
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3404
|
| Posted: Sun Mar 17, 2024 12:41 pm Post subject: |
|
|
| Quote: | | 1. What may cause such disappearances? Other than accidental deletion. |
Did you get ransomed or something?
| Quote: | | 2. Is there a way to create some kind of "Trash bin" that would track down deletions from ~ by any program, not just GUI file managers |
"Trash bin" is windows man's backup.
Just do it the usual way instead.
Inotify can track changes to the filesystem, but it's kinda hard to catch deleted files. Like in: you be informed that a file vanished, but at this point it's already unlinked, its metadata is gone, and physical space it resides in marked as free, so any restore you might attempt is in a race against your system and your luck.
You can use it as an extra tool for collecting information, but it doesn't solve the problem.
_________________
Make Computing Fun Again |
|
| Back to top |
|
|
tkzv
Tux's lil' helper
Joined: 22 Aug 2014
Posts: 88
|
| Posted: Sun Mar 17, 2024 2:04 pm Post subject: |
|
|
| flexibeast wrote: | i presume your downloads directory, and indeed your home directory aren't on a volatile filesystem (i.e. one not preserved by reboots), e.g. a tmpfs system, so i'd be interested to hear any suggestions people might have as to what might be going on here ....
As for a general, non-DE/WM/GUI-specific way of tracking deletions from any program, i'm not immediately aware of pre-existing software that does that (although would be interested to know of any). But one could use the Linux inotify API to build something that basically does what you want: refer to the inotify(7) man page for a general intro, and the man page sfor inotifywatch(1) and inotifywait(1), as binaries you could wrap. There's also fanotify(7), fsnotifywatch(1), and fsnotifywait(1). All of the preceding binaries are available via the sys-fs/inotify-tools package. |
Thanks. Does inotify detect the process that did the deletion?
--------------------------------------------------------------------------------------
| szatox wrote: | | Did you get ransomed or something? |
No. I do suspect misbehavior of a certain application, though.
| szatox wrote: | "Trash bin" is windows man's backup.
Just do it the usual way instead. |
What is "the usual way"?
| szatox wrote: | Inotify can track changes to the filesystem, but it's kinda hard to catch deleted files. Like in: you be informed that a file vanished, but at this point it's already unlinked, its metadata is gone, and physical space it resides in marked as free, so any restore you might attempt is in a race against your system and your luck.
You can use it as an extra tool for collecting information, but it doesn't solve the problem. |
I want to catch the culprit. |
|
| Back to top |
|
|
mrbassie
l33t
Joined: 31 May 2013
Posts: 821
Location: over here
|
| Posted: Sun Mar 17, 2024 9:47 pm Post subject: |
|
|
| tkzv wrote: |
No. I do suspect misbehavior of a certain application, though.
|
Who's the suspect?
_________________
I spent a christmas in Vienna twenty something years ago. It was a beautiful city. Everyone was so friendly. |
|
| Back to top |
|
|
sublogic
Apprentice
Joined: 21 Mar 2022
Posts: 269
Location: Pennsylvania, USA
|
| Posted: Sun Mar 17, 2024 10:05 pm Post subject: |
|
|
| tkzv wrote: | | I want to catch the culprit. |
Something was truncating my /etc/ntp.conf . I restored it from backup and then did: | Code: | # cp /etc/ntp.conf /root/ntp.conf.save
# mount --bind -o ro /root/ntp.conf.save /etc/ntp.conf |
Now /etc/ntp.conf is read-only. Eventually dhcpcd left a message in syslog complaining about a read-only filesystem. Busted ! Knowing that, I solved the problem by tweaking /etc/rc.net .
Maybe you can adapt my recipe to your case ? Your culprit may crash instead of logging an error, but at least you'll know. |
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3404
|
| Posted: Mon Mar 18, 2024 12:29 am Post subject: |
|
|
| Quote: | No. I do suspect misbehavior of a certain application, though. [..]
I want to catch the culprit. |
If you do have a suspect, I think you can catch it red-handed by running it through strace (or even attaching strace to an already existing process with strace -p <target pid>). I recall grepping the output for calls to OPEN when I was looking for files accessed by an application. I don't know what function is called to remove files, but I'm pretty sure it can be identified.
| Quote: | | What is "the usual way"? |
There are many ways, but they tend to involve a scheduled, daily (or nightly) dump of all your files to a more-or-less protected medium.
A jukebox stuffed with tapes is a decent target device, but can just as well use a spare disk you probably already have or at lest know how to use.
Preferably in a different machine (to protect it from stuff like voltage spikes from failing power supply).
In a remote location if possible (to protect from accidents like 400V coming out of 230V socket, floods, fires, theft and meteor strikes).
Depending on your scale of operation, you might want to invest in a backup server (like in a dedicated machine collecting all the data for you) or a simple cron job calling rsync (or borg, which apparently has a bunch of fans here)
_________________
Make Computing Fun Again |
|
| Back to top |
|
|
flexibeast
Guru
Joined: 04 Apr 2022
Posts: 425
Location: Naarm/Melbourne, Australia
|
| Posted: Mon Mar 18, 2024 4:50 am Post subject: |
|
|
| szatox wrote: | | I don't know what function is called to remove files, but I'm pretty sure it can be identified. |
You mentioned files getting 'unlinked' in an earlier comment, and indeed, it's unlink(2) / unlinkat(2).
| tkzv wrote: | | Does inotify detect the process that did the deletion? |
Not that i can tell, from a quick scan over the relevant man pages, but i might well be wrong. |
|
| Back to top |
|
|
tkzv
Tux's lil' helper
Joined: 22 Aug 2014
Posts: 88
|
| Posted: Mon Mar 18, 2024 5:44 am Post subject: |
|
|
| mrbassie wrote: | | tkzv wrote: |
No. I do suspect misbehavior of a certain application, though.
|
Who's the suspect? |
Between downloading those files and finding them missing, I tried to install ZScaler client, which overwrote DNS settings and messed net-misc/iputils binaries. But I just realized it cannot be responsible for KWallet files. That leaves a very paranoid version — I will not accuse that package author without proof.
----------------------------------------------------------------
| sublogic wrote: | | tkzv wrote: | | I want to catch the culprit. |
Something was truncating my /etc/ntp.conf . I restored it from backup and then did: | Code: | # cp /etc/ntp.conf /root/ntp.conf.save
# mount --bind -o ro /root/ntp.conf.save /etc/ntp.conf |
Now /etc/ntp.conf is read-only. Eventually dhcpcd left a message in syslog complaining about a read-only filesystem. Busted ! Knowing that, I solved the problem by tweaking /etc/rc.net .
Maybe you can adapt my recipe to your case ? Your culprit may crash instead of logging an error, but at least you'll know. |
I need kwallet to read and write ~/.local/share/kwalletd/* Other programs have no business there. Right now, kwallet is run by the same user as everything else. Can it be run by a separate user?
----------------------------------------------------------------
| szatox wrote: | | If you do have a suspect, I think you can catch it red-handed by running it through strace (or even attaching strace to an already existing process with strace -p <target pid>). I recall grepping the output for calls to OPEN when I was looking for files accessed by an application. I don't know what function is called to remove files, but I'm pretty sure it can be identified. |
The problem is: files disappear very infrequently.
| szatox wrote: | | Quote: | | What is "the usual way"? |
There are many ways |
You've named only one: backups 
----------------------------------------------------------------
| flexibeast wrote: | | tkzv wrote: | | Does inotify detect the process that did the deletion? |
Not that i can tell, from a quick scan over the relevant man pages, but i might well be wrong. |
Googling for “inotify detect who changed file” suggested sys-process/audit instead. I'll try it. |
|
| Back to top |
|
|
flexibeast
Guru
Joined: 04 Apr 2022
Posts: 425
Location: Naarm/Melbourne, Australia
|
| Posted: Mon Mar 18, 2024 6:08 am Post subject: |
|
|
| tkzv wrote: | | Can it be run by a separate user? |
i would imagine so, via sudo or doas, as long as it can also access any files it needs to start up (i.e. you might need to copy a few files into the analogous locations for that other user).
| tkzv wrote: | | Googling for “inotify detect who changed file” suggested sys-process/audit instead. I'll try it. |
Ah, interesting, i'll be interested to know how it goes (and i'll check it out myself). |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Other Things Gentoo
