Sensor data logging on Raspberry PI and SD card life...

VinzC

Hi all.

I'm using my Raspberry Pi as a testing gear. I have a script that collects data from a sensor and writes it into a CSV file. For some reason, data logging is done every second as follows:

open the CSV file,
write the few line characters from the sensor,
close the file

Now I'm afraid this might speed up the SD card wearing out and reduce the card's life span. What tickles is that the sensor data is stored on the same partition as the root filesystem, which is mounted read-write, of course.

I'm considering a temporary filesystem to store the data and then copy to long-term storage after data collection is done, for example. I also considered some filesystem caching using overlayfs... but all of this seems a bit overkill, I'd say. Ideally I'd tweak the filesystem to buffer the sensor data for a couple of device blocks and then write them to the SD card.

I guess ext4 (or F2FS, maybe) might just have what it takes, right? Just that I have no clue what parameter to tune. Any clue?

EDIT: To get an idea about the volume of data, the file size is about one megabyte after eight hours.

Thanks in advance.
_________________
Gentoo addict: tomorrow I quit, I promise!... Just one more emerge...
1739!

bbgermany · Posted: Wed Apr 04, 2018 12:29 pm Post subject:

Hi,

Which version of the rpi are you using? rpi2 with BCM2837 or rpi3 can boot from usb or via network. So you could get around this issue

greets, bb
_________________
Desktop: Ryzen 5 5600G, 32GB, 2TB, RX7600
Notebook: Dell XPS 13 9370, 16GB, 1TB
Server #1: Ryzen 5 Pro 4650G, 64GB, 16.5TB
Server #2: Ryzen 4800H, 32GB, 22TB

Jaglover · Posted: Wed Apr 04, 2018 1:19 pm Post subject:

Even RPi1 can use USB as external storage. Or use NFS if your device is networked.
_________________
My Gentoo installation notes.
Please learn how to denote units correctly!

VinzC · Posted: Wed Apr 04, 2018 1:51 pm Post subject:

Thanks for your suggestions, guys but I don't want to use anything but the SD card. I only want to know more of how (and if) I can tune the filesystem to not cause "too many" block writes^Werases, which IMHO would reduce the card lifespan. So although what I want to know is irrelevant of the Pi version, mine is a Pi model 2B.
_________________
Gentoo addict: tomorrow I quit, I promise!... Just one more emerge...
1739!

mike155 · Posted: Wed Apr 04, 2018 2:21 pm Post subject:

vm.dirty_writeback_centisecs = 1500

VinzC · Posted: Wed Apr 04, 2018 3:08 pm Post subject:

Thanks mike155, you've answered my question. However — my bad — I should have stated «block erases» instead of «block writes» as the former more than the latter is responsible for the wearing out effect. Meanwhile I've sought the interweb and from what I've learnt I think I'm going to stick with F2FS and its defaults, storing my sensors' data on a separate partition.

From what I've read on StackOverflow invalidating two 4KB block every second would make my 8GB SD card last about 30 years... extrapolating from the example given by DrV, of course. If F2FS can increase life expectancy by ~70% compared against ext4, then there's no reason not to use it.
_________________
Gentoo addict: tomorrow I quit, I promise!... Just one more emerge...
1739!

mike155 · Posted: Wed Apr 04, 2018 3:30 pm Post subject:

VinzC,

here is what I would do to reduce the number of writes to the SD card:

1) I would let the data collector (which touches the output file every second) write to a data file on a tmpfs file system

2) I would write a small cronjob that runs every 5 minutes and that appends records, which were written to the output file on the tmpfs during the last minutes, to a data file, which is stored on the SD card. Something like:

VinzC · Posted: Wed Apr 04, 2018 4:41 pm Post subject:

Thanks mike. I think I'd implement this bucket-write feature in my Python script to avoid any potential side effect like writing to the data file while reading it. That said I believe F2FS is smart enough to do better than any software trick I'd implement — I'd need thorough testing to demonstrate that, of course.
_________________
Gentoo addict: tomorrow I quit, I promise!... Just one more emerge...
1739!

VinzC · Posted: Thu Apr 12, 2018 8:30 pm Post subject:

I've finally found a way: I've used overlayfs so that the data are stored in a file on a tmpfs, which is mounted on /tmp/sensors. OverlayFS needs a permanent storage, which I mounted under /mnt/sensors as F2FS. The data file is then transparently moved from the temporary storage in /tmp/sensors to /mnt/sensors. Done.
_________________
Gentoo addict: tomorrow I quit, I promise!... Just one more emerge...
1739!