2022: Your (home) Gentoo server: how much data do you hoard?

figueroa · Posted: Sat Sep 03, 2022 3:54 am Post subject:

I had no idea I was such a relatively lightweight user. ALL my non-redundant personal data is only a little over 256GB. The backup archive of all my personal data, compressed with zstd level 3 where appropriate (I don't re-compress files that are already compressed), is currently 108GB.
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi

eccerr0r · Posted: Sat Sep 03, 2022 1:59 pm Post subject:

Be glad you don't have large disk space requirements, one of the main reasons for making this thread was to also wonder the other aspect of hoarding data - how to keep it backed up.

Since I had a lot of space on my 4TB RAID array, I was using it to back up my MythTV backend's 2TB disk. But then I've been finding that I was running out of space on the 4TB array and adding additional drives to actually do work on - this is kind of silly as backup data does not need to be online...

I recently ended up getting a few spare, unfortunately small hard drives and set up another slow machine to hold/RAID them - and used that machine for backups. And turn it off when I don't need it. That freed up a lot of space on the main array.

Probably the issue at hand is trying to keep data with similar backup requirements together, as well as figuring out a way to keep the 4TB array backed up (I had been using another 2TB disk to back up the 4TB array, which of course doesn't work very well when it's full.)
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

figueroa · Posted: Sat Sep 03, 2022 5:14 pm Post subject:

My primary desktop backups are non-incremental, compressed (where appropriate) tar archives on my third internal hard drive, sdc, on a rotating daily, weekly, and monthly basis using four large partitions. These are made while I sleep managed by crontab. The standard output of this process is emailed to me after execution. The partitions on sdc are normally unmounted.

My secondary backups are gnupg AES256 encrypted archives of the above onto weekly and monthly rotating USB flash drives, currently four pairs of 128GB and 256GB units. All commands are contained in home-grown bash scripts and calendar reminders also come to me by email. One complete set resides on my key ring and another is off-site.

I'm not paranoid; just careful. My objective is resilience against catastrophe.
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi

eccerr0r · Posted: Sat Sep 03, 2022 6:24 pm Post subject:

Much better than what I have, I suppose that's a benefit of keeping the hoard low.
At most I do full system copies, and some incrementals, technically my thrash is fairly low.

What I need to do is fully go encrypted backups... technically my mythtv box does not need to be backed up onto an encrypted system but it ended up as such "just because" ...

Technically for resilience from catastrophes I need to do offsite backups. And that I don't have.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

Spanik · Veteran Joined: 12 Dec 2003 Posts: 1003 Location: Belgium

Backup is a fully manual thing with me. I have USB hd's that are connected when wanted and these have copies of specific data like photos, recordings, scans. These are 500GB drives. Then there are 3 rotating 3TB usb drives that have a full backup with rsync and one of those is always at an other location. I don't do encryption, there is nothing worth encryping and I have a tendency to loose passwords.
_________________
Expert in non-working solutions

eccerr0r · Posted: Fri Sep 09, 2022 12:07 am Post subject:

Technically speaking, what I was hoping for on my main 4TB server:

Currently I'm using a 2.2TB 4-disk RAID5 array to back up the 4TB 3-disk RAID5 array/server (via rsync), and this works only because the array isn't full yet, and not even half full. This 2.2TB array is on a different machine... and eventually I actually want this machine to be a mirror of the 4TB server such that if the 4TB server goes down, the 2.2TB array can be quickly put into service -- not just a disk backup, but whole machine backup. As usually I really hope I never have to do this, for the same reason nobody wants to actually use their backup media. It's just insurance and trying to keep uptime.

I'm using two roots to do this, one to hold a different IP and do the backup service, other root is the backup root of the 4TB array. It's not fully done yet but ideally all I need to do is change the root=/dev/mapper/lvmvolume and I get the personality of the original machine.

Due to the smaller array I'm currently playing games with lvm volume management - the way it was intended, instead of preallocating space. So I suspect fragmentation will happen over time, but this should still be okay, ideally the 4TB server/array is quickly back up.

Incidentally now the two backup arrays (for the 4TB array and the 2TB PVR box) I have are now FDE, so if these disks go "Tango Uniform" or if I gave a disk away, I don't need to worry about it... This is even more important as these disks have private ssh keys that allow access into their hosts for backup without a password...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

onlinefloh · n00b Joined: 02 May 2009 Posts: 25

For me, with currently net 48TB used, it's just a hair under the 50TB limit. At the moment the machine could with its four 16TB disks take up another ten TB worth of data.

Unfortunately, I don't have all my data online yet: there's still a few hundred rolls of film (photo stuff) waiting to be scanned, but the film scanner is waiting for repair...

Topic-wise, it's a bit more than 1TB Music (flac), a few hundred Gigabytes Audiobooks (~128-160 kbps MP3 or ogg), a good TB each of Photos (RAW of course) and educational content, and the vast majority taken up by Movies as I moved shelves of DVDs and Blurays to harddisk for ease of use.

Granted, the Music and Movies part is not entirely mine alone, as I'm sharing that with two friends: there's three identical machines physically located in our basements, replicating data to the other two every night. Twice a month a snapshot is moved to a fourth machine in my basement which otherwise is offline. Besides Music and Movies, everything else is additionaly backed up to tape every three months, with the tapes going to my workplace for storage. For Music and Movies, there's no real backup to tape as they can relatively easily be restored from original optical media. Here I "only" have metatdata necessary to recreate the files: for music essentialy the abcde logs, and the MKVToolnix save files for Videos. Restoring would then just be a matter of plunging the respective disc into an optical drive and rerip. Sure, Re-ripping all these optical discs would be a major pain, but the likelyhood that all three live copies AND the offline copy going bad at the same time are rather slim.

As there was a question of how long a filesystem repair would take: originally we started out with 4 12TB disks as BTRFS RAID1. There, a full background scrub and rebalance took anything between 20 hours and three days, depending on how much data had to be moved. In the meantime as our hoard grew we migrated to RAID5 Data/RAID1C3 metadata, a full scrub and rebalance taking between two and twelve days, and finally moved on to Single Data/RAID1C3 metadata, dropping maximum observed scrub and rebalance times to three to four days, and later replaced 12 TB disks with 16 TB disks, which did not noticeably increase scrub and rebalance times as the drives are a bit faster.

On my end, I upgraded the network to 10GBit to reduce the time needed to sync the offline machine. Sync between the three online machines is limited by Internet connection speeds and literally would take weeks for a full transfer. In case a disk dies, it's still faster to copy the data locally to fresh hard disks on one machine and send the new disks by mail... ;-)

pa4wdh · l33t Joined: 16 Dec 2005 Posts: 883

My server has 2 1TB SSD's in RAID-1, so i actually have 1TB of storage available.
Of that 1TB currently 474GB is actually in use (allocated to logical volumes).
Of that 474GB, 20GB is allocated to my home dir of which 16GB is used.
Of that 474GB, 300GB is allocated to shared storage of which 275G is used.

As for the poll ...

eccerr0r · Posted: Sun Sep 18, 2022 5:30 pm Post subject:

Yeah I didn't want this to be a disk size competition, I was just curious of how much non-redundant data people were holding onto. It looks like you're hoarding 291GB of the 1TB of storage you have, plus that other 1TB of redundancy. Also I was sort of debating whether or not owning two machines that have rootdisks with 20GB worth of Gentoo "redundant" as they are both OS (so did one hoard 20GB or 40GB?), I'll leave that as a discretion to the reader too.

I think I have maybe 20TB worth of hard drives now (rough guess), but a lot of it is holding onto redundant data as I assume hard drives will fail (RAID, standard backup copies, as well as cold spares), as well as a LOT of unused disk space especially in the cold spares. That number also may include a few drives that are known to be flaky like one of my 2TB disks that I can still sort of use despite massive SMART failures on it.

Perfect Gentleman · Veteran Joined: 18 May 2014 Posts: 1255

pa4wdh · l33t Joined: 16 Dec 2005 Posts: 883

bulletbill22 · n00b Joined: 15 Oct 2022 Posts: 34

I definitely no data hoarder, but I've collected plenty of data in my time. I have a 1 TB "data" disk, I'd guess is about 750GB. The only stuff I use regularly is the music folder, because I don't subscribe to music streaming services and I want to be able to listen to music even when I don't have any internet connection. Also have a 160GB drive I have a blockchain on, maybe 90GB? Not counting the base install and the actual server data.

ChrisJumper · Advocate Joined: 12 Mar 2005 Posts: 2400 Location: Germany

Data is not important. The Community Net will fetch everithing important and burn it to Movies, Books and talking Points.

So i like to have a Backup. And 16 TB of Movies from the last 20 years, i thought, will be important. However its useless in a world of personalized Computer Systemems and Information Age. Since next Generagen can not even toggle the Switch to change a channel or surf a specific url.