| View previous topic :: View next topic |
| Author |
Message |
lars_the_bear
Guru
Joined: 05 Jun 2024
Posts: 522
|
 Posted: Fri Sep 20, 2024 9:20 am Post subject: |
 |
|
| pjp wrote: |
One thing I've considered using one for is the "generally on and available" stuff so I don't have to keep a PC running 24/7. I'm sure it would do most of what I'd want very well, but then I have yet another environment I have to maintain, so it simply hasn't been compelling enough. |
I don't really feel my Pi NAS has to be maintained. If it were exposed to a hostile network environment, that might be a different story. But, sitting in my house running rsyncd, I'm not bothered.
Also, these are commodity devices. If I cook one of them -- and I do this fairly frequently in my embedded applications -- I just throw it away and buy another on eBay. The entire operating system is on an SD card, so I just plug the card into a new board. And if I blow up the card, I have a backup I can flash on a new card in a minute or two.
Of all the computer stuff I have in my house, the Pi-based stuff is the only part of it that is completely reliable.
But, as I've said, my applications are undemanding. I don't care whether my daily backup takes one minute or two.
I haven't been brave enough to install Gentoo on them yet, though. For my embedded applications, I built the operating system from scratch using scripts. I've re-implemented Portage, badly. I'd like to try Gentoo on the Pi, but it's not come to the top of my to-do list. I think this will be a job for my retirement. For the NAS I just use DietPi, with no X or desktop stuff.
There are specific "NAS" images for RPi, but I haven't tried any of them.
BR, Lars. |
|
| Back to top |
|
 |
eccerr0r
Watchman
Joined: 01 Jul 2004
Posts: 9824
Location: almost Mile High in the USA
|
| Posted: Fri Sep 20, 2024 12:44 pm Post subject: |
|
|
I back up my PVR and my NAS/Shell/VM/HTTPD/SMTPD/etc. boxes. Both boxes take about a half to a full hour to rsync -- the PVR has somewhat high churn rate, the shell box VMs take a while to do diffs and depending on how much churn it could take more than an hour to rsync... Both cases I think the bottleneck is the speed of seeks and read rate of the hard drive sets but the PVR backup is real close to being CPU-limited. When it's not cpu limited, it still gets real slow when trying to do diffs between two directory trees with a lot of directories and small files.
The PVR has a 2TB disk (single disk), the shell box is only 4TB (3x2TB RAID5). My rsync backups are to two different machines with encrypted RAID5. I now back up the PVR to a 5 disk 500GB encrypted RAID5 as that's the hard drives I had spare from upgrades or acquired e-waste, and the shell box I back up to more 2TB disks.
Currently thinking about changing my backup strategy after acquiring more e-waste hdd's instead of running so many 500G HDDs... though the RAID works just fine and I now have plenty of 500G disks as cold spares for that array. At least the 500G disk array takes only an hour and a bit to rebuild, my 2T disk array takes 5 hours. Ouch.
I don't think I'll go to 6 disks though that's the limit, mostly to leave one port open for a hot spare if needed on the 6-port motherboards. Need to find sata PCIe cards...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
| Back to top |
|
|
Bob P
Advocate
Joined: 20 Oct 2004
Posts: 3374
Location: USA
|
| Posted: Sun Oct 20, 2024 5:37 pm Post subject: Re: Redundant local external storage that's not always on |
|
|
| maiku wrote: | I need a local configuration where I can store data with the following considerations.
1) It's really just local.
2) I can easily use it on another at least Linux machine.
3) Has redundancy so it can suffer at least partial failure.
4) Is encrypted.
5) Can be powered on only when I need it (which will not be enough times to justify a NAS in my mind).
6) Does not require an infinite money budget.
In my primitive mind the only thing I know that serves this purpose is USB so I default to getting a dumb USB enclosure that supports 2 drives and buying two rust platter drives and putting them in a ZFS RAID, etc.
Is there a better way? Seems like the pitfalls of ZFS RAID over USB might make things interesting. |
Long, long ago I fell victim to the allure of mounting a USB hard drive on a Pi while I convinced myself it was a viable backup solution. I thought it was great to have cheap/disposable commodity hardware running low-power backup on my LAN. Everything was great until the weak links started emerging... things like transient power losses killing the SD card, limited SD card lifespans killing the Pi, RPi that lacked real disk interfaces forcing me to use less than robust USB data connections that routinely failed in use, other flavor Pi that offered SATA connections but still resulted in a less than robust overall solution. Whenever something went wrong it was always a major asspain fixing it. The power and component savings were illusory and were never justified in light of the repair/downtime frustration they caused.
Between then and now, many other solutions were attempted on temporary basis.
Years later the homelab had evolved to a dedicated multicore ZFS server on BSD with lots of ECC RAM, multiple vdevs comprised of redundant Z2 or Z3 arrays, long-duration UPS power and automated shutdowns in the event of a power outage. Today is years after that, and I don't need BSD anymore b/c I'm able to run ZoL. When a disk fails, I slide in a replacement and issue a command to resilver the drive and I'm done. No headaches.
There's just no way I'd go back to a Pi. If you're serious about your data, then ZFS is the answer. You can power it down intermittently if you want to, but doing that creates it's own problems. Why reduce the robustness of your archive system by only allowing it to take intermittent snapshots of your data. Me? I burn the power.
(first post here in 20 years) |
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 5121
Location: Bavaria
|
| Posted: Sun Oct 20, 2024 11:41 pm Post subject: Re: Redundant local external storage that's not always on |
|
|
| Bob P wrote: | | (first post here in 20 years) |
... 18 years ... 
Welcome back! 
_________________
https://wiki.gentoo.org/wiki/User:Pietinger |
|
| Back to top |
|
|
lars_the_bear
Guru
Joined: 05 Jun 2024
Posts: 522
|
| Posted: Mon Oct 21, 2024 7:40 am Post subject: Re: Redundant local external storage that's not always on |
|
|
| Bob P wrote: | Everything was great until the weak links started emerging... things like transient power losses killing the SD card, limited SD card lifespans killing the Pi, RPi that lacked real disk interfaces forcing me to use less than robust USB data connections that routinely failed in use, other flavor Pi that offered SATA connections but still resulted in a less than robust overall solution. Whenever something went wrong it was always a major asspain fixing it.
|
FWIW I'd like to point out that in all my years of running Raspberry Pis as NAS systems, I've never, not once, experienced any of these problems. Maybe I've just been lucky -- I don't really have a representative sample of other people's experiences. Still, I leave in a region where power failures are not uncommon. I had three last week. Pis are fine. I wish I could say the same about everything else.
| Bob P wrote: |
The power and component savings were illusory and were never justified in light of the repair/downtime frustration they caused.
|
The energy savings are easy to calculate, if you use a Kill-a-Watt or similar. In my case, the difference between an average power consumption of about 2W for the Pi, and ~60W for the old PC, is worthwhile. The costs saving in components is modest, and depends on the amount of stuff you already have in the junk pile. Since I've never had to carry out any repairs, or experience any downtime, I can't comment on how frustrating these things are 
Whether any of the cost savings is significant or not, depends on your priorities. For me, minimizing energy usage is a matter of principle, regardless of cost.
BR, Lars. |
|
| Back to top |
|
|
maiku
Guru
Joined: 24 Mar 2004
Posts: 593
Location: Escaping from NY
|
| Posted: Mon Oct 21, 2024 1:33 pm Post subject: Re: Redundant local external storage that's not always on |
|
|
| Bob P wrote: | Long, long ago I fell victim to the allure of mounting a USB hard drive on a Pi while I convinced myself it was a viable backup solution. I thought it was great to have cheap/disposable commodity hardware running low-power backup on my LAN. Everything was great until the weak links started emerging... things like transient power losses killing the SD card, limited SD card lifespans killing the Pi, RPi that lacked real disk interfaces forcing me to use less than robust USB data connections that routinely failed in use, other flavor Pi that offered SATA connections but still resulted in a less than robust overall solution. Whenever something went wrong it was always a major asspain fixing it. The power and component savings were illusory and were never justified in light of the repair/downtime frustration they caused.
Between then and now, many other solutions were attempted on temporary basis.
Years later the homelab had evolved to a dedicated multicore ZFS server on BSD with lots of ECC RAM, multiple vdevs comprised of redundant Z2 or Z3 arrays, long-duration UPS power and automated shutdowns in the event of a power outage. Today is years after that, and I don't need BSD anymore b/c I'm able to run ZoL. When a disk fails, I slide in a replacement and issue a command to resilver the drive and I'm done. No headaches.
There's just no way I'd go back to a Pi. If you're serious about your data, then ZFS is the answer. You can power it down intermittently if you want to, but doing that creates it's own problems. Why reduce the robustness of your archive system by only allowing it to take intermittent snapshots of your data. Me? I burn the power.
(first post here in 20 years) |
Glad to see you back! Thanks for your experience.
Not to jump on the "how dare you flame RPi!" bandwagon, but I use RPis pretty consistently for a side gig I do. I have about 300 deployed in production. I must say, uptime issues and SD card corruptions have not been such a big issue over the 10 years I've had doing it. Sure, some have had to be replaced. But that has been the exception to the rule in my case.
That's why I had mentioned using an RPi in my post. I don't have any problems with them in specific. However, it just seems like such a complicated and heavy sledgehammer to me to solve this problem.
100% agree, ZFS is awesome.
_________________
Michael |
|
| Back to top |
|
|
Bob P
Advocate
Joined: 20 Oct 2004
Posts: 3374
Location: USA
|
| Posted: Mon Oct 21, 2024 8:30 pm Post subject: |
|
|
Oh, I'm not a pi-basher by any stretch of the imagination, but based on the amount of time between my posts I think you can get the idea how old some of my Pi experiences were.
I'm sure that the RPi ecosystem has improved a lot since the early adoption period. I jumped on the bandwagon early. Back then we had to learn the hard way (through failures) about SD card problems that would take down the system and how to design the system to avoid the Pi's inherent weaknesses. Back then that problem wasn't even recognized or discussed on Pi site as we were blazing the trail, trying to press the platform do more than it was originally envisioned to do as a "teaching system for school children." Back then those of us who tried to deploy them as constant-on computing platforms were running into new problems that took some time to figure out. I'm not saying that the Pi is a bad system, but way back when, things weren't very well documented and eventually I got tired of beating my head against the wall solving problems. Today things are a lot better, as anyone who wants to use the system can benefit from the solutions that were provided by all of the people who toughed it out in the early period and documented all of the problems and workarounds. When I was active the system just wasn't as robust as it sounds like it is today, so I moved along.
For me the lack of a proper disk interface forced me into kludge adaptations that weren't sufficiently robust for my needs, so I looked for other solutions. I just couldn't justify trying to press the system do something that it was never designed to do... with what was probably an unreasonable expectation of fault tolerance on my part. I mean things were really buggy back in the beginning. I spent a ridiculous amount of time trying to make the Wolfson Audio card work to build a Pi renderer. It turned out that I wasn't doing anything wrong, it was just that the problem was unsolvable because the card's drivers didn't work as promised.
Fast forward many years later and the ecosystem has matured quite a bit, to the point that the weaknesses are well known, the workarounds are well-documented, and new features exist that improved the plaform. But in the old days the SBC concept just wasn't all that I needed for the jobs I was doing, so I moved on. I tried getting back into the ecosystem when the later versions were relased but their unobtainium status at the time killed my joy. I haven't been back since.
I've never understood why they continue to design those things so that all of the IO ports stuck out in 4 different directions instead of using an inline configuration. That's always been a design compromise that I never undestood.
I still have all of my cards, hats, relay boards, and all kinds of other stuff sitting in a box and I keep promising myself that I'll do something with them someday. For me those things will be one-off projects, rather than production deployments. My problem is that the amount of time that it takes to work out a one-off solution has been cost prohibitive for me. I just don't have enough time. I could see where it would be worth time to develop a system that you could replicate and put into production. That would make the time investment worthwhile, but time has been my worst enemy in that regard.
If you don't mind me asking, what kind of deployments have you put into production on the Pi? |
|
| Back to top |
|
|
Bob P
Advocate
Joined: 20 Oct 2004
Posts: 3374
Location: USA
|
| Posted: Mon Oct 21, 2024 9:20 pm Post subject: |
|
|
| maiku wrote: | | However, it just seems like such a complicated and heavy sledgehammer to me to solve this problem. |
Well, to be fair, the criteria that you mentioned in your first post sounded like they were pretty demanding when I read them. They don't seem to me like an easy problem to solve with minimal hardware that's only on intermittently. I guess it would help if you could explicitly state your objectives when it comes to things like total TB of storage needed, expected system responsiveness, IO bandwidth, fault tolerance, etc. When you said "ZFS" that made me wonder how high your expectations really are.
If you want redundancy and fault tolerance then ZFS is a good solution, but deploying ZFS sets the bar pretty high and it may not be necessary for your needs. If you want ZFS levels of redundancy and fault tolerance you won't get there with a Pi, or any sort of minimal hardware. ZFS requires the "sledgehammer" approach when it comes to hardware requirements. ZFS expects obscene amounts of ECC RAM and expects the OS to be on 24/7 so that it can do it's thing (like automatic scrubs). If you really want / need ZFS levels of performance we can have that discussion, but I don't want to take the conversation down that road if it's not really what you want.
If you want disk encryption on the fly then you're really best off if you're using a recent CPU that has the built-in encryption instruction set, which is going to require some budget. IMO if you want responsive disk encryption/decryption on the fly the performance on old CPU without the encryption instruction set will be poor -- likely intolerable.
I'm not a fan of USB3 disk interfaces for backups, but then that has to do with the duty cycle that you'll ask the disk to perform. I've just had too many experiences with USB chipsets that drop connections during long data transfers (rsync reporting broken pipes). If you want to fill a BIG drive over USB the interface just isn't sufficiently robust IMO for sustained transfers that could take a whole day or more. I've also had USB-enclosed enterprise level drives burn up in their fanned enclosures when trying to fill them with data. A 100% duty cycle for a solid week is just too stressful for a portable USB enclosure. I'll buy USB drives, shuck them and put them on their native interface in a rack enclsoure. Of course this is because run the disks pretty hard -- it can take over a week to populate a Zpool over a gigabit LAN and duty cycle is something that needs serous consideration. Drive temp monitirong becomes important. Of course if you're talking about a system with small storage space needs that doesn't require drive bays that sit in a dedicatedrack enclosure then duty cycle may not be that important. It all depends on your needs.
Tell us more about what your objectives are. It would help to know how many TB of storage you need, what levels of redundancy/fault tolerance you require, what kind of read/write speed you require (do you want to saturate a high speed LAN interface?), etc. I've done very simple setups that just sync a couple of disks for non-real-time redundancy, and I've done very complex high availability external storage setups running dozens of drives. |
|
| Back to top |
|
|
eccerr0r
Watchman
Joined: 01 Jul 2004
Posts: 9824
Location: almost Mile High in the USA
|
| Posted: Mon Oct 21, 2024 10:36 pm Post subject: |
|
|
IMHO if one filesystem requires ECC RAM to get the same reliability as another without ECC RAM, the former is lacking.
If both filesystems benefit equally with ECC RAM, then neither really benefit.
If one filesystem corrupts less than another both without ECC RAM, that filesystem is better...
Sigh. I'm running ext3/ext4fs over LVM over RAID5, without ECC RAM. Been thinking about "downgrading" my Core2Quad server to an Atom that has ECC RAM -- but that Atom is like half to third the speed of the C2Q in GCC and it's expected to be part of my distcc farm. The atom is faster than the C2Q in AES however...
Wow... 1Gbps Ethernet is approximately 8TiB/day. Have to keep this in mind when estimating how fast things go...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
| Back to top |
|
|
Bob P
Advocate
Joined: 20 Oct 2004
Posts: 3374
Location: USA
|
| Posted: Wed Oct 23, 2024 4:34 pm Post subject: |
|
|
When ZFS does it's scrubs, the filesystem gets moved back and forth a lot between disks and memory. The copy-on-write system churns the disk quite a bit to prevent bit-rot.
I once had my box report to me that it suffered, recognized, and corrected a spontaneous memory bit failure during a scrub. I'm glad that the error was fixed automatically by ECC and the BIOS reported it. This happened one time over the course of very many years of service on that box. If I was not running ECC RAM that flipped bit would have been written to disk. Who knows what might have resulted from that. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Gentoo Chat
