View previous topic :: View next topic |
Author |
Message |
maiku Guru
Joined: 24 Mar 2004 Posts: 593 Location: Escaping from NY
|
Posted: Mon Sep 16, 2024 2:49 pm Post subject: Redundant local external storage that's not always on |
|
|
I need a local configuration where I can store data with the following considerations.
1) It's really just local.
2) I can easily use it on another at least Linux machine.
3) Has redundancy so it can suffer at least partial failure.
4) Is encrypted.
5) Can be powered on only when I need it (which will not be enough times to justify a NAS in my mind).
6) Does not require an infinite money budget.
In my primitive mind the only thing I know that serves this purpose is USB so I default to getting a dumb USB enclosure that supports 2 drives and buying two rust platter drives and putting them in a ZFS RAID, etc.
Is there a better way? Seems like the pitfalls of ZFS RAID over USB might make things interesting. _________________ Michael |
|
Back to top |
|
|
pingtoo Veteran
Joined: 10 Sep 2021 Posts: 1339 Location: Richmond Hill, Canada
|
Posted: Mon Sep 16, 2024 2:58 pm Post subject: |
|
|
If there is no performance requirement. I would suggest use two USB storage and just ext4. redundancy is base on automated file system level copy.
This way you can have long longevity as well as easy to move, plus possible versioning to reduce human error. |
|
Back to top |
|
|
lars_the_bear Guru
Joined: 05 Jun 2024 Posts: 537
|
Posted: Mon Sep 16, 2024 3:01 pm Post subject: Re: Redundant local external storage that's not always on |
|
|
Quote: |
5) Can be powered on only when I need it (which will not be enough times to justify a NAS in my mind).
|
My Raspberry Pi 3 NAS draws less than 500mA at idle, and the disks spin down when not in use. Obviously it's a higher energy usage than an uplugged USB stick, but it's something I'm willing to tolerate for the convenience.
I offset the energy wastage by making one fewer car journey a year
BR, Lars. |
|
Back to top |
|
|
maiku Guru
Joined: 24 Mar 2004 Posts: 593 Location: Escaping from NY
|
Posted: Tue Sep 17, 2024 6:24 pm Post subject: Re: Redundant local external storage that's not always on |
|
|
pingtoo wrote: | If there is no performance requirement. I would suggest use two USB storage and just ext4. redundancy is base on automated file system level copy.
This way you can have long longevity as well as easy to move, plus possible versioning to reduce human error. |
Interesting solution. Thanks for this.
lars_the_bear wrote: | Quote: |
5) Can be powered on only when I need it (which will not be enough times to justify a NAS in my mind).
|
My Raspberry Pi 3 NAS draws less than 500mA at idle, and the disks spin down when not in use. Obviously it's a higher energy usage than an uplugged USB stick, but it's something I'm willing to tolerate for the convenience.
I offset the energy wastage by making one fewer car journey a year ;)
BR, Lars. |
Can you tell me a bit more about your set up? Like
1) Is it all out in the open or did you build an enclosure?
2) How are you cooling it and ARE you cooling it?
3) What drive types?
Thanks for the response. _________________ Michael |
|
Back to top |
|
|
lars_the_bear Guru
Joined: 05 Jun 2024 Posts: 537
|
Posted: Wed Sep 18, 2024 7:17 am Post subject: |
|
|
@maiku
Sorry -- I forgot I had upgraded to a Pi 4, because the Pi 3 doesn't support USB 3 (as I recall). I have an open-frame enclosure made from plywood that I had laser-cut so the bits would fit together nicely. There are three compartments -- in the bottom compartment are the mains transformers, the middle compartment has the Pi, and the top compartment two (at present) 8TB USB 3 disks. The disks have their own cooling. The Pi has a propietary heatsink case, so it doesn't need any additional cooling (or, at least, has not so far).
In earlier incarnations of this set-up I managed the disk spin-down using scripts, and it was pretty aggressive -- I never let them spin idle for more than a couple of minutes. But these new disks take 30 seconds to spin up, so I leave the default power-saving mode, which is to spin down after twenty minutes. The spin-up time is inconvenient, but I consider it a price worth paying, to keep the power consumption low. I'll be moving to SSDs when I can afford them.
It's not a blazingly fast system, but I find it OK for daily backups. And it's cheap. Also, because it's a real Linux, I can run other stuff on it as well, which isn't always easy with a proprietary NAS.
BR, Lars. |
|
Back to top |
|
|
sdauth l33t
Joined: 19 Sep 2018 Posts: 659 Location: Ásgarðr
|
Posted: Wed Sep 18, 2024 2:15 pm Post subject: |
|
|
I've built my own "NAS" with decade old parts for this exact purpose. It was way more cheaper than buying usb enclosure since I've had already every parts necessary (not easy to find quality USB enclosure too in the >=4 bay range) with much more capability (proper PSU, proper cooling for drives during long transfer, lots of sata ports, possibility to add a raid card if needed..)
I call it NAS because that's what it is, it runs fully headless, only connected to my router, without any display and fully encrypted too.
I only turn it on for backup / archive purpose since it wouldn't be so good to let that watt monster 24/7 on. (server parts from 2008 are not exactly energy efficient )
It runs gentoo of course with a trimmed down kernel, boots under 10s and use like 80m of ram.
Before that I was long time user of various USB enclosures but there was always an issue at some point (shitty USB chipset, too high temps due to lack of fans..); I still use some USB enclosures though but only with SSD for quick transfer.
I don't know what your storage requirements are but an old laptop (with 2x internal SATA + one extra reusing the optical bay for example) could do the job (assuming you're fine with 2.5" hdd) with the added bonus of embedded UPS (the battery )
Last edited by sdauth on Wed Sep 18, 2024 4:24 pm; edited 1 time in total |
|
Back to top |
|
|
lars_the_bear Guru
Joined: 05 Jun 2024 Posts: 537
|
Posted: Wed Sep 18, 2024 2:58 pm Post subject: |
|
|
sdauth wrote: |
Before that I was long time user of various USB enclosures but there was always an issue at some point (shitty USB chipset, too high temps due to lack of fans..); I still use some USB enclosures though but only with SSD for quick transfer.
|
Unbranded USB enclosures can be cheap, but I have found them unreliable, as you say. On the other hand, a proprietary USB 3 external drive unit only costs about the same as the basic drive itself -- at large capacities, at least. I guess as the drive itself becomes more expensive, the enclosure part becomes a smaller part of the cost. Whatever, I've used Seagate and WD external USB drives for decades, and mostly been happy with the reliability.
USB 3 can probably handle data faster than a Raspberry Pi. Even in a much faster computer, network speed is likely to be the limiting factor, rather than the disk interface. For a long time I used a spare laptop with USB drives, and it worked fine. But a Raspberry Pi uses much less power than a laptop, even with the screen off, and still (probably) won't be the limiting factor in performance.
But the costs and speeds of this kind of thing change almost every day -- I guess you need to do the math, and work out what suits your needs best. I'm looking forward to SSDs becoming cheap enough that I don't need to use magnetic disks any more.
Quote: |
I don't know what your storage requirements are but an old laptop (with 2x internal SATA + one extra reusing the optical bay for example) could do the job (assuming you're fine with 2.5" hdd) with the added bonus of embedded UPS (the battery )
|
In my experience, a bad combination is a laptop with external drives. I had to take the battery out. Otherwise, if there's a power failure, but laptop will keep running, but the disks will fail. They will get marked as RAID failures and, when the power comes back, you'll have to do a full RAID reconstruction. Or take the change and tell mdadm that you're sure the array is clean.
BR, Lars. |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9847 Location: almost Mile High in the USA
|
Posted: Wed Sep 18, 2024 3:13 pm Post subject: |
|
|
What is your churn rate (how many GB do you write every month) and how much total storage is another question.
I'm using desktop PCs with a pile of hard drives acquired free or minimal cost, connected to them as a software NAS, and shutting them down when not used because buying a pile of USB3 hardware is out of the question. My RPi0W is too slow to do this job as I have a couple disks with enough churn rate that doing rsync backups would take an annoying amount of time to work with USB2 (indeed I think a RPi4 is minimum -- though with the old PCs I get SATA connections -- USB2 with cheap hardware/cables is not reliable enough for RAID).
One thing I noted is that encryption is slow. The first machine I want to get rid of is my Athlon II X2 PC that is backing up my PVR. The full disk encryption plus ssh encryption is double encryption and consumes considerable CPU cycles and does feel sluggish while doing backups - the cpu is pretty much fully consumed to maintain the Gigabit Ethernet plus disk encryption. I also have an old Atom server board used for backing up my main server -- it has AES-NI and this is much faster at full disk encryption but ssh encryption is a bit slow -- at least until I figure out how to get it to use AES-NI as well. I haven't gotten remote power working on the Atom board but that may be a long term solution -- once I can get more SATA ports on it as well as how to deal with FDE that requires a password to unlock the LUKS RAID5 (currently only grub/kernel is not encrypted.)
I've been pondering how to reallocate my machines as of late because of the Atom, might have to use it as my main server but again it's slow and pretty much useless as a distccd machine which the current machine contributes to.
Fortunately for me electricity is cheap and my homemade grid tie solar panels is cutting into my power consumption... _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
lars_the_bear Guru
Joined: 05 Jun 2024 Posts: 537
|
Posted: Wed Sep 18, 2024 3:37 pm Post subject: |
|
|
eccerr0r wrote: |
Fortunately for me electricity is cheap and my homemade grid tie solar panels is cutting into my power consumption... |
I think if I were generating my own electricity, I'd make a lot of technical decisions differently.
A RPi NAS uses little power, but I'm aware that it's still mostly power wasted. Still, I estimate that running it for a whole year uses the same amount of energy as driving for forty minutes, or running my central heating for about two hours. So, clearly, I can save energy in better ways than switching off my Rpi NAS when I'm not using it.
BR, Lars. |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9847 Location: almost Mile High in the USA
|
Posted: Wed Sep 18, 2024 3:45 pm Post subject: |
|
|
I don't sweat it as a even a full blown PC, not just a RPi, that's turned on for a few hours a month it's not going to be a huge electrical cost.
I have other PCs running 24/7/365 (they're my distccd/httpd/smptd/nfsd/bind/ssh/... and pvr boxes) and yes those do add up, but that's another thing... _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
lars_the_bear Guru
Joined: 05 Jun 2024 Posts: 537
|
Posted: Wed Sep 18, 2024 4:13 pm Post subject: |
|
|
eccerr0r wrote: | I don't sweat it as a even a full blown PC, not just a RPi, that's turned on for a few hours a month it's not going to be a huge electrical cost.
|
Sure. I went the RPi route so I can leave it on all the time, with minimal guilt.
BR, Lars. |
|
Back to top |
|
|
maiku Guru
Joined: 24 Mar 2004 Posts: 593 Location: Escaping from NY
|
Posted: Wed Sep 18, 2024 5:41 pm Post subject: |
|
|
lars_the_bear wrote: | @maiku
Sorry -- I forgot I had upgraded to a Pi 4, because the Pi 3 doesn't support USB 3 (as I recall). I have an open-frame enclosure made from plywood that I had laser-cut so the bits would fit together nicely. There are three compartments -- in the bottom compartment are the mains transformers, the middle compartment has the Pi, and the top compartment two (at present) 8TB USB 3 disks. The disks have their own cooling. The Pi has a propietary heatsink case, so it doesn't need any additional cooling (or, at least, has not so far).
In earlier incarnations of this set-up I managed the disk spin-down using scripts, and it was pretty aggressive -- I never let them spin idle for more than a couple of minutes. But these new disks take 30 seconds to spin up, so I leave the default power-saving mode, which is to spin down after twenty minutes. The spin-up time is inconvenient, but I consider it a price worth paying, to keep the power consumption low. I'll be moving to SSDs when I can afford them.
It's not a blazingly fast system, but I find it OK for daily backups. And it's cheap. Also, because it's a real Linux, I can run other stuff on it as well, which isn't always easy with a proprietary NAS.
BR, Lars. |
Any pics? _________________ Michael |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9847 Location: almost Mile High in the USA
|
Posted: Thu Sep 19, 2024 1:17 am Post subject: |
|
|
How fast is the rpi4 copying through scp from Gbit ethernet to usb encrypted volumes anyway, can it keep up 100MB/sec? Will it handle RAID5 at speed (write three disks at 50MB/sec each or four disks at 33MB/sec each)? _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
lars_the_bear Guru
Joined: 05 Jun 2024 Posts: 537
|
Posted: Thu Sep 19, 2024 7:39 am Post subject: |
|
|
eccerr0r wrote: | How fast is the rpi4 copying through scp from Gbit ethernet to usb encrypted volumes anyway, can it keep up 100MB/sec? Will it handle RAID5 at speed (write three disks at 50MB/sec each or four disks at 33MB/sec each)? |
I don't think I have the infrastructure to test all that. I only use RAID1, for a start. But, it it's any help, it takes 28 seconds to rsync a 1Gb file to the NAS, from another machine on the same network switch. That comes to about 35Mbytes/sec. But I suspect a substantial part of that time is the encryption on the wire.
Fast, low energy, cheap -- you can have any two out of three, I guess.
BR, Lars. |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9847 Location: almost Mile High in the USA
|
Posted: Thu Sep 19, 2024 4:26 pm Post subject: |
|
|
technically energy cost is also cost, so they're the same thing. So basically it's speed or low cost, pick one.
At 35MB/sec for USB3 devices and Gbit Ethernet, that's pretty horrible, that's like USB2 speeds (if the devices were on separate host controllers of course). Satisfies my curiosity of how fast RPi4's are, bottlenecked at about 70MB/sec (with encryption or just mirroring?). Still better than the RPi0W I have regardless. _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
lars_the_bear Guru
Joined: 05 Jun 2024 Posts: 537
|
Posted: Thu Sep 19, 2024 6:14 pm Post subject: |
|
|
eccerr0r wrote: |
At 35MB/sec for USB3 devices and Gbit Ethernet, that's pretty horrible, that's like USB2 speeds (if the devices were on separate host controllers of course). Satisfies my curiosity of how fast RPi4's are, bottlenecked at about 70MB/sec (with encryption or just mirroring?). Still better than the RPi0W I have regardless. |
Well, Pi's are so cheap that it's worth making your own tests. It isn't necessarily the Pi that is the limitation here. The disks only make 100Mbytes/sec according to benchmarks, and two of them have to be written. The computer I'm sending the files from is a 2015 laptop. Maybe the Pi disk system can be tuned in some way. Don't write off the technology on the basis of my implementation.
BR, Lars. |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9847 Location: almost Mile High in the USA
|
Posted: Thu Sep 19, 2024 6:52 pm Post subject: |
|
|
The theory is that you would have optimized your own solution.
RPis are not free, and I still would need to buy USB3 hardware which I also don't have -- so I'll have to depend on other peoples testing that weren't involved in marketing them. I have a 2012 laptop that will linefill from ssd 100MB/sec to Gbit Ethernet just fine, so I suspect your 2015 can do the same. So it must be the RPi as the bottleneck. _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
pjp Administrator
Joined: 16 Apr 2002 Posts: 20522
|
Posted: Thu Sep 19, 2024 6:57 pm Post subject: |
|
|
Is the 100Mbytes/sec benchmark over the network, or internal to the Pi over USB3? You're using RAID on USB connected drives, correct? Also, is the network interface also over USB? If it has an Ethernet port, is it gigabit?
I'm wondering if RAID on USB is the main problem, though a shared USB bus with networking might at least halve the throughput (once across USB, then once across the USB network).
I like the idea of these things, but they still seem very finicky and of questionable practical use for the cost. _________________ Quis separabit? Quo animo? |
|
Back to top |
|
|
lars_the_bear Guru
Joined: 05 Jun 2024 Posts: 537
|
Posted: Thu Sep 19, 2024 7:20 pm Post subject: |
|
|
pjp wrote: | Is the 100Mbytes/sec benchmark over the network, or internal to the Pi over USB3? You're using RAID on USB connected drives, correct? Also, is the network interface also over USB? If it has an Ethernet port, is it gigabit?
|
The "100Mbytes/sec" is for writing a single drive. These are old-fashioned, not-very-expensive WD drives, but I've found them reliable in the past. I use them in RAID1 pairs. The ethernet interface is notionally gigabit, as is the cabling and the switch.
I only checked the speed because somebody else asked. I'm perfectly happy with how the system performs, for my purposes. I see no particular need to tinker with it.
BR, Lars. |
|
Back to top |
|
|
pjp Administrator
Joined: 16 Apr 2002 Posts: 20522
|
Posted: Thu Sep 19, 2024 7:40 pm Post subject: |
|
|
Thanks. That was entirely for me to learn more about the platform. I've considered buying a Pi multiple times, but it just seems like something that wouldn't be sufficiently useful to me for the amount of effort it would take. I'm sure I don't know enough about ARM as a platform, but it seems sufficiently dodgy in that they lack any consistency in how to get them working. _________________ Quis separabit? Quo animo? |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9847 Location: almost Mile High in the USA
|
Posted: Thu Sep 19, 2024 10:09 pm Post subject: |
|
|
I don't think there's anything wrong with the RPi per-se but the main thing about a lot of ARM platforms are simply not standardized. A custom OS is needed for every ARM implementation, you can't unsolder the eMMC or SD or whatever to one ARM implementation and solder to another, even if both use the exact same eMMC/SD/... chip, like taking a RPi eMMC and sticking into your ARM phone, and expecting it to boot and at least display something on the screen. PC hardware you can usually get some modicum of success when you transplant disks.
This is saying nothing about the speed of these ARM chips. While it seems some of the newer ones do okay, a lot of the older ones are lethargic at best. I would hope that the RPi4 with its GbE and USB3 that it could take advantage of the bandwidth. Perhaps it's only one at a time.
A lot of my PCs, other than the fact I have a lot of slow hard drives, can still copy GbE speeds to the disks and bottleneck at the disks. When I was using USB2 for the disks (BAD decision!) USB2 was the bottleneck for both speed and system reliability, hence I'm a bit hesitant on even using USB3. However as of recent I have some USB3 gear and so far so good, perhaps due to stricter implementation standards. As of now I really don't have much USB3 gear and rely on SATA which is really fast, and RAID5, though it has that penalty of writing parity, is doing a good job spreading the load across the disks so sequential reads/writes are just a little worse than running RAID0 across the disks and get single disk protection.
RAID6 I've not had much luck with. Well, specifically speed. It's horrid. _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
pjp Administrator
Joined: 16 Apr 2002 Posts: 20522
|
Posted: Thu Sep 19, 2024 11:24 pm Post subject: |
|
|
That's why I referred to the "ARM platform" rather than Pi specifically. Pi has popularity going for it, so it seems more consistent, which makes sense. But I often see threads here where there are issues, but I haven't retained a general perception if they are mostly not Pi.
One thing I've considered using one for is the "generally on and available" stuff so I don't have to keep a PC running 24/7. I'm sure it would do most of what I'd want very well, but then I have yet another environment I have to maintain, so it simply hasn't been compelling enough. Dual-NIC would be nice too, which I don't think they've offered yet (though some other non Pi has, but is more obscure). _________________ Quis separabit? Quo animo? |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9847 Location: almost Mile High in the USA
|
Posted: Thu Sep 19, 2024 11:41 pm Post subject: |
|
|
You can always add more usb3 NICs ...
Alas, I keep 24/7 PCs running. I'm down to 2, been running more in the past but started merging them as I got multi-core machines that are faster than what I need. Now trying to merge the PVR and the virtual machine box I'm worried about how stable pass-through PCI works, though still a bit concerned about mythtv security. _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
pjp Administrator
Joined: 16 Apr 2002 Posts: 20522
|
Posted: Fri Sep 20, 2024 12:04 am Post subject: |
|
|
eccerr0r wrote: | You can always add more usb3 NICs ... | Well, yeah. Then you have disk and other stuff going over the same bus. That seems questionable, unless the CPU and USB bus can handle ~8 ports and a networking stack. USB 3.2 can theoretically do 20 Gbps. _________________ Quis separabit? Quo animo? |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9847 Location: almost Mile High in the USA
|
Posted: Fri Sep 20, 2024 4:46 am Post subject: |
|
|
So rule out ARM/RPi (at least RPi4 that is having problems at 70MB/sec written to 2 disks? I wonder how fast it writes to just 1 disk too...)
At least with USB3 at 20Gbps, that's ridiculous bandwidth compared to 1GbE and 2Gbps head rate HDDs, so sharing that is no problem.
Now if you're dealing with SSDs and 10GbE, then it can get tight... _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|