Which RAID should I use?

FizzyWidget · Posted: Mon Jun 02, 2014 11:15 am Post subject: Which RAID should I use?

Here is my problem

I have 3TB (2.72TB) of space in my main system, I use the spare PC as storage, it has 4 500GB hard drives in it, while I am aware I cannot copy everything over I would like to be able to store the majority of my stuff on the space PC, I know that if I use raid 5 or 6 it will give me about 1.3TB (as we all know drives advertised amount of space and actual space you can use never match), where as if I use raid10 i will only get about 934GB of space, out of the choices I have, other than to have 4 separate drives which RAID would people advise?
_________________
I know 43 ways to kill with a SKITTLE, so taste my rainbow bitch.

krinn · Watchman Joined: 02 May 2003 Posts: 7470

If i pickup your space lost for a drive, 500*3-1300 = 66M / drives, than why not using raid0 to get 4*(500-66) and get an array of 1.7T
It depend on your level of paranoia, but 1.7T to backup 2.72T should be fine, and making sure your backup array is fine shouldn't really be a problem. Backing up the backup is sure safe, but you can live well with a backup and the original and don't need to backup the backup

FizzyWidget · Posted: Mon Jun 02, 2014 12:44 pm Post subject:

krinn · Watchman Joined: 02 May 2003 Posts: 7470

It's more or less the same as any disk, except chance of failure is a bit increase by number of disks, but it doesn't change in real your failure time, the failure time comes from the first disk to fail.
But if you use raid0 as primary or backup, it won't change anything. If original fail, you have the backup, if backup fail, you have original.

If you look again at the logic, it would even be more intelligent to invert the disks, using primary the raid0 with the 4x500 disks for increase speed and using the bigger slower disk as backup (as it will get all your raid0 datas backup) while any other solutions would imply a filter and some datas lost.

FizzyWidget · Posted: Mon Jun 02, 2014 1:58 pm Post subject:

I know what you mean, and I have just found a few more 500GB HDD, so if the raid0 fails I have spares, and can copy things back over, I suppose i have to weigh up the space issue and the rebuild time if i were to use raid5 or 6 against how long it would take to copy over from main pc.

Now just to figure out should I used the bios raid or mdadm

_________________
I know 43 ways to kill with a SKITTLE, so taste my rainbow bitch.

Pearlseattle · Posted: Mon Jun 02, 2014 7:37 pm Post subject:

Hi

I'm a big antagonist of raid0 :evil:

Think that if, while you're writing to it, the power is cut then you've lost it, no matter what.
It happened to me a couple of times with a raid5: I plugged in too much stuff into the same plug three times, I switched off the wrong power button two times, once I shut down everything in a hurry after having thrown 1.5l of coke over the power plugs, etc... .
Keep in mind that this kind of stuff usually happens when you're synchronizing your backup with your master copy, so both raids might have problems => I would never use raid0 if not to host temp data.

And I'm an even bigger antagonist of HW-raid :twisted:

I have a mdadm-raid5 with 4 7200rpm 4TB disks which writes with minimum 250MB/s and has spikes of 450MB/s (depends quite a lot on the physical place on the HDD where it's writing and if the area is completely free or not) and the CPU usage is quite low => I don't know why anybody would want to use a HW-card which might slow down the data throughput (all inexpensive cards won't be faster than the CPU controller) and generate a dependency towards the HW-manufacturer as if the card breaks down you won't be able to use those HDDs with a different brand.

I therefore recommend you to
1) decide which raid-levels you consider being candidates.
2) test them with your HW and different parameters and weight their pros/cons & risks.
I wrote here some notes some time ago (not finalized and not reviewed so they might be silly but might give you some inspiration for potential tests).

Have fun!

p.s.: using the BIOS RAID instead of a RAID CARD is the same thing - you'll have to stick to that motherboard without ever being able to upgrade MB/CPU.

szatox · Advocate Joined: 27 Aug 2013 Posts: 3477

FizzyWidget · Posted: Mon Jun 02, 2014 9:13 pm Post subject:

Pearlseattle · Posted: Mon Jun 02, 2014 10:36 pm Post subject:

vaxbrat · l33t Joined: 05 Oct 2005 Posts: 731 Location: DC Burbs

You could do a mirror set with btrfs and enable lzo compression. Depending on the nature of your files, you might be able to squeeze everything on the smaller volume set. Your data will be a damn sight safer than a raid0 or even a raid5

Goverp · Advocate Joined: 07 Mar 2007 Posts: 2186

krinn · Watchman Joined: 02 May 2003 Posts: 7470

Pearlseattle · Posted: Wed Jun 04, 2014 9:31 pm Post subject:

Goverp · Advocate Joined: 07 Mar 2007 Posts: 2186

eccerr0r · Posted: Thu Jun 12, 2014 9:26 pm Post subject:

Speaking of RAID5's I have a 4x500GB RAID5 on SATA 3Gbps. I find that disk i/o performance to it leaves much to be desired.

My current system has root on LVM on on MD-RAID5 (setup via initramfs, not autoconfigure). Boot partition is also on those disks (4 way MD-RAID1), in hopes that if any one of those 4 disks goes belly up, the machine is still bootable, yet I have only 4 spindles going (versus a 6-disk system where I dedicate a raid1 for root/boot). The member disks are "consumer" quality SATA disks. Swap, which is not commonly used, is a swapfile on root (6GB RAM)

However apparently the amount of i/o that goes to the RAID5 on the root partition (reads and writes) seems to really choked out RAID5 performance. hdparm works OK but when I do real read/writes the effective IOPS is really low. The disk io to the virtual machines on the machine are miserably slow, I suppose I get no more than 10MB/sec or possibly worse.

Not sure how people are configuring this and/or if this is a localized issue, but it's pretty pitiful. I have two machines with SATA 6Gbps SSDs in them and they make working on the RAID5 feel like pouring molasses (if even just one fast HDD feels slow compared to the SSD, the raid5 is even worse.)

I am thinking about eating the space inefficiency and going with a 2x2TB RAID1 in the near future because of how bad the disks are to the virtual machines...

Ugh... I just looked:
[ 3.670484] ata5.00: 976771055 sectors, multi 16: LBA48 NCQ (depth 0/32)

No wonder why this is behaving so poorly... Need to get NCQ to work again...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

depontius · Advocate Joined: 05 May 2004 Posts: 3522

While we're talking RAID, I've been running ext4 on top of mdadm RAID1 for years now, serving it as /home over nfsv4 on my lan.

I've getting set to rebuild, and was thinking of moving to the btrfs with its internal raid1, still serving over nfsv4.

An comments / pros / cons?
_________________
.sigs waste space and bandwidth

Pearlseattle · Posted: Fri Jun 13, 2014 8:51 pm Post subject:

depontius · Advocate Joined: 05 May 2004 Posts: 3522

Pearlseattle · Posted: Sat Jun 14, 2014 12:53 am Post subject:

hehehe..., yes, it's shiny.

vaxbrat · l33t Joined: 05 Oct 2005 Posts: 731 Location: DC Burbs

Pro-Tip: make sure your memory is good. The two times I lost btrfs arrays were due to bad memory as later shown by memtest

"cp --reflink=always" is a new switch to keep in mind. It's wonderful to be able to snapshot a VM image file in a couple of seconds and then to be able to roll back those changes in seconds. Great for playing with test software inside a VM but not so hot if your corporate DC decides to throw you out of the domain on a whim because your machine account password got out of whack

I have a 2x4tb mirror set that has nicely handled a bad block replacement or two. Its a wonderful thing to see btrfs messages in your /var/log/messages announcing that it has found a bad checksum and repaired the nasty thing

I've been running a 9tb 4x3tb RAID5 now for about 6 months, and it's about 6tb filled. One of the drives has decided to pend-uncorrect 16 sectors. I've been keeping everything rsync'd to another btrfs RAID5 on another box. I first saw some grief trying to do a rebalance after going to kernel 3.10.7 to 3.12.21-r1. The whoopsie appears to be isolated to a single centos 6.2 vm that I'm going to try to "recover" just for giggles. I'm in the middle of a scrub on the drive, and sure enough, it's whining about the vm image:

eccerr0r

frostschutz · Advocate Joined: 22 Feb 2005 Posts: 2977 Location: Germany

I'm happy with a 7x2TB HDD raid5.

Proper disk monitoring is crucial. Test for read errors, replace failing disks immediately. If you don't, the first time any disk fails and you rebuild your raid, the rebuild will be your first full read test and the chance of hitting previously undiscovered read errors is high. And at that point you have data loss and your raid may not be recoverable at all.

Of course, you also need a backup.

vaxbrat · l33t Joined: 05 Oct 2005 Posts: 731 Location: DC Burbs

I ran the dreaded btrfs.fsck and it didn't find anything wrong with the filesystem. However the data blocks in the VM were still bad so I ended up blasting the image file over with a backup copy. Although I've run this array for a good half year as btrfs, I had previously run it as an ext4 mdadm array for a couple of years. As an indicator of its age, the mobo is an AMD hexacore thuban. It's probably time to consider the drives as being close to their expected lifetime (what's the warranty now, 2 years?.... don't make em like they used to).

I'll probably build an AMD Piledriver based system this weekend with a fresh set of 4tb drives and then have the old box nuke itself with a low level format after everything is copied over. I'm starting to do a manual daily rsync over to my other box now that I know that things are getting a little flakey.