[solved] Questions about setting up and maintaining a RAID5

[Jimini]

Howdy,
at the moment, my fileserver consists of 2 mirrored 1,5TB-harddisks (RAID1). In the next months, I plan to replace this array with a bigger RAID5 - I'd prefer something about 4 TB of usable diskspace. I also want to be able to resize that array, if I need more space. In comparison to RAID1, which is really easy to set up an maintain, RAID5 ist a bit more complex - so I used Virtualbox to simulate such an array and to play around a bit.
This is what I did so far:
1) created the partitions (sdb1, sdc1, sdd1, sde1)
2) created the array: mdadm --create /dev/md0 --level=raid5 --raid-devices=4 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1
3) for testing purpose, I removed one device: mdadm -f /dev/md0 /dev/sde1 && mdadm -r /dev/md0 /dev/sde1
4) then I added one device: mdadm -a /dev/md0 /dev/sdf1 and let the array reshape
5) for testing purpose, I added another device and resized the array: mdadm -a /dev/md0 /dev/sde1 && mdadm --grow /dev/md0 --raid-devices=5 (I did this with sdg1 and sdh1, too)
6) at the end, I resized /dev/md0
Is this procedure correct? Or is there anything to improve?

So this is how it looks now:

[BradN]

I seem to remember there being an option in mdadm to rewrite the raid superblock on a device. Otherwise if that doesn't work, you can completely shut down the raid and --create it again (but make sure to get the device order correct as to the order it lists out in --detail, or your data will be scrambled).

In fact one thing that might be good to test now rather than later is whether after --growing a RAID, can it be properly recreated from the devices (or has some kind of formatting changed to make that impossible or strange)?

But your method looks sound - after you resize the underlying raid, you will want to resize the filesystem residing on it (or if you use LVM, expand your physical volume and logical volume, then the filesystem)

[Jimini]

[BradN]

I think -a is different than --assemble, but I don't have a machine with mdadm here to test with at the moment.

Is there any reason you're trying to run --assemble twice? If you only get that error trying to assemble a raid that's already running, I wouldn't worry about it.

As far as I know, I've never used /etc/mdadm.conf and never had a reason to - my raids are being assembled when the kernel starts up before init scripts even begin to run.

As to the partitions, it's necessary to use partitions (with the correct partition ID, FE if I remember) if you want the raids autoassembled by the kernel, in addition all the partitions should have superblocks.

[Jimini]

[Jimini]

Ok ok, I could't wait and re-created the whole array now.

1) formatted all partitions and re-created the array, everything was fine
2) Reboot and same error as above occured (no superblock on /dev/sde1)
3) stopped the array
4) assembled the array: mdadm: /dev/md0 has been started with 4 drives.
=> Summary so far: the raid works fine. At boot, I get the error that sde1 has no superblock. If I stop and then assemble the array, I do not get this error, only if I assemble the array again.

Best regards,
Jimini
_________________
"The most merciful thing in the world, I think, is the inability of the human mind to correlate all its contents." (H.P. Lovecraft: The Call of Cthulhu)

[BradN]

Hmm, you wouldn't have had to format the individual partitions, simply setting the partition type and putting them in a raid with mdadm should suffice.

Maybe the kernel is starting your RAID at boot time (it should do this if the partition types are set), and then your init scripts are trying to start it again (maybe that's causing the error) - is your raid running correctly when you first log in and check with mdadm --detail? Also you could check dmesg output and see if the kernel was assembling your raid at boot time.

[Jimini]

[BradN]

The kernel itself can assemble raid devices during its startup (this happens before init scripts even run, and requires the partition types to be set, and is useful for root on raid without an initrd), or your boot scripts can do it after the kernel's finished starting (which is where you were seeing the error).

Honestly I'm surprised the boot scripts aren't smart enough to avoid trying to start an already-started raid.

[Jimini]

Ah, that makes sense.
There is one thing I am wondering about:

[BradN]

Hmm, the empty space usage is probably a problem for you to take up with your filesystem - modern ones will allocate a journal of some size even on an empty filesystem, and often other filesystem structures as well. But, that usage (450MB+) does seem way high for a 6GB partition, at least for the filesystems I've been using (reiserfs, ext3 mostly).

What filesystem are you using on there, and does it behave the same when you format a loopback file with it?

example,
dd if=/dev/zero of=mytest.fs bs=1M count=6000
mk???fs mytest.fs
mount -o loop mytest.fs /mnt/test
df -h
umount /mnt/test
rm mytest.fs

Edit: Just thought of another possibility, some filesystems reserve a certain percentage of space for root, maybe that's automatically taken off the available space summary. I suppose to test real space you might want to fill a file until the disk is full.

cat /dev/zero > spacetest
ls -l spacetest
rm spacetest

[Jimini]

I use ext3, which is mounted as /home.
As you recommended, I filled up the space on the array:

[Genone]

[Jimini]

Great, thanks for the explanation. 300M don't hurt at all, I just wanted to clarify that.

Best regards,
Jimini
_________________
"The most merciful thing in the world, I think, is the inability of the human mind to correlate all its contents." (H.P. Lovecraft: The Call of Cthulhu)