Performance problems with mdadm (Raid6) + Crypto Luks + LVM2

[rralf]

Hi!

I have terrible performance problems with my hard disk setup.

This is my Configuration:
4 x 3TB Sata hard drives (/dev/sd[abcd]) formatted with a GPT Partition Table.

/dev/sd[abcd]1 : 100 MiB Boot Drive (RAID1 without encryption/LVM) <- works fine
/dev/sd[abcd]2: 2.73TiB Data Storage (Raid6) <- problem child

According to gparted, both partitions are aligned.

This is what I did for formatting the data storage:

Raid Setup:
mdadm --create --verbose /dev/md0 --level=6 --raid-devices=4 /dev/sd[abcd]2
(15 hours of initialization... )
Speed of initialization: ~30-50Mib/s <- In my opinion, even this is slow
If I do "dd if=/dev/zero of=/dev/sda2 bs=512K count=1024" I have round about 100MB/s write speed.
Well...


Crypto Setup:
cryptsetup luksFormat --cipher aes-cbc-essiv:sha256 --key-size 256 /dev/md0
cryptsetup luksOpen /dev/md0 Storage

LVM2 Setup:
According to /proc/mdstat, chunk size of my Raid6 is 512KiB

pvcreate --dataalignment 512K -M2 /dev/mapper/Storage
vgcreate vg /dev/mapper/Storage
lvcreate -L100G -nStore1 vg
mkfs.ext4 -j -LStore1 /dev/vg/Store1
mount /dev/vg/Store1 /mnt/foobar
Everything works fine except of the performance

Read speed is about ~100MiB/s. That's okay for me.
But write speed is ~20MiB/s, that's really bad.

What am I doing wrong?
I'm sure write speed could be faster than 20Mib/s....

Thanks for help!
Best regards,

Ralf

[chithanh]

Do you have the same problem when you create the filesystem directly on /dev/md0 (instead of using lvm)?

[rralf]

No, I tried it out.

When i directly create a filesystem on /dev/md0, performance is fine.

[chithanh]

So given that read performance is fine and write performance without LVM is fine too, the problem looks like an LVM alignment issue.

[rralf]

Yes, probably in combination with cryptsetup, but i don't know what I'm doing wrong...

I tried to align everything correctly...

[chithanh]

Ah sorry, I misread what you wrote above.

What if you create the filesystem on /dev/mapper/Storage? How is the performance then?
EDIT: cryptsetup does have an --align-payload option, did you try that too?

[rralf]

Ok, i read some good articles:

https://support.mayfirst.org/wiki/disk_alignment
http://wiki.drewhess.com/wiki/Creating_an_encrypted_filesystem_on_a_partition
http://blog.kanojo.de/2011/10/11/raid-system-8tb-home-storage-on-budget/

After tweaking and trying i got it a bit better:

echo 32768 > /sys/block/md127/md/stripe_cache_size

This brought an amazing speedup!

Read speed: ~200MB/s, tested with:
dd if=/dev/md127 of=/dev/null bs=512K

Write speed: ~120MB/s
dd if=/dev/zero of=/dev/md127 bs=512K

That's quite good.

Cryptsetup tweaking
cryptsetup luksFormat --argsfoobar --align-payload 4096 /dev/md127
cryptsetup luksOpen /dev/md127 vault
...

Crypto Read speed: ~100MB/s, tested with:
dd if=/dev/mapper/vault of=/dev/null bs=512K

Crypto Write speed: ~90MB/s
dd if=/dev/zero of=/dev/mapper/vault bs=512K

That's not the truth... Read speed is half than before, write speed decreased also by 30MB/s....
But I suppose, that this is the crypto overhead.

LVM Creation:
pvcreate --metadatasize 2048K --dataalignment 4096 -M2 /dev/mapper/vault
vgcreate vg /dev/mapper/vault
lvcreate -nGentoo -L30G vg

LVM Read speed: ~100MB/s, tested with:
dd if=/dev/vg/Gentoo of=/dev/null bs=512K

LVM Write speed: ~90MB/s
dd if=/dev/zero of=/dev/vg/Gentoo bs=512K

So speed doesn't decrease when using LVM. (This is an important information for me....)

After creating the ext4fs using
mkfs.ext4 -j -LGentoo -b 4096 -E stride=128,stripe-width=256 /dev/vg/Gentoo
mount /dev/vg/Gentoo /mnt/gentoo

Ext4 Read speed: ~100MB/s, calculated with:
sync;date;cp /mnt/gentoo/3gbfile /mnt/ramfs/foobar;sync;date

Ext4 Write speed: ~85MB/s, calculated with:
sync;date;cp /mnt/ramfs/3gbfile /mnt/gentoo/3gbfile;sync;date

So the bottleneck seems to be the cryptolayer....

Anyone further ideas?

Thanks!

[chithanh]

What is the read/write speed if you create a luks volume on a single disk?

Also if your CPU does not support AES acceleration, better use 128-bit encryption (aes-xts-plain with 256 bits, half of which are the AES key) and not 256-bit (aes-xts-plain 512).

[rralf]

R/W on a single disk: ~70MB/s

Urgh, according to /proc/cpuinfo my CPU doesn't support AES instruction set.

Damn....

[chithanh]