LSI MegaRaid 150-6 SATA performance

[Nossie]

I just installed a LSI MegaRaid 150-6 SATA controller and was (negatively) surprised by the performance. I set it up with two Maxtor 160 GB SATA drives in a raid1 array. Read performance was 30 Mb/sec. but writes were about 4 mb/sec. with a very high system load.

After a couple of hours messing with the cards bios(es) and going through the competely incomperhensible and almost totaly useless manual, i found a bios setting to enable the write cache of eache individual drive.
For some reason the default setting for the drive cache is 'disabled'.

With the drives write cache enabled, the write performance was 20 Mb/sec.
The card has 2 bios interfaces, a text based bios and a 'web' based bios. The drive write cache can only be enabled or disabled in the text bios.

Gentoo instalation on the array was easy, just insert the livecd and boot the 2.4.x kernel with the doscsi parameter (2.6.x kernel didn't work for me).

Now I have a raid10 array of 6 drives. Write performance is a little lower than i expected (20 Mb/sec.), but read performance is very good (90 Mb/sec.).

I'll post some benchmarks later...

gr,
Nossie

[Nossie]

In the above post I wrote that I would post some benchmarks later.... well here the are...

This is a raid10 array of 6 Maxtor 160 Gb SATA disks.
The filesystem is reiserfs, and the write policy on the raid controller is set to 'write back'

[the_highhat]

cust srvc @ LSI recommended "write thru" instead of "write back" for RAID !0 arrays. I benchmarked this with h2bench and it made a signficant difference (+30%) on writes alone. I'm not sure of the effect of caching at the drive level with these card though. also, use direct i/o, and use normal reads (no readahead or adapative readahead) for optimal performance


btw, i too use raid10, but many posts seem to think raid 5 + hotspare is the way to go. i don't really get it since raid 10 can survive up to 2 disk failures w/o performance loss and it generally faster than raid 5. sure your loose some capacity, but it seems worth it to me.... why did you go raid 10?

[Nossie]

I choose raid10 because raid10 can can survive multiple disk failures. With a raid10 array of 6 drives, you have a stripe of 3 mirror sets, if 1 drive of each mirror fails, you are still up and running without any data loss.

The second reason i choose raid10 is that when a drive gets flaky (i.e. the drive fails partially and gives corrupted data) you can (if you find the responsible drive) replace the drive, and it gets rebuild without problems.
If this happens with raid5, the parity data will also be corrupted (checksum is created from corrupted data). So if you want to replace the flaky drive, the data will be rebuild with the corrupted parity data, and you will have a rebuild drive that's still corrupted.

I have tried both 'write-through' and 'write-back', and there is a performance difference between the 2 modes. For me write-back was 30% faster than write-through (20Mb/sec vs 26Mb/sec on sequential writes).

cat /proc/megaraid/hba0/raiddrives-0-9

[the_highhat]

Setup:
lLSI Megaraid 150-4 on 64bit 66mhz slot
2xopteron 240 on K8W mobo w/2GB NUMA
4x36GB raptors in RAID 10
32K chunk, normal read, direct I/O
gentoo 2004.1, gcc 3.4 + multilib, glibc 2.3.4 + nptl,
kernel 2.6.7-gentoo-r9
....................................

tiobench is not available for amd64, so using bonnie++ to test.
command issued was "bonnie -u root"



....................................
With elevator=cfq and caching set to "write-back":

Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
programatron 4G 40734 98 43337 17 17510 6 29664 75 50559 10 595.8 1
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 14795 98 +++++ +++ 14923 99 15774 98 +++++ +++ 14564 99
programatron,4G,40734,98,43337,17,17510,6,29664,75,50559,10,595.8,1,16,14795,98,+++++,+++,14923,99,15774,98,+++++,+++,14564,99


....................................
With elevator=cfq and caching set to "write-thru":

Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
programatron 4G 38440 93 62204 31 29313 11 31496 80 83220 18 497.6 1
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 15060 99 +++++ +++ 12539 99 14986 99 +++++ +++ 12250 99
programatron,4G,38440,93,62204,31,29313,11,31496,80,83220,18,497.6,1,16,15060,99,+++++,+++,12539,99,14986,99,+++++,+++,12250,99


....................................
with elevator=as (default) and caching set to "write-thru"

Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
programatron 4G 40333 98 78780 38 28052 10 31082 79 82366 18 398.2 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 15365 98 +++++ +++ 14644 99 15922 99 +++++ +++ 12977 99
programatron,4G,40333,98,78780,38,28052,10,31082,79,82366,18,398.2,0,16,15365,98,+++++,+++,14644,99,15922,99,+++++,+++,12977,99



........................................

can you post a bonnie++ output for reference? thnx

[Nossie]

My bonnie++ results

LSI MegaRaid 150-6 in a 32bit PCI slot
kernel 2.6.7-mm4
FS = reiserfs (with tails)