raid0 vs single drive transfer speeds, no improvement

[awalp]

I'm wondering what is the reason for no improvement with transfer speeds for a raid0 vs single drives
and if it is the CPU/memory/motherboard limiting it.

-- I have two raid0 arrays, the first using older seemingly slower harddrives showing a nice improvement with raid0 over single drives
-- the second using new harddrives which seemingly are faster and are show no improvement or even sometimes a minor loss with raid0 over single drives

detailed specs: Athlon XP 2500+, 512mb ram, PCI 4x SATA add-on card.

MD0 sda,sdb(250gb x2 = 500GB) XFS default chunk size of 64 (matching drives 2 years old) 7200rpm ST3250823AS
MD1 sdc,sdd(750gb x2 = 1.5TB) XFS default chunk size of 64 (matching drives 2 weeks old) 7200rpm WD7500AVDS-63U8B0

[frostschutz]

PCI is the slowest interface by far nowadays... that's why graphics card used to use AGP, and why we have PCI-E now

[krinn]

- Check your sata controller is plug in a bus master capable pci (you can see that with lspci -vv). Some motherboard were having a burst pci (this can only be check in the manual, generally lower latency on that bus and no irq sharing for that one)
- Your md1 is with green WD disks, look here -> https://forums.gentoo.org/viewtopic-t-836411-start-0-postdays-0-postorder-asc-highlight-green.html
- PCI can gave 133MB (266 for pci 2.1) max bandwith, a SATA1 drive can get 150MB, so don't expect the drive to work at full speed with early pci, and with newest the raid cannot run at full speed
- Not always a good idea to pickup a complex filesystem, it should deliver high performance but also should put more pressure on the cpu, not really a good choice for a weak cpu that try to handle software raid array.
Complexity nearly always mean higher cpu usage, even with nice code.
- usb is by far the slowest interface, usb1 run at 12Mb = 1.5MB (lol), usb2 is better at 480Mb, but that's still weak with 60MB, it's easy to lie to users with Mb vs MB, to compare PCI in Mb = 32*33 = 1056Mb

[awalp]

[dmpogo]

It does seem that you hit the limit of ~80-85 Mb/sec, which is perhaps your PCI card

[cyrillic]

Don't forget, the PCI bandwidth is shared with other PCI devices such as your network card(s), and not just the harddrive controller(s).

[awalp]

[Mousee]

[jathlon]

The last time I upgraded a couple of hard drives I took a look at the Caviar Green line of discs. After reading a couple of reviews I spent the extra money and went with the Caviar Black. From an article on The Tech Report about the 1Gb Caviar Green.

[krinn]

You really think XFS is less complex than ext3 ?
I have a pentium4 (i think it's a C, anyway one with 1 core+ HT), it run so slow compare to my system that i wish sometimes to thru it out of the windows (hopefully it's in the garage, and no window there).

And because it's software raid, you can easy check that

[awalp]

[dmpogo]

[dmpogo]

[Mousee]

[awalp]

The problem was chunk sizes.

I will post up some numbers when I'm done transferring all the files back onto the drive.

the solutions was
recreating the mdadm array with proper chunk sizes & formatting XFS with specific variables to properly align to it

Create the Array with a chunk size of 1024k instead the default 64k chunk

[zeek]

[awalp]

hdparm -tT benchmarks with arrays mounted (I get much higher numbers unmounted)


fileserver:~# hdparm -tT /dev/hda /dev/sda /dev/sdb /dev/md0 /dev/sdc /dev/sdd /dev/md1

/dev/hda:
Timing cached reads: 472 MB in 2.00 seconds = 235.79 MB/sec
Timing buffered disk reads: 80 MB in 3.01 seconds = 26.57 MB/sec

------------------------ MD0, ~65MB to 85MB/sec gain,-------------------
/dev/sda:
Timing cached reads: 484 MB in 2.01 seconds = 241.07 MB/sec
Timing buffered disk reads: 196 MB in 3.03 seconds = 64.74 MB/sec

/dev/sdb:
Timing cached reads: 474 MB in 2.00 seconds = 236.67 MB/sec
Timing buffered disk reads: 202 MB in 3.02 seconds = 66.82 MB/sec

/dev/md0:
Timing cached reads: 474 MB in 2.00 seconds = 236.99 MB/sec
Timing buffered disk reads: 256 MB in 3.01 seconds = 84.91 MB/sec
-------------------------------------------------------------------------------------



------------------------ MD1, ~78MB to 85MB/sec gain,-------------------
/dev/sdc:
Timing cached reads: 466 MB in 2.00 seconds = 232.89 MB/sec
Timing buffered disk reads: 234 MB in 3.01 seconds = 77.87 MB/sec

/dev/sdd:
Timing cached reads: 470 MB in 2.01 seconds = 234.32 MB/sec
Timing buffered disk reads: 236 MB in 3.02 seconds = 78.26 MB/sec

/dev/md1:
Timing cached reads: 472 MB in 2.00 seconds = 235.51 MB/sec
Timing buffered disk reads: 258 MB in 3.03 seconds = 85.17 MB/sec
---------------------------------------------------------------------------------

that particular run was on the low end, usually it averages about 5MB/sec higher on MD1 (~82MB/sec to 95MB/sec)

there may be something with 85MB/sec being a limit
both arrays seem to limit themselves to that when they are mounted.
unmounted md1 array reaches 115MB/sec

I'm gonna guess that mounted vs unmounted difference is a bandwidth issue somewhere.


-------------- With Filesystems NOT mounted / unmounted ---------------
------------------------ MD1, ~65MB to 105MB/sec gain,-------------------
/dev/sda:
Timing cached reads: 452 MB in 2.00 seconds = 225.50 MB/sec
Timing buffered disk reads: 198 MB in 3.03 seconds = 65.40 MB/sec

/dev/sdb:
Timing cached reads: 472 MB in 2.01 seconds = 235.36 MB/sec
Timing buffered disk reads: 196 MB in 3.00 seconds = 65.31 MB/sec

/dev/md0:
Timing cached reads: 450 MB in 2.01 seconds = 224.37 MB/sec
Timing buffered disk reads: 320 MB in 3.00 seconds = 106.56 MB/sec


------------------------ MD1, ~78MB to 110MB/sec gain,-------------------
/dev/sdc:
Timing cached reads: 464 MB in 2.01 seconds = 231.07 MB/sec
Timing buffered disk reads: 236 MB in 3.01 seconds = 78.49 MB/sec

/dev/sdd:
Timing cached reads: 454 MB in 2.00 seconds = 226.92 MB/sec
Timing buffered disk reads: 234 MB in 3.01 seconds = 77.71 MB/sec

/dev/md1:
Timing cached reads: 468 MB in 2.00 seconds = 233.79 MB/sec
Timing buffered disk reads: 332 MB in 3.01 seconds = 110.31 MB/sec
------------------------------------------------------------------------------------



There is a huge difference between mounted and unmounted numbers, whether that is just a bandwidth limitation of the hardware or if I can manage to pull those numbers by additionally tweaking the system. I have no idea.

I'll look into bonnie++ for some better numbers soon.

[awalp]

also, could this error be causing a performance loss?

[ 1632.756007] Filesystem "md0": Disabling barriers, not supported by the underlying device
[ 1632.835622] XFS mounting filesystem md0
[ 1632.952003] Ending clean XFS mount for filesystem: md0

Would enabling the feature "Disabling barriers" change performance?

[krinn]

or just use dd

write 2gb
dd if=/dev/zero of=/tmp/ddtest bs=8k count=250k
256000+0 enregistrements lus
256000+0 enregistrements écrits
2097152000 octets (2,1 GB) copiés, 12,3672 s, 170 MB/s

read
dd if=/tmp/ddtest of=/dev/null bs=8k count=250k
256000+0 enregistrements lus
256000+0 enregistrements écrits
2097152000 octets (2,1 GB) copiés, 3,87711 s, 541 MB/s

[shamen]

I registered just to reply to this as I saw exactly the same thing on my NAS; specs:

P4 1.8Ghz
Sil3512 SATA card (2 port)
512Mb ram
Asus P4s333 mobo
Software raid 1
Disk 1: Maxtor 1Tb 7200 SATA
Disk 2: WD Caviar Green 1Tb SATA

After thinking the issue could be the WD, I removed the WD caviar from the array so I was running on one disk. Just wanted to make sure I was not being pulled down by the 'greenness'

The first thing I looked into was PCI bus transfer - thats a max of 133MB/s (1Gbps) on the whole bus. I have disabled every internal device - Sound, NIC, LAN, second IDE etc in the BIOS, so theres just this SATA card on the pci bus and my AGP graphics. I was seeing Timed cached reads of about 400MB/sec (though strangely, only about 200MB/sec for the first 12 hours after being rebooted?!) and timed buffered disk reads of about 20MB/sec - very low for SATA buffered reads. 0_DIRECT cached reads were about 65MB/sec and 0_DIRECT disk reads were about 95MB/sec :