| View previous topic :: View next topic |
| Author |
Message |
thechris
Veteran
Joined: 12 Oct 2003
Posts: 1203
|
 Posted: Sun Dec 03, 2006 12:04 am Post subject: Speed of Kernel RAID5 |
 |
|
I just finished setting up a software, kernel raid 5 setup with XFS as the filesystem (i've heard that ext3/reisefs were unstable for this application).
I found that i could download the stage3 tarball (~100MB) from the internet at about 300kB/s. I then tried to copy the file to my RAID5 array. it took a couple minutes and the speed came out to about 600kB/s.
This means that SW RAID5 isn't just slow, its embaressingly slow.
Is there any way to get decent write performance from SW raid5, or is it really just a proof of concept thing -- not useful for anything but showing that it can function?
_________________
HW problems. It's a VIA thing. |
|
| Back to top |
|
 |
JRV
Apprentice
Joined: 10 Jan 2004
Posts: 291
|
| Posted: Sun Dec 03, 2006 2:20 am Post subject: Re: Speed of Kernel RAID5 |
|
|
| thechris wrote: | | I just finished setting up a software, kernel raid 5 setup with XFS as the filesystem (i've heard that ext3/reisefs were unstable for this application). |
Where does it say that? The RAID stability should not really be affected by the filesystem used ontop of it.
| Quote: | | Is there any way to get decent write performance from SW raid5, or is it really just a proof of concept thing -- not useful for anything but showing that it can function? |
I'm using a software RAID 5 with ext3 and three SATA disks (each 250 GB, RAID total = 500 GB). I'm getting around 25MB/s when copying a ~900MB file from one place in the RAID to another.
Are you sure DMA is enabled on all of your RAID disks? Also, is your RAID fully built (reconstructed) yet?
JRV |
|
| Back to top |
|
|
Aefron
n00b
Joined: 27 Oct 2006
Posts: 41
|
| Posted: Sun Dec 03, 2006 2:41 am Post subject: |
|
|
I'm very interested in this kinda things and would have a few questions to both thechris and JRV :
@JRV and @thechris : - what interface do yout have for your setup (I mean, PCI-based controllers are inherently slow... I wonder what could practically be obtained with PCI-E SATAII)
- which CPU and how much RAM?
@thechris : - how many hdds do you have (I heard that a lot of software managed RAID5 hdds made the computer calculate so much parity check that it was getting lower and lower along with an ascending number of hdds...) |
|
| Back to top |
|
|
JRV
Apprentice
Joined: 10 Jan 2004
Posts: 291
|
| Posted: Sun Dec 03, 2006 2:45 am Post subject: |
|
|
| Aefron wrote: | @JRV and @thechris : - what interface do yout have for your setup (I mean, PCI-based controllers are inherently slow... I wonder what could practically be obtained with PCI-E SATAII)
- which CPU and how much RAM? |
The three RAID disks are on a SATA-controller card on a PCI-X (not PCI-E!) bus. It's a dual processor Xeon 2.8 GHz system with 512MB RAM.
JRV |
|
| Back to top |
|
|
Aefron
n00b
Joined: 27 Oct 2006
Posts: 41
|
| Posted: Sun Dec 03, 2006 2:51 am Post subject: |
|
|
| Tanks JRV... Well so, it doesn't seem your PCI-X is saturated with usefull transfer... so, have you tried with a fourth disk to see if it does enhance things or is it the overhead of parity checks that limits here? |
|
| Back to top |
|
|
richfish
Apprentice
Joined: 03 Mar 2006
Posts: 202
Location: Phoenix, AZ
|
| Posted: Sun Dec 03, 2006 2:57 am Post subject: |
|
|
| Aefron wrote: |
@thechris : - how many hdds do you have (I heard that a lot of software managed RAID5 hdds made the computer calculate so much parity check that it was getting lower and lower along with an ascending number of hdds...) |
This is an unavoidable fact of RAID5 arrays, whether software or hardware. In order to calculate the parity block, the parity engine needs to know the data for the other disks in the array. So a write to a 3-disk array may require a read from one disk to calculate the parity of that data and the data being written. So you may have 1 read and 2 writes (1 for data, 1 for parity). In a 4-disk array, the controller may need to read from 2 disks...so 2 reads and 2 writes. And so on.
Hardware RAID controllers win in this case because they have large memory caches (usually backed up by battery) so they can do this work in the background, and achieve higher throughputs than software arrays.
And of course, this is a pathalogical case. There are going to be times where the data from the other disk(s) will already be available in RAM, so no IOs are necessary. How often these "cache hits" occur would depend greatly on the application and filesystem layout. |
|
| Back to top |
|
|
richfish
Apprentice
Joined: 03 Mar 2006
Posts: 202
Location: Phoenix, AZ
|
| Posted: Sun Dec 03, 2006 3:00 am Post subject: |
|
|
| JRV wrote: |
The three RAID disks are on a SATA-controller card on a PCI-X (not PCI-E!) bus. It's a dual processor Xeon 2.8 GHz system with 512MB RAM.
JRV |
What does hdparm -Tt /dev/sdX report on each of the raided disks? |
|
| Back to top |
|
|
JRV
Apprentice
Joined: 10 Jan 2004
Posts: 291
|
| Posted: Sun Dec 03, 2006 12:33 pm Post subject: |
|
|
| richfish wrote: | | What does hdparm -Tt /dev/sdX report on each of the raided disks? |
The three separate disks:
| Code: | $ hdparm -Tt /dev/sdd; hdparm -Tt /dev/sde; hdparm -Tt /dev/sdf
/dev/sdd:
Timing cached reads: 3100 MB in 2.00 seconds = 1550.36 MB/sec
Timing buffered disk reads: 188 MB in 3.02 seconds = 62.25 MB/sec
/dev/sde:
Timing cached reads: 2956 MB in 2.00 seconds = 1478.35 MB/sec
Timing buffered disk reads: 176 MB in 3.02 seconds = 58.20 MB/sec
/dev/sdf:
Timing cached reads: 2992 MB in 2.00 seconds = 1496.69 MB/sec
Timing buffered disk reads: 178 MB in 3.03 seconds = 58.79 MB/sec |
The RAID-array md0:
| Code: | $ hdparm -Tt /dev/md0
/dev/md0:
Timing cached reads: 3060 MB in 2.00 seconds = 1530.46 MB/sec
Timing buffered disk reads: 344 MB in 3.01 seconds = 114.31 MB/sec |
|
|
| Back to top |
|
|
JRV
Apprentice
Joined: 10 Jan 2004
Posts: 291
|
| Posted: Sun Dec 03, 2006 12:34 pm Post subject: |
|
|
| Aefron wrote: | | Tanks JRV... Well so, it doesn't seem your PCI-X is saturated with usefull transfer... so, have you tried with a fourth disk to see if it does enhance things or is it the overhead of parity checks that limits here? |
Nope, haven't tried that! |
|
| Back to top |
|
|
richfish
Apprentice
Joined: 03 Mar 2006
Posts: 202
Location: Phoenix, AZ
|
| Posted: Sun Dec 03, 2006 5:58 pm Post subject: |
|
|
You aren't by chance using the "sync" option when mounting the filesystem on the array, are you? That would account for a huge performance drop.
If you don't care about the contents of the filesystem at this point, unmount it, and try writing directly to it with dd. Something like "dd if=/dev/zero of=/dev/md0 bs=1024k count=3000" should give you a nice test of the raw write speed of the array.
[edit]I seem to have gotten confused at who the OP was. I actually meant for thechris to run and report the hdparm tests. |
|
| Back to top |
|
|
thechris
Veteran
Joined: 12 Oct 2003
Posts: 1203
|
| Posted: Sat Dec 09, 2006 4:02 am Post subject: |
|
|
sorry for the delay, my power has been down for the past week or so. I should be getting power back soon. maybe i'll be able to salvage the system.
_________________
HW problems. It's a VIA thing. |
|
| Back to top |
|
|
thechris
Veteran
Joined: 12 Oct 2003
Posts: 1203
|
| Posted: Mon Dec 11, 2006 3:16 am Post subject: |
|
|
still terrible performance. there isn't a sync option, but it comes up as 650kB/s
I think hdparm is reading the read performance. comes up fine.
i'll try the sil raid5 on the card. see if it works and doens't come up with terrible performance.
_________________
HW problems. It's a VIA thing. |
|
| Back to top |
|
|
|
Other Things Gentoo
