| View previous topic :: View next topic |
| Author |
Message |
deepspace9
Apprentice
Joined: 29 Jan 2003
Posts: 214
Location: Netherlands
|
 Posted: Fri Apr 07, 2006 12:34 pm Post subject: Nforce RAID unstable and bad performance |
 |
|
Since I have my system, it has been highly unstable under high HD IO load. I use the Nfoce 4 raid (raid1 with two 250 GB drives) with dmraid.
The problem is: under high disk load, the system get's unstable, stop responing, and even crashes. This happens for example while unraring large files, or when beagled is indexing.
I was blessed with SCSI on my old system, and it never fails or stalled the system. But SataII should perform not bad either... But this now... horrible... Besides being unstable, its also not very fast. My old system had a 4 disk software RAID 5 array, and it performed far better, without the system going into it's knees if you do some file access.
Here a bit of bonnie++:
| Code: | bonnie++ -s 4096 -r 1024 -u root
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.93c ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
defaint 4G 216 99 50141 23 22155 7 1318 96 39680 8 118.0 1
Latency 141ms 10506ms 547ms 63972us 99337us 607ms
Version 1.93c ------Sequential Create------ --------Random Create--------
defaint -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 12896 70 +++++ +++ 12902 96 11945 74 +++++ +++ 12577 99
Latency 52659us 1420us 17964us 26235us 37us 3288us
|
That's not really great...
So, has anyone the same experience, or something simmilar. I'm thinking of putting a read SATA raid card into my PC. Only there are just a few PCIe solutions available atm.
I also tried several kernels, and raised the Nforce core voltage a bit, but it's all no use...
_________________
Athlon 64 X2 3800+
MSI K8N Neo4 FI
Geil Ultra-X PC3200 400MHz CL2 5-2-2 DDR 2x512 GB
Asus 6600 Silence GFX
Watercooling (I just love the silence  ) |
|
| Back to top |
|
 |
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54391
Location: 56N 3W
|
| Posted: Fri Apr 07, 2006 2:07 pm Post subject: |
|
|
deepspace9,
Do you really have to use dmraid ?
Its BIOS software raid. The only reason for using it is that the raid set needs to be used in Windows too.
A switch to kernel raid will require a reinstall.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
speeddemon
Apprentice
Joined: 27 Sep 2003
Posts: 162
|
| Posted: Fri Apr 07, 2006 3:32 pm Post subject: |
|
|
I would just switch to kernel raid if I were you. I just finished my install using kernel raid on 2 SATA WD caviar drives, and haven't had the least problem yet. I even installed Gentoo using the Ubuntu livecd and didn't have any complications.
blueshee boot # hdparm -tT /dev/md2
/dev/md2:
Timing cached reads: 3948 MB in 2.00 seconds = 1973.88 MB/sec
Timing buffered disk reads: 344 MB in 3.00 seconds = 114.66 MB/sec |
|
| Back to top |
|
|
deepspace9
Apprentice
Joined: 29 Jan 2003
Posts: 214
Location: Netherlands
|
| Posted: Fri Apr 07, 2006 5:49 pm Post subject: |
|
|
Yes, but this BIOS raid stuff is nothing more that a fancy charade for software raid, and as far as I know, the linux kernel will just use software raid code with dmraid. Or am I mistaken?
A reinstall is out of the question, but there might be some other means to pull it off. I have only used 36 GB of my 250 G yet, so I could just back up the shit to another system over nfs, and simply redo the disk stuff.. But I'm still not sure if it will help.. is dmraid really that bad?
_________________
Athlon 64 X2 3800+
MSI K8N Neo4 FI
Geil Ultra-X PC3200 400MHz CL2 5-2-2 DDR 2x512 GB
Asus 6600 Silence GFX
Watercooling (I just love the silence ) |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54391
Location: 56N 3W
|
| Posted: Fri Apr 07, 2006 5:57 pm Post subject: |
|
|
deepspace9,
The dmraid kernel code talks to the BIOS RAID controller to actually get things done. Its nothing like kernel software raid, except in intent. The Kernel software raid code is not used and the format of the data on the drives is dependent on the BIOS code.
I wouldn't say its bad, its just less mature.
Edit =======
Since you have raid1, you may be able to migrate to kernel raid. Take once drive out of the BIOS raid set and use it to create a degraded kernel raid set. Copy the data ovet, when you are happy the kernel raid works in degrated mode, remove the second drive from the BIOS raid, partition it and raid hot add it to your degraded kernel raid.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
deepspace9
Apprentice
Joined: 29 Jan 2003
Posts: 214
Location: Netherlands
|
| Posted: Fri Apr 07, 2006 6:02 pm Post subject: |
|
|
| NeddySeagoon wrote: | | I wouldn't say its bad, its just less mature. |
Well, if all this causes my instabilities and bad performance, it is bad (and less mature) 
I'll try to switch.. I hope I can pull it off somehow.. It appears that I have two SATA chips on my board, one SATAII of the Nforce4 ultra, and one Silicon Image SATA chip (Sil3114) . Both four channels, but atm, with nvraid they both show up as Nvidia devices:
| Code: | 0000:00:07.0 RAID bus controller: nVidia Corporation CK804 Serial ATA Controller (rev f3)
0000:00:08.0 RAID bus controller: nVidia Corporation CK804 Serial ATA Controller (rev f3)
|
Next question is then: Use the Sil of the Nforce?
Actually I like my option of backuping data better. If this bios does something funny I still have everything
_________________
Athlon 64 X2 3800+
MSI K8N Neo4 FI
Geil Ultra-X PC3200 400MHz CL2 5-2-2 DDR 2x512 GB
Asus 6600 Silence GFX
Watercooling (I just love the silence ) |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54391
Location: 56N 3W
|
| Posted: Fri Apr 07, 2006 6:09 pm Post subject: |
|
|
deepspace9,
I have a SIL 3112 on an nForce2 board and use it for kernel raid. Be aware that this chip and the 3114 I think, have data rate issues with some drives. That is, the kernel runs a blacklist of SATA controllers and SATA chips that have issues and uses it to turn some speed features off to keep your data secure.
To see if you are affected | Code: | | grep -R <drive_part_no> /usr/src/linux/* |
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
deepspace9
Apprentice
Joined: 29 Jan 2003
Posts: 214
Location: Netherlands
|
| Posted: Fri Apr 07, 2006 6:17 pm Post subject: |
|
|
You mean like the model number, as found in cat /proc/scsi/scsi?
Well, this does not show any result, so I guess I'm fine.
Well, I'm up to packing up now.. wish me luck... I guess, I'll create a new livecd right now, just in case the old one does not work anymore
Thanks for all the help so far
Damn it, now my (backup) server appears to crash if I transfer large amounts of data via nfs... Crap stuff! The big problem here is: I can't backup the shit because the raid stuf is so crappy 
I might have some more luck with the livecd maybee...
_________________
Athlon 64 X2 3800+
MSI K8N Neo4 FI
Geil Ultra-X PC3200 400MHz CL2 5-2-2 DDR 2x512 GB
Asus 6600 Silence GFX
Watercooling (I just love the silence ) |
|
| Back to top |
|
|
deepspace9
Apprentice
Joined: 29 Jan 2003
Posts: 214
Location: Netherlands
|
| Posted: Sat Apr 08, 2006 2:12 pm Post subject: |
|
|
Well, I did it. At least most of it. tared the disk to a nfs drive, killed the nvraid, repartitioned with mdadm, and untarred the old data back on the fresh raid partitions, did grub-install, and rebooted.
And voila... Instant succes!
I hope it will remain stable for now. I'll let you know. At least the peformance is much better already. hdparam can get to 132 MB/sec on my new raid 0 root  . I'll post some new benches on my rood and raid 1 home soom.
Thanks again for all the help!
Edit:
Benchmarks of my raid 0 /:
| Code: |
Version 1.93c ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
defaint 4G 208 99 96079 39 43465 14 996 97 113001 24 259.6 3
Latency 64139us 2073ms 210ms 60825us 109ms 419ms
Version 1.93c ------Sequential Create------ --------Random Create--------
defaint -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 13873 76 +++++ +++ 11131 82 13337 74 +++++ +++ 11031 85
Latency 9220us 1255us 1327us 468us 109us 821us
|
Benchmarks of my raid 1 /home:
| Code: |
Version 1.93c ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
defaint 4G 216 98 40247 16 23666 7 1114 99 55742 11 257.6 5
Latency 88712us 7030ms 1070ms 18846us 103ms 168ms
Version 1.93c ------Sequential Create------ --------Random Create--------
defaint -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 10841 70 +++++ +++ 15283 88 13611 70 +++++ +++ 14247 90
Latency 161ms 1266us 3054us 1563us 119us 22567us
|
This is almost like old SCSI times. I can live with this if it remains stable 
_________________
Athlon 64 X2 3800+
MSI K8N Neo4 FI
Geil Ultra-X PC3200 400MHz CL2 5-2-2 DDR 2x512 GB
Asus 6600 Silence GFX
Watercooling (I just love the silence ) |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54391
Location: 56N 3W
|
| Posted: Sat Apr 08, 2006 2:55 pm Post subject: |
|
|
deepspace9,
It sounds like you are close to the head/platter data rate limit, which is the bottleneck in modern systesms.
Thats as good as it gets.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
deepspace9
Apprentice
Joined: 29 Jan 2003
Posts: 214
Location: Netherlands
|
| Posted: Sat Apr 08, 2006 2:57 pm Post subject: |
|
|
Yes I know And for now, it's still stable..
_________________
Athlon 64 X2 3800+
MSI K8N Neo4 FI
Geil Ultra-X PC3200 400MHz CL2 5-2-2 DDR 2x512 GB
Asus 6600 Silence GFX
Watercooling (I just love the silence ) |
|
| Back to top |
|
|
longship
Apprentice
Joined: 04 Jun 2005
Posts: 294
Location: Ontario, CA USA
|
| Posted: Sun Apr 09, 2006 4:35 am Post subject: |
|
|
| deepspace9 wrote: | | NeddySeagoon wrote: | | I wouldn't say its bad, its just less mature. |
Well, if all this causes my instabilities and bad performance, it is bad (and less mature)
I'll try to switch.. I hope I can pull it off somehow.. It appears that I have two SATA chips on my board, one SATAII of the Nforce4 ultra, and one Silicon Image SATA chip (Sil3114) . Both four channels, but atm, with nvraid they both show up as Nvidia devices:
| Code: | 0000:00:07.0 RAID bus controller: nVidia Corporation CK804 Serial ATA Controller (rev f3)
0000:00:08.0 RAID bus controller: nVidia Corporation CK804 Serial ATA Controller (rev f3)
|
Next question is then: Use the Sil of the Nforce?
Actually I like my option of backuping data better. If this bios does something funny I still have everything |
I think that you'll find that the Silicon Image controller is tied to the BIOS RAID quite closely. You may have to configure it as JBOD in the BIOS to use kernel RAID with it.
I have similar MoBo (ASUS A8N-SLi Prem). I have four drives configured as two RAID arrays using the kernel. Right now I have them all on the nForce controller but I keep promsing myself that I'm going to move two drives to the Silicon Image controller.
If you play around with the second controller I'd like to hear about it.
BTW, just a tip. Do yourself a favor and check out mdadm for configuring your kernel RAID. Just my opinion, but it really makes things simple.
_________________
AMD 64 X2 4400+, ASUS A8N-Sli Premium + Lian Li PC-V1000A Plus
AMD Duron 800, ABit KT7E, junk case
VIA C3-Nehemiah, VIA EPIA MII12000 + Scythe e-Otonashi fanless case |
|
| Back to top |
|
|
deepspace9
Apprentice
Joined: 29 Jan 2003
Posts: 214
Location: Netherlands
|
| Posted: Sun Apr 09, 2006 8:11 am Post subject: |
|
|
| longship wrote: | | I have similar MoBo (ASUS A8N-SLi Prem). I have four drives configured as two RAID arrays using the kernel. Right now I have them all on the nForce controller but I keep promsing myself that I'm going to move two drives to the Silicon Image controller. |
Well, I still have everything on the nforce controler right now. Best reason is that the SIL is just a PCI chip on the PCI bus. The Nforce is more intigrated with the chipset itself and has far more bandwith to it's disposal. If can manage 130 MB/sec with two disks, the SIL will be useless with more than two disks (since the PCI bus is already filled then). Another plus for the Nforce: NCQ, but I don't know if the current linux kernel driver can handle it?
| Quote: |
BTW, just a tip. Do yourself a favor and check out mdadm for configuring your kernel RAID. Just my opinion, but it really makes things simple. |
I already did. It's really childsplay
Well, so far the system has been rock solid, even under heavy load, so I'm pleased
_________________
Athlon 64 X2 3800+
MSI K8N Neo4 FI
Geil Ultra-X PC3200 400MHz CL2 5-2-2 DDR 2x512 GB
Asus 6600 Silence GFX
Watercooling (I just love the silence ) |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Gentoo on AMD64
