mdraid max sync speed

eccerr0r · Posted: Sat Sep 14, 2024 2:52 pm Post subject: mdraid max sync speed

Just wondering if anyone knows anything about this:

It appears the default sync_speed_max is 200000 and it has been this way for ever since at least I'm aware of mdraid - I've been using it for probably almost 2 decades by now.

Sync_speed_max is the maximum disk bandwidth it uses to check/rebuild/repair parity blocks on a RAID. It is in KB/sec.

200,000 KB/sec or 200MB/sec still exceeds the hard drive speeds that I have, some even far, far exceeds. which means it will completely eat out all available disk bandwidth. It's not until about now where SSDs can now exceed 200MB/sec. I suspect some HDDs finally can do it now too.

Has Linux always been designed for SSDs in mind or 2024 hard drives? :D

(My fastest sequential read hard drive is around 180MB/sec now... which is still too slow and thus will be fully consumed by default resync speeds. I just wonder why this 200000 was chosen back in the days...)

Also BTW, this isn't a bad thing as resilver is important as your redundancy is unavailable during this time, at least for RAID1/RAID5. But why 200000 and not simply "whatever bandwidth is available..." as when hard disks reach 300MB/sec there's redundancy left on the table...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

druggo · Posted: Mon Sep 23, 2024 10:04 am Post subject:

The sync speed of an array composed of multiple hard drives.
For example, my 8-disk SSD RAID10, if not limited, can reach 2225960 (2.2GB/s), with each disk's speed approximately 480MB/s.
Therefore, in the era of mechanical HDD, the 200MB/s limit still made sense.

By the way, I found the patch that changes this limit:
https://lore.kernel.org/lkml/Pine.LNX.4.33.0112191607110.7629-200000@localhost.localdomain/T/
_________________
AMD FX-8300 / Loongson-2f (lemote-yeeloong)

eccerr0r · Posted: Mon Sep 23, 2024 1:58 pm Post subject:

Well, your SSD RAID clearly can exceed the 200MB/sec limit in the kernel, do you actually bump up the number? Have you had a failure yet?

I'm just saying for my mechanical drives, even back in 2001, 200MB/sec may well be 400MB/sec or even infinite. IIRC in 2001, hard disks were in the 10GB? range, and disk head rates were a few MB/sec. I don't think people had 100 disk RAIDs back then... that would tax that 200MB/sec number?
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

druggo · Posted: Mon Sep 23, 2024 2:32 pm Post subject:

yes, to release the potential ( echo 20000000 > sync_speed_max ), a new raid wait for failure

back in 2001, I had a 40GB IBM glass-based disk support ATA100 , so max speed is 100MB/s .
_________________
AMD FX-8300 / Loongson-2f (lemote-yeeloong)

eccerr0r · Posted: Mon Sep 23, 2024 2:44 pm Post subject:

There was no way those 40GB disks reached 100MB/sec and I think it's more like ATA33. Head rates are at most 20MB/sec or so.

I think it wasn't until 60GB disks where some had ATA66/ATA100. The fastest head rate I've seen on 120GB ATA100s were like 60MB/sec or so.

The fastest ATA100/133 disk I've seen was around 70MB/sec and this was more like 2008 ish... In any case this is still well lower than 200MB/sec.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

druggo · Posted: Mon Sep 23, 2024 3:38 pm Post subject:

You are right, a 40GB disk cannot achieve 100MB/s, as the ATA100 was the interface specification.

According to the article https://www.anandtech.com/show/591 , written in 2000:
40GB IBM deskstar can reach 32MB/s, then a six disk raid may syncing up to 192MB/s

_________________
AMD FX-8300 / Loongson-2f (lemote-yeeloong)

eccerr0r · Posted: Mon Sep 23, 2024 4:45 pm Post subject:

However this sync speed is not 192MB/sec it's still 32MB/sec. This 200MB/sec number appears to be bottlenecking the writes, not reads.

When assembling arrays for the first time which actually does a "recovery," despite the 200MB/sec setting, my test 5-disk RAID5 was clearly reading four disks at 100MB/sec -- 400MB/sec read -- and writing the fifth at 100MB/sec. So it's not attainable for me now (unless I really wanted to do something silly with all of the small SSDs I have...) and definitely not attainable then.

BTW I suspect back then they were for certain not targeting ATA, UDMA or not -- it was U160 or maybe U320 SCSI most likely. But even those disks didn't have head rates this high, and U160 bus itself would be bottlenecked at 160MB/sec anyway (and U320 a bit better but you'd still need to read disks, which eat a huge amount of bandwidth). So what the heck with this 200MB/sec number?!
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

druggo · Posted: Tue Sep 24, 2024 3:09 am Post subject:

Oh, I foget the recovery!
Then only scrubbing can reach that speed at the time sata1.0(1.5G) and fiber channel(2G) could (by asking gpt)
_________________
AMD FX-8300 / Loongson-2f (lemote-yeeloong)

Goverp · Advocate Joined: 07 Mar 2007 Posts: 2185

I wonder if one of a few possibilities apply:
(a) it was meant to be low, to stop sync from impacting the system, but they forgot it was in KB/sec;
(b) it was thought to be so high as never to limit the sync; or
(c) it has some relationship with the technology of the time.

Google found me an hp article that says it's deliberate, out of date, and nobody knows a better value, if I paraphrase correctly.
_________________
Greybeard

eccerr0r · Posted: Tue Sep 24, 2024 3:11 pm Post subject:

Yeah, that's what we were trying to figure out.

Apparently it was decided in 2001-ish and hard drives of that era could NOT hit 200MB/sec. Not even close. This number might well be infinite for drives of that era. SATA probably was just coming out if at all. Ultra/wide SCSI was probably the only thing, as well as some fiber channel. But hard drive spin rate and density could not hit 200MB/sec. I'd say 60MB/sec with some SCA U2W/U3W/FC disks at best in 2001.

But today 200MB/sec has been reached and exceeded with SSDs just 10+ year after this number, but yet RAID never was intended with SSDs. Now 20 years after the choice of 200MB/sec mechanical hard drives seem to have just about started to hit that 200MB/sec mark.

I'd say this 200MB/sec number should have just been infinite, should have just written 9999MB/sec or just use 0xFFFFFFFF or whatnot... It didn't make sense then, and today people using fast disks are being bottlenecked by a 2001 decision.

BTW anyone work with these 200MB/sec+ mechanical hard drives? (as in, not NVMe, not SSD) Having such would indeed be helpful in RAID rebuilds, closing the redundancy hole when a disk fails. My fastest mechanical disk is 180MB/sec or so and is already about 10 years old, but I have a lot of slower disks that will bottleneck reads and having limited number of SATA gen 3 (6Gbps) ports. Technically having just 2 SATA Gen3 ports is fine for RAID1 but I'm all RAID5 here, though at this rate using the SATA Gen2 (3Gbps) ports will be just fine for some time for me...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?