Raid 5 Perfomance issues

[fullerf]

Hi,

i built an linux software raid-5 with 3x300gb samsung sata2 drives attached to
the sata2 ports of a Asus P5B-Deluxe, equipped with a Intel Core2-6300
and 2gb of DDR-800 Memory, running with kernel 2.6.18-gentoo and
mdadm 2.5.2.

The arry contains only one JFS volume, created with mkfs.jfs (no additional
parameters) out of jfsutils 1.1.8, with a raid stripe size of 128kb and a total size of 559gb.

The volume is mounted and contains folders with my media repository
(about 15k mp3 files ~ 90gb, 1k films ~210gb and obout 5k small files)
as so my home directories, as i mounted “<arry-mountpoint>/home -o bind /home”.

Usage comes from proftpd, serveing about 1-2 concurrent downloads over a 100mbit lan.

The values for creating the arry and the filesystem originated from benchmarking
with iozone and some timed file copying from/to a tmpfs.

In the beginning there was no problem, but (i think so) as the filesystem filled and files
began to defragment due to deletion of files and creating new ones, read processes got
very disk intensive, as the noise level of the hard drives became seriously annoying,
and slower.
The results are occasionally slow writes (about 8mb/s), noisy operation at all and
odd slow reads (about 20mb/s) on the older ,unmoved files.

I also noticed that _if_ files got read fast with about 60mb/s, most of the time
the transfer rate slowly decreases within the read process to a value half the
initial rate.


As of now im looking for methods to “fix” the filesystem, like some sort
of defragmentation or general reordering of files for a more efficent acces.
Or a proof that this is a known missbehavior of some of the involved
software (kernel, mdadm) or the hardware drivers, which is going to be
fixed recently.

If this is no possible or this is the “normal” behavior in this usage scenario,
im going to try wiht recreating the fs using another filesystem (ext3 or xfs), with the
expense of downtime and a need for a backup space.

Before going this far, i ask u if there is any expirience with similar cases,


Thanks in advance,
Felix

[gd96]

well first off with only 3 drives raid 5 wont be the speediest system out there, the more drive you put in the faster it will become. The other thing I noticed is that your stripe size is huge, I would make it 64 or even 32 in size. 128 is way too big. from my understanding your using ext3, have you tried reiser3 or 4? Also you dont want to use XFS unless you have a good UPS. On myserver I have 3x320 drives in software raid 5, and I'm pretty happy with the performance using reiser3 and 64k chunks.

If I do a hdparm -t on my raid I get back Timing buffered disk reads: 262 MB in 3.01 seconds = 86.97 MB/sec

there might be faster out there, but Itsfine for my needs.

[bexamous]

I've got a 4x250 reiserfs array and a 3x250 xfs array, I'm kinda sick of reiserfs on large arrays. XFS is much better suited for the tast. And as for the comment about needing a UPS, that was only an issue with some old version of XFS... at least that is my understanding. Also one neato part about XFS is:

NAME
xfs_fsr - filesystem reorganizer for XFS

SYNOPSIS
xfs_fsr [-v] [-t seconds] [-f leftoff] [-m mtab]
xfs_fsr [-v] [xfsdev | file] ...

DESCRIPTION
xfs_fsr is applicable only to XFS filesystems.

xfs_fsr improves the organization of mounted filesystems. The reorganization algorithm
operates on one file at a time, compacting or otherwise improving the layout of the file
extents (contiguous blocks of file data).

[fullerf]

Ok, as i considered upgrading to 4 devices anyways this will be on an additional benefit when recreating the arry.
Do i have to create the arry with one additional device to be able to add a drive later? Or may i "expand" the the
arry afterwards?

I heard a lot about problems with reiser3 and especially v4 on raid, furthermore my belives in reiserfs as a reliable
and stabel fs are not that big.
Ext3 seems to be the mostly tested and "grown" fs for linux in my eyes, how do you think about this?
But the concerns about going with xfs remain, despite it has this realy nice sounding defrag util.
Is there anything similar for ext3? Is there a need for it?

In most of the benchmark articles i read, xfs was niether the top performing nor otherwise preferred fs,
is this grounded on the kind of articles or can i take it for good, that it has no great weakness?

Thanks for your reply
Felix

[bexamous]

Ext3 is what everything else is compared against, its a very solid fs. Read over https://forums.gentoo.org/viewtopic-t-305871.html and you'll probabaly become a ext3 supporter heh. Between the journaling options and the recovery tools it is probably one of the safest file systems and its performance is not really that bad imo. You can just search the forum for many file system discussions.

XFS's only weakness anymore is its slow at deleting files. Theres actually some topic talking about how to speed it up but anyways. Its not great with small files, then again its not bad either, but it was really designed for large arrays with large files.

You can look up EVMS if you plan on expanding your array it supports expanding arrays pretty well. I've done it a few times while testing, last time I tried it though (first time using the GUI) it locked the system up.. array was still fine but scary experience.

I don't think there is an actual ext3 defrag utility, theres some defrag script somewhere that basically just copies files

#!/bin/sh
# defrag v0.06 by Con Kolivas <kernel@kolivas.org
# Braindead fs-agnostic defrag to rewrite files in order largest to smallest
# Run this in the directory you want all the files and subdirectories to be
# reordered. It will only affect one partition.

Might be a newer version somewhere, think I found this on gentoo foruns.

[bollucks]

One thing that isn't mentioned often enough is that all filesystems perform badly if more than 75% full. Make sure you always use them below this capacity.

[Janne Pikkarainen]

[fullerf]

Of course i dont want to leave my arry at 80% usage of space
So XFS is the one to go, ok?