Filesystem Speeds

charsent · n00b Joined: 23 Apr 2012 Posts: 7

Hi there,

This is my first post so I'm not totally sure if this is the right location, but it seemed like the best place!

I'm currently investigating filesystem speeds (currently focusing on ext4) across a few Linux Distributions, currently the 3 we have tested on are Ubuntu, Centos and Gentoo (all 64 bit, all the most recent versions as of about 2 weeks ago). I expected CentOS to be quicker than Ubuntu and in turn Gentoo to be the fastest, and our tests have agreed with this. However, the scale of results was quite extreme. On file operations per second and write throughput, CentOS was about 10% faster than Ubuntu and Gentoo was about 80% faster than CentOS.

My question is, does this seem a bit too good? Does it seem more likely a stupid configuration error I've done on Ubuntu / CentOS? I've looked online and can't find anything conclusive!

Any comments are greatly appreciated.

Regards,

Chris

i92guboj · Posted: Mon Apr 23, 2012 11:45 am Post subject: Re: Filesystem Speeds

Veldrin · Posted: Mon Apr 23, 2012 11:45 am Post subject:

So you are saying (just that I got the numbers right), that gentoo is (almost) twice as fast a ubuntu?

A few thoughts about testing:
* how different is the hardware where it has been tested on?
- preferably install the OS on a single disk each, and have one common test disk/partition, which only contains the test partitions
- I assume, that you are running the tests on the same system, otherwise the test will not tell anything.
* how different is the kernel? what options are enabled
- can you rebuild a kernel on each distro using the config from another
- to get some comparable results, use ubuntu lts (10.4), centos 6.x and use gentoo-sources-2.6.32. that way to get at least the same basic kernel for every distro (IIRC all should then be running 2.6.32)
* what kernel version were used?
- only a small version change can change the entire result - to better or worse
* how was the filesytem created? did you use the distros defaults? did you create one generic partitions to run the tests on?
* what kind of software did you install? just a plain base install? default install?
- what happens, if you compare the installed package sets?

OTOH, why would you think - at least in the beginning - that CentOS would come out first?
I would have placed my bet on gentoo, as it can be optimized (and slimmed down) best.

like I said, just some thoughts. I am not sure, if I missed anything. But those numbers look a bit to good to me.
V.
_________________
read the portage output!
If my answer is too concise, ask for an explanation.

i92guboj · Posted: Mon Apr 23, 2012 11:53 am Post subject:

Just to clear the thing out, provided the distros are configured the same way, and you use the same kernel, you should really not get any noticeable difference in this test. I/O is mostly a matter of the hardware these days, so as long as the drivers and the kernel involved are the same, the result should also be very similar. Userland tools shouldn't matter too much at all, that is, assuming you use the same test-case and the same partition or data set to test across all the distros.

The fact that in Gentoo you compile everything using --omg-optimized will have little to no impact on this.

charsent · n00b Joined: 23 Apr 2012 Posts: 7

Thanks for the replies folks! Veldrin, just to confirm, my wording was a bit off but I meant I expected Gentoo to be the fastest of the 3, with CentOS being in the middle of the 3.

The hardware involved in each test is exactly the same (its all on the same machine)

The installations have been as standard as possible:

- Ubuntu 10.04 LTS, minimal install, with the newest version of the ext tools installed from git (we are testing with a 50TB volume which the standard installed version doesn't seem to support creating). Kernel is 2.6.32-24-generic.
- CentOS 6.2, same as Ubuntu. We didn't update the kernel, I believe it was 2.6.32-220, I'm afraid I'll have to reinstall the ISO to check.
- Gentoo 20120323, using the genkernel and the standard version of the ext tools it installs with. Kernel is 3.2.12-gentoo.

Does it seem possible the newest kernel would make this big a difference?

For the other points raised:

- The disk scheduler has been left as "cfq".
- The file systems have been made in the same way with the same switches.

Unfortunately some of the info I am lacking as these were just early tests and we weren't rigorous enough with noting down certain system details. However the input I've been provided is excellent for helping improve the testing going forwards!

I hope my response here might provide a little more information to help, and thanks for the responses so far!

Chris

aCOSwt · Posted: Mon Apr 23, 2012 12:01 pm Post subject:

This experiment you did sounds surprising to me.

At the end of the day... filesystem efficiency is something kernel-only dependent.
Gentoo-Sources is not very far from stock kernel and I do not think Gentoo-patches concern the filesystems and related thingies + schedulers.

However, you can get huge differences from one kernel version to another and of course huge differences depending on what you actually test.
As an example of this, on a 4G RAM system, reading a 4G file is around 250% quicker under a 3.2.1 kernel than under a 2.6.38 one.

On another side, 16 threads competing for writing the same file will complete 25% sooner on & 2.6.38 than under a 3.2.1

So... you see... kernel version is crucial, kernel config is crucial too (do you get control groups enabled or not, type of IO scheduler, timer frequency, type of preemption...) and then finally, what do you test ? Mind you... depending on what you test, you might well be turning into testing more the efficiency of your caches than the efficiency of the filesystem itself.
_________________

charsent · n00b Joined: 23 Apr 2012 Posts: 7

Some additional information. The test we are running is a custom developed application to replicate the standard operations we expect the systems to perform in future. In each case it is using 500 threads writing files of a random size (between 5mb and 20mb) in 32kb blocks. On top of that it is checking if Directories exists, randomly creating them, reserving space before writing, opening and closing files, etc.

i92guboj · Posted: Mon Apr 23, 2012 12:15 pm Post subject:

charsent · n00b Joined: 23 Apr 2012 Posts: 7

Definitely sounds like everyone agrees its likely the kernel!

I'm going to try Ubuntu with the latest kernel and see what happens, thanks for the input, I'll let people know how it goes!

Thanks again folks.

Dont Panic · Posted: Mon Apr 23, 2012 1:46 pm Post subject:

What benchmark are you using?

charsent · n00b Joined: 23 Apr 2012 Posts: 7

We've written our own custom tool that performs a load of operations logging speed/operations per sec and such at 15 second intervals (a little more information in one of my earlier posts). We then compare the results. Its written to emulate the tasks the boxes will be performing in production.

charsent · n00b Joined: 23 Apr 2012 Posts: 7

To correct a mistake, it logs in 15 minute intervals, not seconds! :-)

.

i92guboj · Posted: Mon Apr 23, 2012 2:06 pm Post subject:

Since the test case works in threads, you will also need to make sure that the numbers you are collecting are wide enough to make an statistical analysis. Since each run can be quite different from the previous one (due the threads and cpu and i/o schedulers) you will have to take a big number of samples. But I guess you already took that into account.

srs5694 · Posted: Mon Apr 23, 2012 2:34 pm Post subject:

One more point: You can always compile a kernel with whatever options you want for any distribution. With rare exceptions, you can compile any version of a kernel for use with any distribution, too. Given your use case, that's probably a good idea. This should render the distribution-to-distribution performance differences nil. If you're trying to decide which distribution to use, a kernel recompilation will most likely take performance out of the picture.

charsent · n00b Joined: 23 Apr 2012 Posts: 7

Just finished a very unscientific test on Ubuntu 10.04 running kernel 3.2 and its considerably faster. Its still about 15% slower than the Gentoo test, but as I say, its very unscientific test and its a big enough difference to make the point.

Time for us to think exactly what we're after, whether we stick with the favoured distro used here (Ubuntu) now we can get the performance up, or continue to consider alternatives! Either way, lots more testing to do!

Thanks for all the help folks, its greatly appreciated :-)

.

Chris

aCOSwt · Posted: Mon Apr 23, 2012 3:06 pm Post subject:

Goverp · Advocate Joined: 07 Mar 2007 Posts: 2021

Having brought cache into the discussion, the next question is whether the Gentoo setup has all the same services installed and running s the other systems. If not, it may have a lot more free space for file system cache, and thereby avoid a lot of real IO. One of Gentoo advantages is tailoring your installation's services to what you need, rather than some one-size-fits-all platform. vmstat output might show significant differences.
_________________
Greybeard