Looking for advice on Linux benchmarking...

[chix4mat]

Hi all:

I'm currently in the process of revising a few review methodologies at Techgage.com, and for our processor reviews, I'd like to introduce some Linux elements if possible. So, the main question I have is... what could we add to our processor reviews that would prove useful to you, and also most Linux users?

The bind I'm in is this. On the Windows side, we test with many common applications, such as Autodesk's 3ds Max, Adobe's Lightroom, and even Microsoft's Excel. Even if people don't use those particular applications, they've probably at least have heard of them. On the Linux side, there are hundreds of distros, and each application set varies. So which applications could we possibly benchmark that people could relate to?

One reason Linux tends to scare some people is because they associate it with being a command-line OS. So ideally, I'd like to stay away from using the command-line if possible, since few end-users are going to be concerned with how well an elaborate script runs. But then the issue again arises, which desktop applications are worth benchmarking?

A benchmark that's sure to return is application compiling, because most Linux users (especially in the Gentoo world) are going to find themselves in that scenario at some point. Plus, depending on the application or tool being compiled, the time-to-compile can vary wildly depending on the processor. And not to mention the multi-threaded nature of GCC... a huge plus.

In the past, we ran a small script that used something like GPhoto2 (I believe) to convert a collection of photos in RAW format to resized JPEGs, and while the scenario is a good one, the single-threadedness of the application pretty much ruined things (especially when we're benchmarking 2 - 8 core setups).

All recommendations are welcomed, but please bear in mind that any potential tests should be understandable by most people, and also be of a multi-threaded nature if at all possible.

I also have a quick question or two regarding setting up a proper OS for benchmarking on a given architecture.

Does it matter how the kernel or the entire OS was compiled when benchmarking across various architectures? What I mean is, if all of Gentoo was compiled with an Intel-specific CFLAGS option, would results on an AMD platform be inaccurate, even if I changed the CFLAGS option before I began compiling an application for benchmarking purposes?

The reason I ask is this. We'll be benchmarking processors from three differing processor architectures: AMD, Intel Core 2 and also Intel Core i7. My thinking is that I'll need three completely separate Gentoo installs, one optimized for each architecture (I'm basing my choices primarily on the "Safe CFLAGS" page at Gentoo-Wiki, and also choosing the most common-sense option under the "Processor Type" in the kernel configuration). Am I wrong?

The other question would be, in order to "optimize" for each architecture, would it be alright to simply continue off the same Gentoo install, changing the processor type in the kernel and also the CFLAGS option, and then simply do an emerge -e (while also recompiling the kernel)?

Alright, I've written far more here than I probably should have, so I'll tear my hands away from the keyboard. Any help at all is really appreciated, because in the end, I want to make sure that the benchmarks and methodologies we choose are both relevant and given the A-OK from serious Linux users.

Thanks much!
_________________
emerge -NuDe world

[Icer]

Probably good desktop apps would be openoffice, gimp and firefox. That's just what comes in mind first.

However you could check phoronix test suite for ideas: http://www.phoronix-test-suite.com/
Hmm... I was wondering about it's availability but it turns out it is in the portage already.

[chix4mat]

Hi Icer:

Thanks for the response!

OpenOffice, Firefox and GIMP are three of the most common applications in Linux, but a) they can't really be benchmarked and b) how many people are truly going to be interested in the performance there? In OpenOffice, the only scenario I could understand is executing a complex macro, but we already do that with Microsoft Excel, and even then, most people who do run such a file are likely forced to use Windows because of their jobs.

While I appreciate what Phoronix is doing with their test suite, I'll admit that I find no real value in most of the tests included. This isn't so much the fault of the test suite, but rather the fact that there's an overall limit of benchmarking choices in the Linux world, hence my predicament. As I mentioned before, on the Windows side we use many popular applications, and even if most people haven't used them, they've at least heard of them, and understand them.

There are many tests inside of the PTS that I've considered in the past, but in today's scheme of things, most serve little purpose. When's the last time you've contemplated MPlayer performance, really? And when's the last time you were concerned that one GPU scored 400 FPS in Nexuiz when another scored only 350 FPS?

The goal at our site is to focus on real-world scenarios as much as possible, and if we're forced to use some synthetic benchmark, then we'd at least like for it to be relevant. Our compiler test, for example. It's a little geeky, but at least it's relevant to the majority of Linux users, and the majority can understand the real benefits.

On the Windows side, we use applications like 3ds Max to render an environment, Adobe Lightroom to convert RAW to JPEG, TMPGEnc XPress to convert a 1080p video into a mobile format, and so on. It'd be great to be able to offer up similar real-world scenarios on the Linux side as well, so that anyone who reads that page in our review will understand exactly what we're doing. I don't want to confuse people with complex scripts or results that matter none in the real-world.

One thing I guess I need to do is simply figure out which applications most people use for given scenarios under Linux. I admit that I run a lot of applications through Wine, and also run VMware for a full-blown XP installation, so I might overlook some of the popular choices on the Linux side.

I also recalled two other benchmarks I used in the past, and both are kind of geeky, but they do hold real value. One is checking the MD5 checksum on a large file, while the other is .tar.gz and .bz2 archiving. I don't think tar.gz is multi-threaded, but if memory serves, .bz2 is.

I'll take a deeper look later today at things... I'd love to include video encoding, music encoding and photo manipulation if possible. I just hope I can find some applications that are multi-threaded...

Thanks for the advice on the system configuration also!
_________________
emerge -NuDe world

[ianw1974]

If you want to test video encoding from avi/mpeg to DVD format, you could try DeVeDe or Tovid GUI although I prefer DeVeDe. If you want re-encoding video from one format to another, you can try HandBrake. DeVeDe will take advantage of multiple CPU's etc, so it should be ideal. HandBrake I'm unsure of but possibly.
_________________
Ian Walker

Light travels faster than sound. This is why some people appear bright until you hear them speak.........

Linux Systems Limited | Masternode Monitoring

[chix4mat]

Hi ianw1974:

For various reasons in the past, I haven't really come to enjoy DeVeDe, but this is the first I believe I've heard of Handbrake. I gave it a download though, and I have to give a huge thanks, because it's seemingly perfect! I encoded a 1080p video to a format playable on the PS3, and it actually uses about 75% of a Quad-Core processor throughout the process. Seems like a perfect addition to me. I'll test it out more in the days to come to find out for sure.

Still looking for more suggestions, so bring 'em on peeps!
_________________
emerge -NuDe world

[HeissFuss]

As you mentioned,
CD ripping (LAME, FLAC, faad)
Compression/extraction (pbzip2 is the only parallel one I know of, and I don't think it's used very often)
h264 decoding/x264 encoding, maybe theora too.
Firefox javascript performance maybe.

Also, I don't use it myself, but maybe Blender for 3d rendering.

[chix4mat]

Hi HeissFuss:

I've considered LAME and FLAC benchmarks before, but the problem there is that they're not multi-threaded (or at least since last check). I do wonder if an application exists for Linux that could handle use them in a multi-threaded manner, though (dBpoweramp does that in Windows). Compression is definitely one I want to look into though, so thanks for the pbzip2 idea. I'm not sure how that compares to regular bzip2, but I'll find out soon enough.

H.264... that will be tackled with HandBrake. That has proven to be a stellar benchmark thus far, so I'll no doubt do both an H.264 test along with another (perhaps FFMPEG). As for Firefox, that's a tough one, because I'm not sure of the real interest there, or even how to run one. I think metrics like that would be more suitable for a browser review, not be used as a basis for CPU performance.

Thanks for the ideas!
_________________
emerge -NuDe world

[ianw1974]

For audio:

Try sound-juicer for ripping CD's - Gnome/GTK based.
Try sound-converter for converting from flac to mp3 for example.

I don't know if multi-threaded but it's what I've used in the past without problems.
_________________
Ian Walker

Light travels faster than sound. This is why some people appear bright until you hear them speak.........

Linux Systems Limited | Masternode Monitoring

[chix4mat]

Hi ianw1974:

We won't be able to include a benchmark that actively rips audio CDs, because that's heavily dependant on the ODD, not the CPU or anything else in the machine. Even at the top-end, the ODD's speed would still be nothing but a joke to the CPU.

I looked at the SoundConverter website though, and it clearly states, "Thanks to its multithreaded design, it will use as many cores as possible to speed up the conversion."... so, you really seem to know how to pick 'em, haha. Thanks once again... I'm working on another machine right now, but once I get a chance, I'll boot that machine up again and give that program a go. If it's multi-threaded, I have no immediate reason to believe it won't be used.

I'm also going to look into F-Spot and possibly gThumb as well, to see if either would be appropriate for batch image manipulation. If so, then I think we pretty much have video, images, audio and also compiling covered... all real-world benchmarks. That should be plenty for our reviews, I'd reckon. Although for good measure, I'll probably also add the archiving scenario, and possibly also something simple like MD5 checking.
_________________
emerge -NuDe world

[pdr]

Check phoronix.com - they have a linux benchmark suite.

[chix4mat]

[pdr]

Since most apps dynamically link to the standard C lib, if you want to use custom CFLAGS for the kernels you'll also need it for the std c lib (glibc?) - meaning separate installs. However from the general linux population point of view, that uses binary distros, you might not want to have custom CFLAGS - it is not real-world for them. When I put Ubuntu onto my laptop I did not get an option to download one compiled for my pentium-m.

When it comes to CPU, my only personal power requirements are video transcoding. I use my own scripts and/or pygtk programs for this, and use mencoder (with ffmpeg lib). And those are written so that if I'm dealing with multiple videos I have multiple processes spawned; so that is the kind of bench I'd be interested in.

You might consider that route: have someone write something that stresses, prints timing, and is useful - and offer it on Techgage as FOSS. As an example: I have a python script called tvdvd which, given an optional background image, optional sound clip (both for the menu), optional fonts, colors etc, and a set of video files will create a DVD image directory with a menu to the videos. It uses various commandline tools - imagemagick, png2yuv, mpeg2enc, mplex, spumux, mencoder, etc and whenever multiple steps can be run in parallel it runs them as separate processes; as an example, when converting the video clips to mpeg format - those can each run parallel. I use this to back up directories containing TV show episodes to DVD, so it is useful.

I haven't started playing with it yet, but another useful app that might be CPU intensive might be to automate some replaygain stuff. Or - again from a video perspective: I used to use a bash script with lsdvd to list out the titles on a DVD, prompt, and then use mplayer to optionally preview and then -dumpstream the title to disk. A useful (in fact I think I'll write one) app might be to list the titles, let the user optionally preview each, select mulitple ones and select file names, format (eg h.264) and rip/transcode them (in parallel).

The thing is, such apps - in particular if scripted - are easy to mod to print benchmark timing data.