Low % cpu usage, high load problem.

atct · n00b Joined: 21 Sep 2005 Posts: 9

I run a Dual Xeon server with 2 HT capable processors and 2 gig of ram.

The following problem happens on a 100Hz kernel, 250Hz kernel, a kernel using a different memsplit so i don't need highmem for the 2 gig, and a standard kernel with which only 884MB of mem gets adressed.

There are 3 gameservers running on this server, all HLDS running TFC.

The problem is that even when the cpu's are doing 4% usage each, sometimes the load goes up to 1, sometimes even to 4.
That is causing problems on the performance part offcourse, and i want to isolate the cause of that irradic behaviour.

The question is, how do i isolate the cause?

With top, htop, vmstat, mpstat, iostat, etc i don't see anything interesting when this happens.

Can anyone tell me the right procedure to track this problem down?

erik258

what are your use flags, kernel config, and what are the programs doing when they are running ? Top can tell you whether they are waiting on i/o or something

also have a look through dmesg for anything suspicious . post and let me have a look too.

atct · n00b Joined: 21 Sep 2005 Posts: 9

Sure, brace yourself. :wink:

It's now running a kernel without highmem or non standard memsplit, just because i had to be sure it wasn't an issue having to do with one of them.

atct · n00b Joined: 21 Sep 2005 Posts: 9

[Bump]

Anyone knows something that can help me tackle this?

I still need help.

erik258 · Posted: Sun Apr 09, 2006 3:22 am Post subject: i wonder

I wonder whether it isn't a kernel issue. I wouldn't know where to begin fixing it, especailly sitting here potentially t
housands of miles away but, it sounds like a kernel issue to me.

are your harddrives running with dma mode 5 enabled? If idle time is high maybe it's a scheduling problem , if it's high usage maybe a dma or other io problem; memory might be a possibility too... but you said you checked all this. There must be some bottleneck, but if not, i would guess kernel config issue.

Good luck, this sounds like a tricky one ; )

atct · n00b Joined: 21 Sep 2005 Posts: 9

So the best i can do is compiling kernels with different options, and try them one by one?
I'll look into it, but don't really know what to change in the config because as fas as i know, it's setup right.
If i would compile a new one from a config i would make without looking to the older one, i bet it would be the same.

erik258 · Posted: Sat Apr 15, 2006 4:27 pm Post subject: I think so too

That's just what seems the most obvious solution to the problem. A kernel config, i agree, isn't likely to help much.

My question is, do other people have this problem? Is it a motherboard thing? Post specs to have me help dig up some info. Seems likely that your exact installation would do fine on another similarly-architectured box.

But did you turn on hyper-transport, HPEC timer, all like that? Oh sorry, i think the intel-based one is hyper-threading, right?

atct · n00b Joined: 21 Sep 2005 Posts: 9

Specs:

MB: Asus PC-DL Deluxe
2x intel Xeom 2.66Ghz 533FSB
2x 1Gig Ocz DDR400
Samsung Sata Hdd's

The kernel is using hyperthreading.
The things i read about HT are very inconsistent. One says it does improve performance, another says it degrades performance.
I have to find that out for myself i think so i'll compile 2 the same kernels, one with and one without.

Tomorrow i have to go to the datacenter anyway, so i'll check a few kernels are working (without fuckups like forgetting to compile lo i had once

)on that machine so i can reboot to a new one each night.

erik258 · Posted: Tue Apr 18, 2006 7:42 pm Post subject:

as i understand it (don't have any currently installed p4 processors up myself) HT helps IF the 'data pipeline' is full. If there isn't enough info to process, the easy stuff must be pushed through by something coming up behind it or something like that

let me know how the kernels perform would you? I think you're missing a driver for an on-motherboard hardware device that helps with multithreading etc. that's just a guess, but I have not seen / heard of this problem before.

diamondsw · n00b Joined: 05 May 2004 Posts: 67

This sounds eerily similar to a problem I've been facing the last couple weeks. I did a huge update of my system (first emerge in several months), and ever since, my system load never drops below 1.0 (I also have a dual processor system, although its a P2-400). CPU usage is never above a couple percent as I watch. The system - especially networking - is extremely sluggish. Copying 300KB over NFS on 100baseT takes 5 minutes, for example, and ssh sessions are noticably laggy on the same LAN segment.

Here's a good example from top:

atct · n00b Joined: 21 Sep 2005 Posts: 9

I don't have any lagging ssh issues and the load is not constant, it just sometimes decides to be around 1 or above for a long time without any reason.
The testing is going a bit slow atm because first i want to move the game to another machine.
After i did that i can start to make the datacenter crazy with reboot requests when it fucks up.

It's a shame those IP KVM extenders are so expensive, otherwise that would be a great thing to get, but for those prices i could just exchange the whole machine for a nice dual opteron.

So i'm a bit delayed trying to solve the problem, but when i start testing, i'll post the results here.
I'm also going to exchange the NIC for another one to see if that helps, when i rebooted it last time with a new kernel it was bugged in a strange way, so that gave me a hinch...

erik258 · Posted: Wed Apr 26, 2006 4:03 am Post subject:

i do not envy you, having to solve this, although i wish i owned that computer. but good luck, and i will continue to watch more for my own good than yours ; ) . i might learn something, after all.
_________________
Configuring a Firewall? Try my iptables configuration
LinuxCommando.com is my blog for linux-related scraps and tidbits. Stop by for a visit!

syg00 · Posted: Wed Apr 26, 2006 5:40 am Post subject:

Mmmmm - why am I (apparently) the only person that doesn't think that 1 constitutes a high load value ???.
If it's not running processes that appear to be the problem, try top, and (reverse) sort the process status field (S). You're looking for a "D" for tasks in uninterruptable (disk) wait - these are also counted for load calculations.

Don't know why it'd be an issue, but am also interested in anything you find.

diamondsw · n00b Joined: 05 May 2004 Posts: 67

syg00 · Posted: Fri Apr 28, 2006 6:30 am Post subject:

I've been noticing some (gentoo) threads with performance issues on moving to 2.6.16 - is this your case as well ???.
Nothing obvious in the changelog

I don't have a gentoo on .16, but my Arch systems at that level seem o.k.
I'm about to build a .17-rc2 to test a (kernel) patchset, I'll see if that shows any oddities.

erik258 · Posted: Fri Apr 28, 2006 7:03 pm Post subject: 2.6.16 and one processor

I have 2.6.16 freshly installed, just this week, on a p4 celeron. Of course the architecture isn't exactly the same, but I am very pleased with the results.

I do not think the problem is inhereint in 2.6.16

I have noticed, however, that the load in top does seem a little higher than normal. The computer is responding fine thougth, and I have no reason to think anything's wrong. Unlike you, unfortunately.

I wonder if it could be an SMP problem or something. I of course have it disabled, having a celeron means no hyper-thereading for me. Perhaps there is another difference between our systems too.

I feel bad for you, I don't think any of us know what to say about this problem. If there's anything I can do to help, just let me know.
_________________
Configuring a Firewall? Try my iptables configuration
LinuxCommando.com is my blog for linux-related scraps and tidbits. Stop by for a visit!

diamondsw · n00b Joined: 05 May 2004 Posts: 67

Problem turned out to be a Dell firmware update module that appeared in the kernel at some point along the way. Took that out and load returned to normal.

erik258 · Posted: Tue May 02, 2006 7:24 pm Post subject: lucky you

lucky you, you now have an extremely quick computer, don't you? enjoy it
btw congradulations on your successful fixing of the problem.

_________________
Configuring a Firewall? Try my iptables configuration
LinuxCommando.com is my blog for linux-related scraps and tidbits. Stop by for a visit!

Treiks · Tux's lil' helper Joined: 15 Apr 2004 Posts: 82

I have the same. CPU ~0,0% and load never below 1.0!!! I've noticed this after changing kernel from 2.6.14.7 vanilla with grsec to 2.6.16.9 vanilla with grsec only. The same while using 2.6.16.12.

erik258 · Posted: Thu May 04, 2006 6:41 pm Post subject: config!

i believe you should check yourconfig, as in that was this fellow's problem. Are you sure there's nothing in there you don't want? Or maybe, can't have?
_________________
Configuring a Firewall? Try my iptables configuration
LinuxCommando.com is my blog for linux-related scraps and tidbits. Stop by for a visit!

Treiks · Tux's lil' helper Joined: 15 Apr 2004 Posts: 82

I turned off most services on this machine.
Take a look it's only couple minutes after reboot only with sshd and ppp:

syg00 · Posted: Fri May 05, 2006 8:25 am Post subject:

See my previous post. If that thing is in a (constant) uninteruptable wait, I'd be complaining. Supposed to be (very) short-term.

johnnyboyro · n00b Joined: 19 Sep 2003 Posts: 57 Location: Iasi, Romania

Don't know if it's related or not, but I did experienced some time high loads while having some zombie processes. Just killing those zombies helped.
Another time I got this it was an hardware problem: the system was using some SCSI RAID hardware wich had a too small cache on it, while the number of users and processes running on this machine was quite big ...