Excessive cpu usage when accessed by ssh

[seansmr]

I have a small headless server that I changed to a hardened profile a while ago. Now I have just noticed that it is constantly keeping the cpu busy when when accessed through the lan. I plugged in a keyboard and screen and found the cpu usage to be normal. This happens on a recently updated system and the same system last updated about 4 months ago.
I have very few services running and none of them appear to be hogging the cpu. I have no idea how to fix this.

The cpu usage, in top, is alway divided between user, system and idle. Idle rarely gets over 33%
The strange thing is that the usage percentage are alway a multiple of nice round fractions. Usually 25%, 33%, 50% or 100%. Occasionally I will get 1/5 and 1/6, and even 1/7. Never will the cpu usage be more or less than an exact multiple of one of these fractions.

Does anyone have an idea of what is going on?

[gorkypl]

Try start with using htop to watch the load - it gives much more information, especially on multicore machines. If the machine has more than one core, then the fraction of CPU load may come from 100% usage of one core and close to 0% usage of another ones.
_________________
BTW, TWM FTW!

[seansmr]

Thanks for your reply.

I don't have htop, and I am unable to install it now. It is a 1 GHz, single core SoC, running on a 4GB CF card. Typical reading are like this:
33.3%us, 66.7%sy, 0.0%id
25.0%us, 50.0%sy, 25.0%id
50.0%us, 50.0%sy, 0.0%id
33.3%us, 33.3%sy, 33.3%id
100.0%us, 0.0%sy, 0.0%id
40.0%us, 40.0%sy, 20.0%id
50.0%us, 25.0%sy, 25.0%id
66.7%us, 33.3%sy, 0.0%id

It is never even 0.1% different. I can make top use up to 23% of the cpu by setting it to update every .1 sec., however the reported cpu usage never deviates from the fractional pattern.

[seansmr]

I managed to get htop installed on this. It just leaves me more confused. htop does show more usage by processes, but the sum of each process is greater than 100%
Htop itself is shown as using 83 or 100%, occasionally mysqld and others use either 16 or 20% However, the cpu gauge at the top of the screen only shows htop's usage. If I start top as well, the cpu gauge in htop shows the sum of htop and top, but not other processes. Top then shows htop as using 4.8%

When I plug the small computer into a monitor, it is not on the network, this is where the cpu usage seems normal, 3 - 6%
I will have to try connecting the lan cable then to see if cpu usage is still high.

Before I switched to hardened-sources, I had never seen the cpu use high like this.

[gorkypl]

Looks really strange. I'd try with another hardened kernel, as it might be the easiest test for now.
_________________
BTW, TWM FTW!

[seansmr]

I was just trying that.

I was using the 3.2.11 hardened kernel, before that filesystem got corrupted by trying a non hardened kernel on it. That kernel had the problem. Now I have been using a 3.2.2-r1 kernel, it has the problem.

Now I have tried the linux-3.0.4-hardened-r4 kernel, it does not have the problem. So, the problem only occurs with the 3.2 hardened kernel, when connected to my router.
However, on my desk, I have seen the problem exist and not exist, when not connected to the LAN and connect to as keyboard and screen, with the 3.2 kernel.

I guess that the next thing to try is a newer kernel. Does anyone have any idea on which one will be the next unmasked one?
I am only going to try this once, I will be compiling on a flash card, which is probably not good.

[gorkypl]

Uh, compililng on a flash card is generally not the best idea - can't you connect a network drive? It will be slow but safe.
Second option is to create tmpfs in RAM. With 1G of RAM it won't shine, but should suffice.

As for the kernel version - I'd just take the newest one, it makes little difference if a kernel is marked stable, because if a vulnerability is foound it's going to be removed from portage anyway.
_________________
BTW, TWM FTW!

[seansmr]

Thanks for you help. I have the latest kernel now. It has been compiling for over 45 minutes, still going. I tried a rotational drive in a USB case, but it seems that the case doesn't work. So it is using flash. I have three 4GB flash drives, it will be outdated before it wears out. I had been planning to not do any compiling while away from home.

Will let you know if this fixes the problem.

[seansmr]

The hardened 3.4.4 kernel is installed and running and I still have the problem of excessive cpu usage.

I have tried stopping all services that I can. Today, I get some periods of time where cpu usage seems normal. Then running top or htop in another terminal returns it to almost always 100% Also, starting mysql or sshd does as well. Running top or htop has a 1 - 1 correlation of causing to to go to max cpu, but not in stopping it. Starting and stopping services is only partly correlated.

It kind of seems like a measurement error. Before I knew of the problem, the box was getting unusually hot, however now on my desk if does not get hot. I have no idea what to try next, except to always use an old kernel.

This computer has been running for a number of years. It is set to 586, march and chost. The cpu is set in the kernel to pentium classic.