A [risky] way to make emerge really slow

[Irre]

1) unmask and emerge cpulimit
2) limit to one process and one job at a time, for example:

[mike155]

Please tell us why (or in which cases) 'cpulimit' is better than 'nice'.

[Irre]

[Hu]

Your system should be kept cool by its hardware cooling system.

Nice will handle giving other processes CPU preference over the Portage job.

According to the documentation, cpulimit supports being given the command to run, which is likely to be easier to use than resolving the pid manually. Also, based on a quick read of the source, it appears to be vulnerable to a buffer overflow when a process has a very large pid. It looks like cpulimit tries to track parental relations by walking proc, which is racy.

Some programs may react badly to unexpected signaling, particularly anything which uses ptrace as part of its tests. This would be a bug in the impacted program, but still a problem for the users harmed by the bad reaction.

Rather than play games with frequent signaling, why not use the cpu cgroup/scheduler integration to directly instruct the kernel not to run the processes so often?

[Irre]

To tell cpulimit to limit "emerge" is meaningless. You have to identify which subprocess is eating CPU, this is done as I described.

[Hu]

That supposes that only one type of command will be a CPU hog throughout the build. You would be better off limiting emerge and telling cpulimit to also limit all children.

[mike155]

Irre, thanks for introducing me to cpulimit. There are various use cases for such a program:

1. To limit CPU power consumption in order to reduce CPU / system temperature or fan noise. This is a frequent application for me and I usually use cpupower to achieve that:
   

   Code:

   cpupower frequency-info -l    # show frequencies available cpupower frequency-set --max 2GHz

   
   
2. To give more CPU time to other processes. This is also a frequent application for me and I usually use nice to achieve that:
   

   Code:

   nice emerge ...

   
   Sometimes I also use renice to fine-tune running applications.
   
   
3. To limit power consumption of services (Apache, for example). I had such a use case 3 months ago and I used control groups and Systemd. It worked quite well. Systemd allows to limit control CPU usage, Memory Usage, Network bandwidth, etc. See this article for an introduction.
   

I'm not sure whether cpulimit can help me in these use cases:

Regarding use case 1: it's probably better to run the CPU with a constant (reduced) frequency, than to send start/stop signals to an application. Cpulimit might help on CPUs which do not support frequency scaling.

Regarding use case 2: I agree with Hu that it's better to let the OS scheduler control CPU usage.

And regarding use case 3: I would definitely recommend control groups and Systemd. I know that many people don't link Systemd, but Resource Management is something they did right.

[Irre]

[Hu]

As a minor point, systemd provides a way to use the kernel's cgroups to limit resources in this area. You can use cgroups without systemd to achieve the same effect, though the interface for expressing your intent may not be as nice.

[Irre]

Now I understand why cpulimit was masked. While testing, the system crashed hard and I had to run fsck on all file systems!

Don't try "cpulimit" if you have important systems running!

[Ant P.]

It's been over a decade since I've used a filesystem that needed fsck after a crash. Sounds like you don't have an X-Y problem so much as an X-$rest_of_alphabet problem...

[Hu]

In what way did the system crash? Does the system crash that same way if you skip cpulimit and let Portage go to full parallelism? If so, it may be as simple as that you have a problem with your system and cpulimit is not an adequate workaround for the underlying problem. (You mentioned earlier keeping the machine cold. Failure to manage temperature can cause crashes.)

[Irre]

Unfortunately I have no logs as all logs have been written to tmpfs. I don't want to test on this failing machine any more as it is too important for me. It is used as a sambaserver and fileserver. Hardware is Cubieboard 20 (ARMv7). I have rebuilt world and I don't remember any previous crash.

Future tests will be on raspberry pi and gentoo under Virtualbox.

Edit: cpulimit was not masked on ~amd64. But useless. This command locked the system and all fans got crazy. I could enter "poweroff" command and system shutdown started but newer completed! I had to power off my WM!

do not run this command: MAKEOPTS="-j1" cpulimit -l 10 -m -- emerge --oneshot --jobs 1 cmake

[geki]

Ninja is a great resource hog escaping niceness. Using cgroups without systemd is quite easy.
See discussion about chromium build here (cgexec, cgset and cgcreate from dev-libs/libcgroup).
To properly build chromium not running oom on my build bot(4gb ram) I have these settings (I do not care if it is redundant ):

[fpemud]

[Irre]

I emerged stable version of cpulimit and omitted -m option. Now the system didn't crash.

Test: