A [risky] way to make emerge really slow

Irre · Guru Joined: 09 Nov 2013 Posts: 434 Location: Stockholm

1) unmask and emerge cpulimit
2) limit to one process and one job at a time, for example:

mike155 · Posted: Sat Mar 31, 2018 10:55 am Post subject:

Please tell us why (or in which cases) 'cpulimit' is better than 'nice'.

Irre · Guru Joined: 09 Nov 2013 Posts: 434 Location: Stockholm

Hu · Administrator Joined: 06 Mar 2007 Posts: 22754

Your system should be kept cool by its hardware cooling system.

Nice will handle giving other processes CPU preference over the Portage job.

According to the documentation, cpulimit supports being given the command to run, which is likely to be easier to use than resolving the pid manually. Also, based on a quick read of the source, it appears to be vulnerable to a buffer overflow when a process has a very large pid. It looks like cpulimit tries to track parental relations by walking proc, which is racy.

Some programs may react badly to unexpected signaling, particularly anything which uses ptrace as part of its tests. This would be a bug in the impacted program, but still a problem for the users harmed by the bad reaction.

Rather than play games with frequent signaling, why not use the cpu cgroup/scheduler integration to directly instruct the kernel not to run the processes so often?

Irre · Guru Joined: 09 Nov 2013 Posts: 434 Location: Stockholm

To tell cpulimit to limit "emerge" is meaningless. You have to identify which subprocess is eating CPU, this is done as I described.

Hu · Administrator Joined: 06 Mar 2007 Posts: 22754

That supposes that only one type of command will be a CPU hog throughout the build. You would be better off limiting emerge and telling cpulimit to also limit all children.

mike155 · Posted: Sat Mar 31, 2018 9:00 pm Post subject:

Irre, thanks for introducing me to cpulimit. There are various use cases for such a program:

To limit CPU power consumption in order to reduce CPU / system temperature or fan noise. This is a frequent application for me and I usually use cpupower to achieve that:

Code:

cpupower frequency-info -l # show frequencies available
cpupower frequency-set --max 2GHz
To give more CPU time to other processes. This is also a frequent application for me and I usually use nice to achieve that:

Code:

nice emerge ...

Sometimes I also use renice to fine-tune running applications.
To limit power consumption of services (Apache, for example). I had such a use case 3 months ago and I used control groups and Systemd. It worked quite well. Systemd allows to limit control CPU usage, Memory Usage, Network bandwidth, etc. See this article for an introduction.

I'm not sure whether cpulimit can help me in these use cases:

Regarding use case 1: it's probably better to run the CPU with a constant (reduced) frequency, than to send start/stop signals to an application. Cpulimit might help on CPUs which do not support frequency scaling.

Regarding use case 2: I agree with Hu that it's better to let the OS scheduler control CPU usage.

And regarding use case 3: I would definitely recommend control groups and Systemd. I know that many people don't link Systemd, but Resource Management is something they did right.

Irre · Guru Joined: 09 Nov 2013 Posts: 434 Location: Stockholm

Hu · Administrator Joined: 06 Mar 2007 Posts: 22754

As a minor point, systemd provides a way to use the kernel's cgroups to limit resources in this area. You can use cgroups without systemd to achieve the same effect, though the interface for expressing your intent may not be as nice.

Irre · Guru Joined: 09 Nov 2013 Posts: 434 Location: Stockholm

Now I understand why cpulimit was masked. While testing, the system crashed hard and I had to run fsck on all file systems!

Don't try "cpulimit" if you have important systems running!

Ant P. · Watchman Joined: 18 Apr 2009 Posts: 6920

It's been over a decade since I've used a filesystem that needed fsck after a crash. Sounds like you don't have an X-Y problem so much as an X-$rest_of_alphabet problem...

Hu · Administrator Joined: 06 Mar 2007 Posts: 22754

In what way did the system crash? Does the system crash that same way if you skip cpulimit and let Portage go to full parallelism? If so, it may be as simple as that you have a problem with your system and cpulimit is not an adequate workaround for the underlying problem. (You mentioned earlier keeping the machine cold. Failure to manage temperature can cause crashes.)

Irre · Guru Joined: 09 Nov 2013 Posts: 434 Location: Stockholm

Unfortunately I have no logs as all logs have been written to tmpfs. I don't want to test on this failing machine any more as it is too important for me. It is used as a sambaserver and fileserver. Hardware is Cubieboard 20 (ARMv7). I have rebuilt world and I don't remember any previous crash.

Future tests will be on raspberry pi and gentoo under Virtualbox.

Edit: cpulimit was not masked on ~amd64. But useless. This command locked the system and all fans got crazy. I could enter "poweroff" command and system shutdown started but newer completed! I had to power off my WM!

do not run this command: MAKEOPTS="-j1" cpulimit -l 10 -m -- emerge --oneshot --jobs 1 cmake

geki · Posted: Sun Apr 01, 2018 7:18 am Post subject:

Ninja is a great resource hog escaping niceness. Using cgroups without systemd is quite easy.
See discussion about chromium build here (cgexec, cgset and cgcreate from dev-libs/libcgroup).
To properly build chromium not running oom on my build bot(4gb ram) I have these settings (I do not care if it is redundant

):

fpemud · Guru Joined: 15 Feb 2012 Posts: 350

Irre · Guru Joined: 09 Nov 2013 Posts: 434 Location: Stockholm

I emerged stable version of cpulimit and omitted -m option. Now the system didn't crash.

Test: