Is there an inflection point for emerge speed vs cores?

[Amity88]

There definitely is a speed gain when I emerge stuff on a system with 2 cores vs 4. Did any of you guys try emerging on a system with a LOT of cores? like a Xeon 28 core machine?

I'm really curious if there is a point where more cores wouldn't help, assuming that you also have plenty of memory and aren't limited by disk I/O.
_________________

[NeddySeagoon]

Amity88,

I have access to a 96 core arm64 system.
Very few single packages use more than 40 cores but I do run

[toralf]

At a Xeon with 6 cores/12 threads it scales linear for GCC, qtwebengine et al for -j1 till -j11 (I do always use NUM_CPU - 1 b/c one core should be reserved IMO for the OS and remaining tasks).

[krinn]

It's slower (!) than real cores because of network/distcc but well, it should gave you an idea
https://www.youtube.com/watch?v=bJLr0AWqJx0

[Ant P.]

Speed of emerge itself? 1.

[Amity88]

[eccerr0r]

Yeah portage/emerge is single threaded, does not take advantage of cpus/threads/cores. You have a slow single thread CPU? The dependency calculations (currently) cannot be sped up with more cores. (Will portage dependency calculations be multithreaded some day? Not sure how well the communication between threads will work because packages depend on each other.)

Oh you mean make/ninja/waf(booooo)/...? Depends on how well parallelized the build system was built for that particular package, and which packages you are trying to build.

I'm surprised a compute farm is charging by the hour and not core*hours. Core*hours is more related to their cost as it's related to power consumption... In this case they're better ripping you off by giving you just one core and it'd take longer for you to finish...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

[NeddySeagoon]

Amity88,

Its a single arm64 host.
Its a maxed out Cavium Thunder II system provisioned for Gentoos use by packet.net under the "Works on ARM" initiative.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[Amity88]

@eccerr0r,
They actually factor the core count into the hourly compute fee. I did some experiments since my OP, but didn't use AWS EC2 for it. They have a lot of additional charges (bandwidth, disk usage etc) that are sort of hidden that drives up the cost.

However, I found a VPS provider that was much more easier to use and had a very clear cost structure based on configuration. The cost transparency itself was a major selling point for me.


@NeddySeagoon,
Wow, an ARM server, that's gotta be interesting. Do you think they'd ever chip away Intel's market share? I feel like ARM can't catch up with the raw performance of x86. Also, the latter is quite well entrenched at this point.


So, about the experiment. I tried two things on 12,8,6,4 core configurations and checked for CPU utilization and compile speed.

1. Start with a Stage3 and rebuild gcc.
2. Start with a Stage3 and do emerge --quiet -e --jobs=<number_of_cores> system.

In both these tests, I kept the MAKEOPTS="-j<number_of_cores + 1>"
I also compared against my i5-4690 4 core Virtualbox instance. The compute farm's KVM 8 core setup was 3 times as fast.

These were the observations:

1. Performance scaled up really well (almost linearly) when compiling heavy pacakages like gcc. This utilized the cores really well.
2. Performance doesn't scale up well with regular emerge operation, even with multiple jobs. There are very brief spikes where all cores are used, but for the most part they're underutilized.
2.1 I'd wager that this depends on the percentages of heavy packages in the emerge.

Conclusions

1. I think that for Gentoo, we'd benefit the most from few beefy cores. Maybe a really performant 6 core CPU or a decent 8 core CPU would work out best.
2. This experiment was sort of a feasibility study to check how practical it was to build for my Celeron NUC on a Compute farm. All things considered, I'd rather let it take it's sweet time compiling. Engineering time definitely is more expensive than Compute hours.

_________________