The abomination called "dev-qt/qtwebengine"

eccerr0r

I'm not sure how I avoided it for a few cycles, perhaps laziness or not noticing qtwebkit going away, but a recent attempt at building some package pulled it into the dep tree and oh-my-gawd...

6GB disk usage...

no, not that.

2GB cc1plus proceses!

No, not just that... hundreds of 2GB cc1plus processes that needs to be run to complete!

I suppose using jumbo-build (default?) supposedly decreases build time but oh my, this uses a lot of RAM. What's scary is that I have distcc going and...

QTWEBENGINE single handedly KILLED MY WHOLE DISTCC FARM! (This isn't ddos or dos, more like smdocf or something... single machine denial of compute farm) Well, not exactly 'killed' due to nice, but distcc indeed spread the pain. Oh my goodness, all of my machines were taking these jumbo builds and swap thrashing their usual jobs with the ginormous distcc cc1plus jobs...

Not only they're ginormous, distcc gave up waiting for some of the machines waiting for them to complete, wasting that build time even though it eventually finished with swapping.

Ok done with rant. Probably with distcc I should disable jumbo build, or what have people been using for this?

2021-03-18T10:07:43 >>> dev-qt/qtwebengine-5.15.2_p20210224: 2:12:27
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

Zucca

eccerr0r · Posted: Sat Mar 20, 2021 2:32 pm Post subject:

Not sure if it was deprecated but it's still the default USE for the version I built. Disabling jumbo builds probably will help, and if long term this needs to be disable, all the better for consistency.

I'm just surprised these jobs were so large - the GbE network didn't flinch at the files transmitted to the helpers, but once the helpers got the job, they themselves needed help, I didn't recall ever seeing distcc jobs so big but I've heard of all these 2GB c++ jobs before, but not until now did I experience this pain.

At least this seems to be the exception -- and I think I need to put more RAM into the quad core machines...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

fedeliallalinea

CaptainBlood · Advocate Joined: 24 Jan 2010 Posts: 3858

@Zucca,

Zucca

eccerr0r · Posted: Sat Mar 20, 2021 4:17 pm Post subject:

I don't think it's nproc/2 that's best... for x86_64, if you have a little bit of swap, it seems that the best is take how much RAM in gigabytes you have and divide by 2. That should be your max processes for qtwebengine[jumbo-build]. This should work for virtual CPU(multithread) machines too that nproc might get too greedy.

For distcc farms, this is more of a challenge since distcc seems to willy nilly submit jobs and hope they come back and can't ask the remote machines how much RAM they have... eek.

Will have to see what the difference is if I disable jumbo build when it comes time to build again...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

CaptainBlood · Advocate Joined: 24 Jan 2010 Posts: 3858

I was more thanking for exposing the command itself, so nothing regarding qtwebengine per see, especially.
Likely to be helpful simplify SMT status integration to MAKEOPTS default and package specific settings.

Thks 4 ur attention, interest & support.
_________________
USE="-* ..." in /etc/portage/make.conf here, i.e. a countermeasure to portage implicit braces, belt & diaper paradigm
LT: "I've been doing a passable imitation of the Fontana di Trevi, except my medium is mucus. Sooo much mucus. "

eccerr0r · Posted: Sat Mar 20, 2021 6:13 pm Post subject:

Ah okay, still trying to figure out what's the best build option for speed. Probably not a one size fit all, which kind of makes it tough to maintain.

Sort of trouble for me too, when I upgraded a dual core to a quad core CPU... what are the correct options after the upgrade when the amount of RAM did not increase. Some packages will still take benefit from the two extra cores...

... and this abomination qtwebengine will just thrash.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

Zucca · Posted: Sat Mar 20, 2021 6:25 pm Post subject:

eccerr0r · Posted: Sat Mar 20, 2021 6:38 pm Post subject:

Hard to say what the best solution is. Polling information about helpers sounds like a good idea, though, and then the help requester could figure out how to best distribute jobs - dump the big jobs on the big machines, and try to localhost all the small jobs.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

Hu · Administrator Joined: 06 Mar 2007 Posts: 22650

distcc understands DISTCC_HOSTS to limit the hosts to which it can distribute, and you can set that in package.env. That is still a manual solution though, as it requires the administrator to determine on a per-package basis which hosts should be used/skipped. On the other hand, the administrator may want to tune MAKEOPTS down when running packages for which few hosts will offer distcc assistance, so at least both knobs would be set in the same place.

eccerr0r · Posted: Sat Mar 20, 2021 7:00 pm Post subject:

It appears distcc will select the earlier machines in DISTCC_HOST as preference but it would be kind of dumb to send the small jobs to the preferred machine first, then when a big job comes, submit it to the nonpreferred machines...

But yeah unfortunately seems that it's necessary to maintain custom distcc configs for qtwebengine along with gcc and rust, my package.env is getting contaminated with tons of crap...

(gcc - compile locally because gcc doesn't properly honor different make flags for stage 1, and stage 2/3 can't be distributed, and gcc-fortran technically should be distributed... ARRGH.
rust - again, or at least partially stage 0/1 can technically can be distributed, but the rusty parts cannot and should have its own flags.
and now qtwebengine needs only the biggest and best machines, and drop all the mere 1G machines from distcc_hosts...)

I wish "load average" was a better metric of actual CPU utilization... but instead it's a reason for ninja to go ahead and submit all -j 999 jobs to localhost when the load average drops to 0.5 after a bunch of jobs finish at the same time...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

xahodo · Posted: Sun Mar 21, 2021 5:44 pm Post subject:

Well, I just put -j1 into my MAKEOPTS for qtwebengine. It'll take longer to build (24 hours), but at least I can still use my computer.

Gatsby · Posted: Tue Mar 23, 2021 9:09 pm Post subject:

Yeah, qtwebengine is indeed insane shit, like almost all, coming from Google.
_________________
"Its your Gentoo, your way. When it breaks, you can keep all the pieces."
-- NeddySeagoon@forums.gentoo.org

Zucca · Posted: Tue Mar 23, 2021 10:07 pm Post subject:

asturm · Developer Joined: 05 Apr 2007 Posts: 9277

The rough lineage is KHTML -> webkit -> chromium/webengine.

Zucca · Posted: Tue Mar 23, 2021 10:13 pm Post subject:

eccerr0r · Posted: Wed Mar 24, 2021 2:19 am Post subject:

sdauth · l33t Joined: 19 Sep 2018 Posts: 650 Location: Ásgarðr

Personally, I was only tied to it because of media-video/openshot (and dev-python/PyQtWebEngine) so I replaced openshot with media-video/shotcut to get rid of it.

Jojobinha_2009

Whenever I go to install Gentoo, the first thing I do is edit the make.conf and place "-qtwebengine" and "-webengine" in my USE flags.
Avoids 4+ hours of compiling...
_________________
Intel Core i5-9400F / 24GB DDR4 2666MHz / GeForce GTX 1060 3GB

Powered by Gentoo for x86_64

======================================================

Seize the day, and remember to have fun!

CaptainBlood · Advocate Joined: 24 Jan 2010 Posts: 3858

Zucca · Posted: Sun Apr 04, 2021 12:07 pm Post subject: