qtwebengine taking more than a day to emerge

[Goverp]

[<3]

[Hu]

Jumbo build concatenates multiple source files together, then compiles the result. This improves total compilation time, but drives up memory requirements for the compiler. If you picked a parallelism that is right at the edge of usable on your hardware with a normal compile, and then that parallelism is used for a jumbo build with its higher memory requirements, then you will begin using swap, and may run out of memory entirely.

[Chiitoo]

Merged in the topic Time required to emerge package (1094558), with its 13 replies, starting from here and stopping here.

It never did come to light if USE="jumbo-build" was the culprit for the start of the discussion there, as the default state was not yet changed in 5.12.1, but it seemed relevant enough to me to fit it in here. It also had my answer to the 'why', which I kind of thought I had posted here... but obviously didn't.
_________________
Kindest of regardses.

[Goverp]

An observation that might or might not be relevant or useful:
I fell foul of jumbo-build causing swap, but not directly. jumbo-build was the last straw in a storm of memory eaters: -j4 -l4, --jobs 4, using tempfs for portage work files. tempfs in combination with --jobs is a particularly tricky way to load the gun and wait for it to point at your feet.
_________________
Greybeard

[fturco]

According to qlop on my desktop computer I emerged qtwebengine 55 times since late 2016... LOL.
The last emerge took only 5 hours and 35 minutes.
I have an Intel Core 2 Duo CPU and 8 GiB or RAM.
The jumbo-build USE flag is enabled.

[dmpogo]

[Spirch]

i had to google to find out why that one was taking so much time to do

i guess having a brand new computer (with the 3900x) and with 32 gig of ram pay off

-j24 -l20

build time was 47 minutes based on what i'm seeing

(btw i'm back into gentoo after nearly 6-7 years away)

[skellr]

[OldTurk]

Hi,

I also had this problem. The results below are from a VM :

[Delicates]

Confirming that excessive build time started with qtwebengine-5.12 versions due to removal of the default jumbo-build USE flag.

Building the current version on a on a hyper-threaded hex-core system with 48 GiB of RAM inside tmpfs ramdisk.
Interestingly the number of parallel make jobs doesn't really seem to affect the build time.

Without jumbo-build USE flag on -j3:

[JustAnother]

[Fitzcarraldo]

[flyerone]

This topic still comes up as a winner on Google..

I have the i7-8700 with 64 GB RAM and I've set up 14 GB as a tmpfs for portage. I've had the jumbo-build disabled globally for just the chromium package which bugged but I can try it with qtwebengine in package.use after I've read this. I don't feel like testing right now as I have 20 minutes left of the build.

I've set up the tmpfs to ease the wear on the nvme drive and using tmpfs does it also pay off to omit the -pipe flag?

Perhaps I'll start a pipe vs jumbo thread of my own after some testing and I'll win Google in a week.

[NeddySeagoon]

flyerone,

-pipe is a hint to gcc that it should pass intermediate files in RAM if it can.
Its not a command.

If you omit -pipe, then the normal disk based intermediate file will be used but I don't know where gcc would write these files.

Whatever, -pipe is a good thing.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[flyerone]

[Hu]

-pipe is passed if the caller chooses to pass it. If it is present, gcc may try to use an anonymous pipe to pass data to the next process. If successful, then even if your normal temporary directory is on a disk, the temporary files will not be written to the disk. Otherwise, it will write to a temporary file. Files can be written to a tmpfs, in which case they likely stay in memory anyway. -pipe was more likely to help when it was uncommon to use a tmpfs for build products. Even now, it is very unlikely to hurt.

[NeddySeagoon]

flyerone,

Consider the way that tmpfs works. Its the normal low level filesystem code but there is no home for the content permanently on disk.
In effect, that means that if you have the RAM to build in tmpfs, you don't need to because the kernel will keep everything in the disk cache anyway.
Hence you don't see any speed improvement.

However, building on permanent storage when you don't need to incurs writes that will never be read.
As they will be done using DMA, the CPU overhead is tiny. Also the CPU will win memory bus arbitration when it need to access RAM and a DMA for a disk transfer is in progress.

Saving writes that will never be read is a good thing for SSDs. Even that doesn't matter so much these days.
The SSDs will all be retired before they wear out.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[ff11]

[Anon-E-moose]

Some ssds behave strangely, especially as they near their "max writes" threshold.
I have lost the link but several years ago some people did an endurance test for multiple ssd, multiple vendors, etc.
Some did well (samsung is one of the better ones) others like intel, didn't fade away (like a rust bucket) but just had sudden death (but even with that it was at the life write threshold anyway).
But even with that, we're talking several years ago behavior, ssds are now coming into their own.

Is it possible that an ssd would just die, even today? Yep, but with each gen of newer drives/controllers/etc that diminishes.
And I've had hdd just suddenly die too.

All in all, I'd trust a relatively new ssd just as much as I'd trust spinning rust.

As far as trusting data on any storage device not to have a problem some where, sometime, look at Neddy's sig

Edit to add: I don't do emerges/compiles on the ssd though, I do use a tmpfs for that (plenty of memory) but emerge sync's write to the nvme daily.
_________________
UM780, 6.1 zen kernel, gcc 13, profile 17.0 (custom bare multilib), openrc, wayland

[NeddySeagoon]

Anon-E-moose,

There was this small scale experiment The SSD Endurance Experiment
Wearing a drive out with writes has nothing to do with random failures, which is what the MTBF tries to asses.

Backblaze has more failure data than is useful from their actual installed drives.
I'm not sure if it includes SSDs.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[ValerieVonck]

On my i7 Nuc, (4 threads), the compilation takes around 5 hours, 30 minutes, for version 5.14. (SSD disk)
I am not using JUMBO_BUILD

On my VirtualBox (1 thread), i5 processor, 4 GB ram, 2 GB swap, the builds are taking 18 hours+
On my other VirtualBox (1 thread), i7 processor, 4 GB ram, 2 GB swap I even do no try it anymore...

I really prefer, a binary package, with all flags enabled, I could not care if the download takes around more then 1 or 2 GB.

Is this possible?

In my USE flags, I am including "icu".
If I should remove it, should QT dissapear? Or should I add "-qt" in the USE flag?
And then do a world update?

I then can emerge firefox-bin or chrome binary.. which is normally against my Gentoo philosophy..

I know its used for Falkon (and other programs), which does not even start, and for some other libraries

But seriously, its becoming ridiculously, the compilation times.
_________________
Inter antecessum est melius

[AstroFloyd]

I may be asking something very stupid, but would it be useful (and if so, is it possible) to *not* clean the build dir for a given package? If not cleaned, and the source files don't change, make could just take the existing object files and e.g. link against the new version of a dependency...?

[Fitzcarraldo]

[Hu]