| View previous topic :: View next topic |
| Author |
Message |
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
 Posted: Tue Jan 02, 2018 2:19 pm Post subject: Save 30-50% off Chromium compile time with one simple trick! |
 |
|
</clickbait>
It's real though!
The build system in newer versions of chromium has support for source file concatenation, like what sqlite3 does (or to a lesser extent, the USE=pch flag). It's not enabled by default, but you can try it out yourself by doing this:
| /etc/portage/package.env: | | www-client/chromium chromium.conf |
| /etc/portage/env/chromium.conf: | | EXTRA_GN="use_jumbo_build=true jumbo_file_merge_limit=25" |
Update: the chromium ebuilds have USE=jumbo-build now, so you can leave the first variable out (or omit the entire thing and use the defaults, if you're brave).
Feel free to increase the number if you have the RAM to handle it - 25 is just a safe number I picked because it worked on my 2GB netbook. The default is 100, which has some gcc processes eating 2.5GB or more.
The improvements can range from several hours to half a day or more -
| Code: | $ qlop -vgH chromium
chromium-63.0.3239.59: Tue Nov 28 11:35:50 2017: 7 hours, 29 minutes, 8 seconds
chromium-64.0.3282.24: Fri Dec 22 04:12:14 2017: 5 hours, 57 minutes, 19 seconds
$ lscpu | grep P
CPU op-mode(s): 32-bit, 64-bit
CPU(s): 4
On-line CPU(s) list: 0-3
CPU family: 16
Model name: AMD Phenom(tm) II X4 20 Processor
CPU MHz: 800.000
CPU max MHz: 2800.0000
CPU min MHz: 800.0000
BogoMIPS: 5625.36 |
| Code: | $ qlop -vgH chromium
chromium-60.0.3112.24: Mon Jun 19 20:48:02 2017: 1 day, 2 hours, 8 minutes, 37 seconds
chromium-60.0.3112.24: Mon Jun 19 20:48:02 2017: 1 day, 2 hours, 8 minutes, 37 seconds
chromium-62.0.3202.18: Sat Sep 30 00:34:10 2017: 1 day, 2 hours, 50 minutes, 43 seconds
chromium-64.0.3282.24: Fri Dec 22 19:10:06 2017: 21 hours, 18 minutes, 46 seconds
# (not pictured: a regular compile of 64.0 that OOMed at the linker stage after 36 hours)
$ lscpu | grep P
CPU op-mode(s): 32-bit
CPU(s): 2
On-line CPU(s) list: 0,1
CPU family: 6
Model name: Intel(R) Atom(TM) CPU N270 @ 1.60GHz
CPU MHz: 800.000
CPU max MHz: 1600.0000
CPU min MHz: 800.0000
BogoMIPS: 3193.13 |
See also: bug 641786
Last edited by Ant P. on Thu Mar 01, 2018 8:01 am; edited 1 time in total |
|
| Back to top |
|
 |
Juippisi
Developer
Joined: 30 Sep 2005
Posts: 757
Location: /home
|
| Posted: Tue Jan 02, 2018 2:29 pm Post subject: |
|
|
| Thanks for bringing this out in the open, I wouldnt have found that bug report on my own. This seems to work, and is a huge improvement in compiling Chromium. |
|
| Back to top |
|
|
Marlo
Veteran
Joined: 26 Jul 2003
Posts: 1591
|
| Posted: Wed Jan 03, 2018 6:04 am Post subject: |
|
|
Yes it works.
At 100 it crashed but 75 gives:
| Code: |
# genlop -t chromium
* www-client/chromium
Wed Dec 20 08:13:44 2017 >>> www-client/chromium-64.0.3282.24
merge time: 4 hours, 17 minutes and 54 seconds.
Mon Dec 25 02:38:39 2017 >>> www-client/chromium-64.0.3282.39
merge time: 4 hours, 22 minutes and 55 seconds.
Wed Jan 3 02:22:42 2018 >>> www-client/chromium-64.0.3282.39
merge time: 1 hour, 59 minutes and 58 seconds. |
Can you explain to me in simple words how this works?
Ma
EDIT
Is it possible to do that with other programs?
Or in other words. Can it be used as a standard for everything?
_________________
------------------------------------------------------------------
http://radio.garden/ |
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Thu Jan 04, 2018 9:05 am Post subject: |
|
|
It concatenates 25/75/100 source files together at a time and compiles the result with a single GCC process. Apart from saving 99% of the overhead of starting up the compiler, it also means header files with multiple-include guards won't have to be processed over and over, and a lot more inter-function and dead code optimisation can be done at compile-time (the stuff usually reserved for -flto). Probably the latter thing is where most of the RAM usage comes from, but it also takes some off the final linking step.
It's hardcoded in chromium's build scripts, and is only possible because they already rewrote what autoconf/make does from scratch and control the entire compilation process. Making it work anywhere else would be pretty hard, definitely not something you can throw in make.conf and forget about. |
|
| Back to top |
|
|
Marlo
Veteran
Joined: 26 Jul 2003
Posts: 1591
|
| Posted: Thu Jan 04, 2018 9:21 am Post subject: |
|
|
Thanks Ant. P. I wanted to know that.
This tip is a fantastic time saver.
Thanks again
Ma
_________________
------------------------------------------------------------------
http://radio.garden/ |
|
| Back to top |
|
|
wxqc11h
n00b
Joined: 18 Sep 2007
Posts: 26
|
| Posted: Tue Jan 23, 2018 11:32 am Post subject: |
|
|
| Does it work for webkit-gtk too? Man merging it takes ages. |
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Wed Jan 24, 2018 6:01 am Post subject: |
|
|
| Nope, chromium-only. I don't recall Webkit-GTK having compile times as dire as this last time I had it installed though... was closer to firefox (minus rust) if anything. |
|
| Back to top |
|
|
Princess Nell
l33t
Joined: 15 Apr 2005
Posts: 927
|
| Posted: Sun Jan 28, 2018 9:09 pm Post subject: |
|
|
Off-topic here, but qlop must have serious bugs. Looking at emerge.log, genlop is quite correct.
| Code: |
# genlop -t webkit-gtk |tail -n 3
Tue Jan 16 00:04:25 2018 >>> net-libs/webkit-gtk-2.18.5
merge time: 2 hours, 40 minutes and 40 seconds.
# qlop -vgH webkit-gtk |tail -n 3
webkit-gtk-2.18.4: Thu Dec 28 22:31:26 2017: 2 hours, 39 minutes, 55 seconds
webkit-gtk-2.18.5: Sun Jan 14 18:22:37 2018: 1 day, 5 hours, 41 minutes, 48 seconds
webkit-gtk: 19 times
#
|
|
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Sun Jan 28, 2018 11:37 pm Post subject: |
|
|
| Yeah, I noticed the same. It works if you give it a very narrow date range but seems to drop recent entries otherwise. |
|
| Back to top |
|
|
Princess Nell
l33t
Joined: 15 Apr 2005
Posts: 927
|
| Posted: Wed Jan 31, 2018 7:59 am Post subject: |
|
|
Impressive. jumbo_file_merge_limit=60 (100/90/80/70 crashed).
| Code: |
Tue Jan 30 06:08:52 2018 >>> www-client/chromium-64.0.3282.123
merge time: 8 hours, 20 minutes and 21 seconds.
Wed Jan 31 03:18:11 2018 >>> www-client/chromium-64.0.3282.123
merge time: 4 hours, 7 minutes and 32 seconds.
|
|
|
| Back to top |
|
|
walkingcorpse
n00b
Joined: 24 Apr 2017
Posts: 21
Location: Europe
|
| Posted: Fri Feb 09, 2018 7:04 pm Post subject: |
|
|
jumbo_file_merge_limit=10 8GB ram and -j4. 
| Code: | Sat Jan 20 20:30:37 2018 >>> www-client/chromium-65.0.3315.3
merge time: 4 hours, 10 minutes and 39 seconds.
Fri Feb 9 05:26:53 2018 >>> www-client/chromium-65.0.3315.3
merge time: 2 hours, 5 minutes and 23 seconds |
About the jumbo_file_merge_limit from the Chromium-dev google forum:
| Quote: | | Eventually we could have 10 or 20 for people with many cores, 50 for people with few cores, 100 or infinity for single core/full compilation. |
|
|
| Back to top |
|
|
Section_8
l33t
Joined: 22 May 2004
Posts: 627
|
| Posted: Fri Feb 09, 2018 9:02 pm Post subject: |
|
|
I noticed the chromium ebuild has a jumbo-build use flag that must implement this:
| Code: | john@officepc ~ $ grep jumbo `equery which chromium`
IUSE="component-build cups gnome-keyring +hangouts jumbo-build kerberos neon pic +proprietary-codecs pulseaudio selinux +suid +system-ffmpeg +system-icu +system-libvpx +tcmalloc widevine"
# https://chromium.googlesource.com/chromium/src/+/lkcr/docs/jumbo.md
myconf_gn+=" use_jumbo_build=$(usex jumbo-build true false)" |
I wonder what file merge limit it uses. |
|
| Back to top |
|
|
walkingcorpse
n00b
Joined: 24 Apr 2017
Posts: 21
Location: Europe
|
|
| Back to top |
|
|
ceric35
Tux's lil' helper
Joined: 27 Aug 2006
Posts: 115
|
| Posted: Thu Nov 14, 2019 11:02 am Post subject: |
|
|
| walkingcorpse wrote: | | https://chromium.googlesource.com/chromium/src/+/master/docs/jumbo.md#tuning |
This links now says "Jumbo / Unity builds are no longer supported in Chromium."
EDIT : use has been removed from chromium-80 |
|
| Back to top |
|
|
Chiitoo
Administrator
Joined: 28 Feb 2010
Posts: 2742
Location: Here and Away Again
|
| Posted: Thu Nov 14, 2019 1:34 pm Post subject: |
|
|
| ceric35 wrote: | | walkingcorpse wrote: | | https://chromium.googlesource.com/chromium/src/+/master/docs/jumbo.md#tuning |
This links now says "Jumbo / Unity builds are no longer supported in Chromium."
EDIT : use has been removed from chromium-80 |
Yep.
- https://chromium.googlesource.com/chromium/src/+/6e1ce9f4
| chromium/src/6e1ce9f4 wrote: | Force jumbo builds to fail.
We no longer have any builders supporting the jumbo build config,
and it's time to now update all of the build files to remove
the jumbo templates. Before we do that, though, it's probably good
to figure out how to handle existing uses of the jumbo builds.
This CL asserts that |use_jumbo_build| is set to false. Once
we've run with this for a while (to ensure that people have
adjusted their configs), we can probably update any CI build
configs to remove the use of it and the other jumbo-related GN
args and then remove the actual declare_args() of the three
jumbo args, turning uses of them into warnings. |
Also
- https://groups.google.com/a/chromium.org/forum/#!topic/chromium-dev/DP9TQszzQLI/discussion
| Dirk wrote: | Hi all,
We have decided to remove support for the jumbo (aka unity) build configuration from the project.
We are very conscious of the fact that Chromium is very expensive and slow to compile, and that that represents a real barrier to contributing.
However, based on the experience we've gained with this configuration over the past many months it's been supported, we've reached the conclusion that the way jumbo is currently implemented, it interferes too much with the way people frequently write C++ code; you have to work around jumbo-specific conflicts in a way that can be awkward, unnatural, or otherwise unnecessary. As a result, we feel that the cost to the project of maintaining jumbo is not worth the benefit it is providing.
We continue to invest in other approaches to reducing build time. We've released the Goma build accelerator code and hope to continue making it easier to use for others. We are also working to enable our Goma clusters for external contributors (anyone who has tryjob access) who don't have their own build acceleration tools. We're starting with Linux support now and will have more to share on that soon. If you are interested in getting access to this system, please fill out this form. We're also working on Clang and LLVM and have made significant improvements this year to cycle time as a result, and we expect to continue to do so for the foreseeable future.
We plan to remove support a week from now, on August 22nd.
Please let me know if you have questions about this.
-- Dirk |
_________________
Kindest of regardses. |
|
| Back to top |
|
|
Goverp
Advocate
Joined: 07 Mar 2007
Posts: 2191
|
| Posted: Fri Nov 15, 2019 8:58 am Post subject: |
|
|
| ceric35 wrote: |
...
This links now says "Jumbo / Unity builds are no longer supported in Chromium."
EDIT : use has been removed from chromium-80 |
Arrggghhhh! I don't run chromium for many reasons, but KDE uses qtwebengine in several places. Since qtwebengine is based on chromium, presumably it too will stop supporting jumbo build, and I'll be stuck with a 24hr or longer emerge. 
Pretty please, can someone start a qtwebengine-bin package as an alternative?
_________________
Greybeard |
|
| Back to top |
|
|
Chiitoo
Administrator
Joined: 28 Feb 2010
Posts: 2742
Location: Here and Away Again
|
| Posted: Fri Nov 15, 2019 9:23 am Post subject: |
|
|
| Goverp wrote: | | Pretty please, can someone start a qtwebengine-bin package as an alternative? |
I would, if it were a simple thing to do. Or at least not as much work as it likely will be.
See also: Bug 669082 - dev-qt/qtwebengine-bin package
_________________
Kindest of regardses. |
|
| Back to top |
|
|
jserink
Veteran
Joined: 30 Jan 2004
Posts: 1013
|
| Posted: Mon Jan 13, 2020 6:13 am Post subject: |
|
|
| ceric35 wrote: | | walkingcorpse wrote: | | https://chromium.googlesource.com/chromium/src/+/master/docs/jumbo.md#tuning |
This links now says "Jumbo / Unity builds are no longer supported in Chromium."
EDIT : use has been removed from chromium-80 |
That explains this:
Thu Sep 12 11:28:19 2019 >>> www-client/chromium-77.0.3865.75
merge time: 1 hour, 48 minutes and 54 seconds.
Mon Sep 30 17:36:23 2019 >>> www-client/chromium-77.0.3865.90
merge time: 1 hour, 42 minutes and 55 seconds.
Tue Oct 22 11:36:41 2019 >>> www-client/chromium-77.0.3865.90
merge time: 2 hours, 17 minutes and 43 seconds.
Thu Oct 24 11:14:47 2019 >>> www-client/chromium-77.0.3865.120
merge time: 2 hours, 14 minutes and 32 seconds.
Mon Oct 28 02:08:33 2019 >>> www-client/chromium-78.0.3904.70
merge time: 1 hour, 46 minutes and 52 seconds.
Mon Nov 4 14:22:45 2019 >>> www-client/chromium-78.0.3904.87
merge time: 1 hour, 56 minutes and 24 seconds.
Wed Nov 13 21:55:30 2019 >>> www-client/chromium-78.0.3904.97
merge time: 1 hour, 38 minutes and 34 seconds.
Mon Nov 25 11:35:55 2019 >>> www-client/chromium-78.0.3904.108
merge time: 3 hours, 3 minutes and 9 seconds.
Tue Dec 24 22:14:58 2019 >>> www-client/chromium-79.0.3945.88
merge time: 4 hours, 55 minutes and 24 seconds.
Sat Jan 4 13:47:48 2020 >>> www-client/chromium-79.0.3945.88
merge time: 5 hours, 7 minutes and 43 seconds.
Fri Jan 10 01:14:34 2020 >>> www-client/chromium-79.0.3945.88
merge time: 5 hours, 2 minutes and 44 seconds.
I was wondering why it all of a sudden became 4 hours from 1+.
Now I know.
Cheers,
john
Last edited by jserink on Tue Jan 14, 2020 12:07 am; edited 1 time in total |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20543
|
| Posted: Mon Jan 13, 2020 5:13 pm Post subject: |
|
|
Moved from Documentation, Tips & Tricks to Gentoo Chat since this is no longer a possibility due to upstream.
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Mon Jan 13, 2020 8:24 pm Post subject: |
|
|
| webkit-gtk seems to have sprouted a USE=jumbo-build recently, but I haven't tested it as it doesn't compile for other reasons. |
|
| Back to top |
|
|
nilreM
n00b
Joined: 22 Nov 2007
Posts: 23
Location: Voronezh
|
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
|
| Back to top |
|
|
gorg86
Guru
Joined: 20 May 2011
Posts: 320
|
| Posted: Thu Feb 13, 2020 3:26 am Post subject: |
|
|
Here are my compile times (FX-8350 16GB RAM, never used jumbo build):
https://pastebin.com/DYThWt6C |
|
| Back to top |
|
|
|
Gentoo Chat
