| View previous topic :: View next topic |
| Author |
Message |
mi_unixbird
Tux's lil' helper
Joined: 24 Jul 2015
Posts: 125
|
 Posted: Fri Sep 13, 2024 1:05 am Post subject: Chromium compile doesn't seem to be using ccache |
 |
|
| Code: | —— — ccache -s
Cacheable calls: 2085603 / 3286807 (63.45%)
Hits: 482121 / 2085603 (23.12%)
Direct: 240773 / 482121 (49.94%)
Preprocessed: 241348 / 482121 (50.06%)
Misses: 1603482 / 2085603 (76.88%)
Uncacheable calls: 499694 / 3286807 (15.20%)
Errors: 701510 / 3286807 (21.34%)
Local storage:
Cache size (GB): 30.0 / 30.0 (99.98%)
Cleanups: 384
Hits: 482121 / 2085603 (23.12%)
Misses: 1603482 / 2085603 (76.88%)
—— — sudo emerge -1 chromium
[ebuild U ] www-client/chromium-128.0.6613.137 [128.0.6613.119]
Would you like to merge these packages? [Yes/No]
>>> Verifying ebuild manifests
>>> Running pre-merge checks for www-client/chromium-128.0.6613.137
* Checking for at least 4 GiB RAM ... [ ok ]
* Checking for at least 25 GiB disk space at "/var/tmp/build/portage/www-client/chromium-128.0.6613.137/temp" ... [ ok ]
>>> Emerging (1 of 1) www-client/chromium-128.0.6613.137::gentoo
>>> Installing (1 of 1) www-client/chromium-128.0.6613.137::gentoo
>>> Completed (1 of 1) www-client/chromium-128.0.6613.137::gentoo
* Messages for package www-client/chromium-128.0.6613.137:
* PYTHONPATH='/home/user/.local/lib/Python'
* VA-API is disabled by default at runtime. You have to enable it
* by adding --enable-features=VaapiVideoDecoder to CHROMIUM_FLAGS
* in /etc/chromium/default.
—— — ccache -s
Cacheable calls: 2085603 / 3326157 (62.70%)
Hits: 482121 / 2085603 (23.12%)
Direct: 240773 / 482121 (49.94%)
Preprocessed: 241348 / 482121 (50.06%)
Misses: 1603482 / 2085603 (76.88%)
Uncacheable calls: 499760 / 3326157 (15.03%)
Errors: 740794 / 3326157 (22.27%)
Local storage:
Cache size (GB): 30.0 / 30.0 (99.98%)
Cleanups: 384
Hits: 482121 / 2085603 (23.12%)
Misses: 1603482 / 2085603 (76.88%)
|
The way I interpret this with the “cachable calls” number remaining identical at 2085603 befoe and after is that not a single call was “cachable” which seems weird. I won't profess to exactly know what al this means but I assume it means that it somehow can't use ccache? Does this have something to do with some kind of custom build system it uses?
Also, for what it's worth:
| Code: |
—— — ls -l /usr/lib/ccache/bin/
total 104
lrwxrwxrwx 1 root root 15 Sep 7 04:48 c++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 c99 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 cc -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 clang -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 clang++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 clang++-18 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 clang-18 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 g++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 g++-14 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 gcc -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 gcc-14 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 i686-pc-linux-gnu-clang -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 i686-pc-linux-gnu-clang++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 i686-pc-linux-gnu-clang++-18 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 i686-pc-linux-gnu-clang-18 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 x86_64-pc-linux-gnu-c++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 x86_64-pc-linux-gnu-cc -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 x86_64-pc-linux-gnu-clang -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 x86_64-pc-linux-gnu-clang++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 x86_64-pc-linux-gnu-clang++-18 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 x86_64-pc-linux-gnu-clang-18 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 x86_64-pc-linux-gnu-g++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 x86_64-pc-linux-gnu-g++-14 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 x86_64-pc-linux-gnu-gcc -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 x86_64-pc-linux-gnu-gcc-14 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep 7 04:48 zvbi-ntsc-cc -> /usr/bin/ccache
|
_________________
execctl --path exec filectl --current-directory list |
|
| Back to top |
|
 |
mi_unixbird
Tux's lil' helper
Joined: 24 Jul 2015
Posts: 125
|
| Posted: Fri Sep 13, 2024 4:07 pm Post subject: |
|
|
In case anyone later stumbles upon this thread. I solved the issue, more or less, “ccache -sv” revealed that the error was “compiler check failed” so I removed the “compiler_check” configuration option in distcc.conf. This now does apparently mean that a re-installation of the same compiler version will invalidate the cache but I never had that.
_________________
execctl --path exec filectl --current-directory list |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2206
|
| Posted: Fri Sep 13, 2024 5:12 pm Post subject: |
|
|
I spent a lot of time on that back when I had a dual core CPU with maximum limit of 8BG RAM which I didn't even have. Chromium doesn't benefit from ccache either way unless you recompile one and the same version.
Best Regards,
Georgi |
|
| Back to top |
|
|
mi_unixbird
Tux's lil' helper
Joined: 24 Jul 2015
Posts: 125
|
| Posted: Sat Sep 14, 2024 1:56 am Post subject: |
|
|
| logrusx wrote: | I spent a lot of time on that back when I had a dual core CPU with maximum limit of 8BG RAM which I didn't even have. Chromium doesn't benefit from ccache either way unless you recompile one and the same version.
Best Regards,
Georgi |
I remember Chromium benefiting immensely from ccache wen it still worked for me and it cutting down the compilation times from 8 hours to 2.5 hours:
| Code: |
Mon Sep 4 05:40:19 2023 >>> www-client/chromium-116.0.5845.140
merge time: 7 hours, 54 minutes and 13 seconds.
Wed Sep 13 09:39:21 2023 >>> www-client/chromium-116.0.5845.187
merge time: 2 hours, 19 minutes and 58 seconds.
Wed Oct 4 07:36:08 2023 >>> www-client/chromium-117.0.5938.132
merge time: 7 hours, 53 minutes and 23 seconds.
Fri Oct 13 05:29:07 2023 >>> www-client/chromium-117.0.5938.149
merge time: 2 hours, 47 minutes and 43 seconds.
|
I read a lot of things on the internet about ccache supposedly not doing much and only slowing down, all of those posts not giving any numbers but theoretical reasons like that cache misses slow things down, which should be true, but it should also be completely trivial compared to cache hits because hashes are incredibly cheap compared to actual compilation. The numbers on my system at least speak for themselves I might do an actual test later with a smaller package than chromium which is of course immense but these numbers above with chromium while a compiler update happened twice speak for tthemselves. In fact, ccache makes it so much faster oe could argue it's worth it to not update compilers for a considerable time merely for the sheer compilation speed benefits it brings.
Anyway, I found the ultimate source now. The compiler_check was set to “%compiler -v%” but on clang that does't return the version at all like on GCC but is the verbose output flag and it errors out when not given anything to copile of course; I wonder when that change happened. I honestly have been thinking for a while now that chromium actually got this ridiculously big with ccache that it took 9 hours now with it, not realizing that it wasn't uing it at all but still had to do a copiler check for every file and was only slowing it down.
_________________
execctl --path exec filectl --current-directory list |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2206
|
| Posted: Sat Sep 14, 2024 3:23 am Post subject: |
|
|
I haven't compiled chromium for a long time, it still used GCC when I did. Back then it was written in a way ccache wasn't doing much. It used to take around two and a half hours to compile, but with ccache it was taking maybe 15 minutes more when the same version is recompiled and it had very high hit rates.
I then switched to ungoogled-chromium-bin, but ultimately I switched to Firefox.
Best Regards,
Georgi |
|
| Back to top |
|
|
mi_unixbird
Tux's lil' helper
Joined: 24 Jul 2015
Posts: 125
|
| Posted: Sat Sep 14, 2024 4:56 am Post subject: |
|
|
| logrusx wrote: | | I haven't compiled chromium for a long time, it still used GCC when I did. Back then it was written in a way ccache wasn't doing much. It used to take around two and a half hours to compile, but with ccache it was taking maybe 15 minutes more when the same version is recompiled and it had very high hit rates. |
How does this work? Surely the number of cache hits is really the only thing that matters for time reduction?
We'll see what difference it brings with the next update though.
And yes, I still remember when Chromium took a mere two hours without cccache. I really wondner what happened. I've been considering switching to Firefox purely for the compilation speed advantage but from what I read it's less snappy and I would need to redo all my settings. The Chomium distfiles are also 7 GiB by now.
_________________
execctl --path exec filectl --current-directory list |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2206
|
| Posted: Sat Sep 14, 2024 6:59 am Post subject: |
|
|
| mi_unixbird wrote: | | logrusx wrote: | | I haven't compiled chromium for a long time, it still used GCC when I did. Back then it was written in a way ccache wasn't doing much. It used to take around two and a half hours to compile, but with ccache it was taking maybe 15 minutes more when the same version is recompiled and it had very high hit rates. |
How does this work? Surely the number of cache hits is really the only thing that matters for time reduction? |
Disk IO. Waiting for SSD is still an endless wait for the CPU. Ironically enough, it was faster to compile the files than looking them up from the cache on the disk.
| mi_unixbird wrote: | | And yes, I still remember when Chromium took a mere two hours without cccache. |
Don't get me wrong, it took 2 and a half hours on a Ryzen 7 5800H with 32GB RAM in a cold room 3 and a half years ago. On my old dual core computer it took more like 7 hours with jumbo build back when they supported it and maybe 40 when they stopped supporting it more than 4 years ago. I used to use ccache as a means of resuming the build process because it was fragile back then and was breaking when resuming with keepwork feature enabled. This way I could compile for several consecutive nights while still being able to use my computer through the day. Later I discovered a not guaranteed to work fix was to delete .setuped in the temp directory, so that portage was forced to recreate the environment and successfully resume. I think they fix that too later.
I would test it now to see how it works but it wants to recompile nodejs as well and I don't want to go down that route and compile at least 6-7 hours just to test.
Best Regards,
Georgi |
|
| Back to top |
|
|
mi_unixbird
Tux's lil' helper
Joined: 24 Jul 2015
Posts: 125
|
| Posted: Sat Sep 14, 2024 8:57 am Post subject: |
|
|
| logrusx wrote: | | mi_unixbird wrote: | | logrusx wrote: | | I haven't compiled chromium for a long time, it still used GCC when I did. Back then it was written in a way ccache wasn't doing much. It used to take around two and a half hours to compile, but with ccache it was taking maybe 15 minutes more when the same version is recompiled and it had very high hit rates. |
How does this work? Surely the number of cache hits is really the only thing that matters for time reduction? |
Disk IO. Waiting for SSD is still an endless wait for the CPU. Ironically enough, it was faster to compile the files than looking them up from the cache on the disk. |
This doesn't make any sense to me. New compilations have to write the files to the drive somehow which is far slower while ccache only compares hashes of identical compiler and compilation options stored for each cached file. I would assume that if it copy files, it's smart enough to use reflink=auto which is practically free on a copy-on-write filesystem.
| Quote: | | Don't get me wrong, it took 2 and a half hours on a Ryzen 7 5800H with 32GB RAM in a cold room 3 and a half years ago. On my old dual core computer it took more like 7 hours with jumbo build back when they supported it and maybe 40 when they stopped supporting it more than 4 years ago. I used to use ccache as a means of resuming the build process because it was fragile back then and was breaking when resuming with keepwork feature enabled. This way I could compile for several consecutive nights while still being able to use my computer through the day. Later I discovered a not guaranteed to work fix was to delete .setuped in the temp directory, so that portage was forced to recreate the environment and successfully resume. I think they fix that too later. |
Two hours for me was when I first installed it, which was about a decade ago I'd say. On my machine now, it's about 8-9 hours now without the cache at least, but suffering the pœnalty of that it tries it at least. With the next update I will know how much time ccache cuts again but the numbers of even early this year look very promising with 2.5 hour compile times being the norm opposed to the 8 they take now.
_________________
execctl --path exec filectl --current-directory list |
|
| Back to top |
|
|
Chiitoo
Administrator
Joined: 28 Feb 2010
Posts: 2683
Location: Here and Away Again
|
| Posted: Sat Sep 14, 2024 12:44 pm Post subject: |
|
|
I believe the last version of Chromium to have USE="jumbo-build" was 78.0.3904.108, which was removed back in 2019 [1].
That flag, when enabled, would usually cut down the build time to half, while using more RAM.
Aside from that, I think Chromium just got bigger.
(Qt still maintains 'jumbo-build' for their fork of Chromium, so one can someawhats test the difference using that still.)
| Code: | $ genlop -t chromium
* www-client/chromium
Thu May 8 01:39:04 2014 >>> www-client/chromium-35.0.1916.86
merge time: 42 minutes and 38 seconds.
Mon Mar 16 09:00:10 2015 >>> www-client/chromium-41.0.2272.76
merge time: 1 hour, 27 minutes and 29 seconds.
Sat Jan 28 02:31:53 2023 >>> www-client/chromium-110.0.5481.38
merge time: 6 hours, 44 minutes and 37 seconds.
Sat May 6 16:19:52 2023 >>> www-client/chromium-113.0.5672.63
merge time: 1 hour, 22 minutes and 12 seconds.
Mon May 15 10:20:34 2023 >>> www-client/chromium-113.0.5672.63
merge time: 11 seconds.
Wed May 17 09:34:44 2023 >>> www-client/chromium-113.0.5672.92
merge time: 1 hour, 33 minutes and 59 seconds.
Fri May 26 10:58:24 2023 >>> www-client/chromium-113.0.5672.126
merge time: 1 hour, 39 minutes and 19 seconds.
Sat May 27 14:49:57 2023 >>> www-client/chromium-113.0.5672.126
merge time: 8 minutes and 24 seconds.
Sat May 27 21:07:17 2023 >>> www-client/chromium-113.0.5672.126
merge time: 1 hour, 19 minutes and 29 seconds.
Sun Jun 11 22:36:50 2023 >>> www-client/chromium-114.0.5735.110
merge time: 1 hour, 41 minutes and 40 seconds. |
The builds before 2023 were using an AMD Phenom II X6 1090T, while the later builds used AMD Ryzen 9 7950X (don't mind the 6-hour one; something special definitely was going on there).
I never really tested it with Chromium, but perhaps 'ccache' is still useful when moving between versions such as 116.0.5845.140 and 116.0.5845.187, but yeah, definitely seen that it has never been much, if any use between major versions.
1. https://gitweb.gentoo.org/repo/gentoo.git/commit/www-client/chromium?id=0cd82be6bd5b1b1de7a3a830dc5ef3517e346f15
_________________
Kindest of regardses. |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2206
|
| Posted: Sat Sep 14, 2024 1:23 pm Post subject: |
|
|
| Chiitoo wrote: |
I never really tested it with Chromium, but perhaps 'ccache' is still useful when moving between versions such as 116.0.5845.140 and 116.0.5845.187, but yeah, definitely seen that it has never been much, if any use between major versions.
|
It hasn't been much even with minor versions. They used to make changes that would result in untouched files giving completely different result after being preprocessed so ccache was just a waste. I hesitate to argue it still is because although I don't think something changed in the development of chromium, I have seen lucky runs which decreased time significantly in the past but were one-offs. Who knows, they may really have improved their development process.
Best Regards,
Georgi |
|
| Back to top |
|
|
mi_unixbird
Tux's lil' helper
Joined: 24 Jul 2015
Posts: 125
|
| Posted: Sun Sep 15, 2024 10:30 am Post subject: |
|
|
Well, we'll see what changes with the next compile, but those three hour build times of Chromium I still had earlier this year before ccache failed do certainly look attractive. We'll see if it go back to that but it's hard to argue with theory against the statistics that in my genlop -t logs of chromium there are 3 hour build times [ccache succeeded], 8 hour build times [ccache didn't succeed due to say a compiler update or the recent problem], and pretty much nothing in between.
_________________
execctl --path exec filectl --current-directory list |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
