Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Chromium compile doesn't seem to be using ccache
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
mi_unixbird
Tux's lil' helper
Tux's lil' helper


Joined: 24 Jul 2015
Posts: 130

PostPosted: Fri Sep 13, 2024 1:05 am    Post subject: Chromium compile doesn't seem to be using ccache Reply with quote

Code:
—— — ccache -s
Cacheable calls:   2085603 / 3286807 (63.45%)
  Hits:             482121 / 2085603 (23.12%)
    Direct:         240773 /  482121 (49.94%)
    Preprocessed:   241348 /  482121 (50.06%)
  Misses:          1603482 / 2085603 (76.88%)
Uncacheable calls:  499694 / 3286807 (15.20%)
Errors:             701510 / 3286807 (21.34%)
Local storage:
  Cache size (GB):    30.0 /    30.0 (99.98%)
  Cleanups:            384
  Hits:             482121 / 2085603 (23.12%)
  Misses:          1603482 / 2085603 (76.88%)
 —— — sudo emerge -1 chromium
[ebuild     U ] www-client/chromium-128.0.6613.137 [128.0.6613.119]

Would you like to merge these packages? [Yes/No]
>>> Verifying ebuild manifests
>>> Running pre-merge checks for www-client/chromium-128.0.6613.137
 * Checking for at least 4 GiB RAM ...                                                     [ ok ]
 * Checking for at least 25 GiB disk space at "/var/tmp/build/portage/www-client/chromium-128.0.6613.137/temp" ...                                                                           [ ok ]
>>> Emerging (1 of 1) www-client/chromium-128.0.6613.137::gentoo
>>> Installing (1 of 1) www-client/chromium-128.0.6613.137::gentoo
>>> Completed (1 of 1) www-client/chromium-128.0.6613.137::gentoo

 * Messages for package www-client/chromium-128.0.6613.137:

 * PYTHONPATH='/home/user/.local/lib/Python'
 * VA-API is disabled by default at runtime. You have to enable it
 * by adding --enable-features=VaapiVideoDecoder to CHROMIUM_FLAGS
 * in /etc/chromium/default.
 —— — ccache -s
Cacheable calls:   2085603 / 3326157 (62.70%)
  Hits:             482121 / 2085603 (23.12%)
    Direct:         240773 /  482121 (49.94%)
    Preprocessed:   241348 /  482121 (50.06%)
  Misses:          1603482 / 2085603 (76.88%)
Uncacheable calls:  499760 / 3326157 (15.03%)
Errors:             740794 / 3326157 (22.27%)
Local storage:
  Cache size (GB):    30.0 /    30.0 (99.98%)
  Cleanups:            384
  Hits:             482121 / 2085603 (23.12%)
  Misses:          1603482 / 2085603 (76.88%)


The way I interpret this with the “cachable calls” number remaining identical at 2085603 befoe and after is that not a single call was “cachable” which seems weird. I won't profess to exactly know what al this means but I assume it means that it somehow can't use ccache? Does this have something to do with some kind of custom build system it uses?

Also, for what it's worth:

Code:

 —— — ls -l /usr/lib/ccache/bin/
total 104
lrwxrwxrwx 1 root root 15 Sep  7 04:48 c++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 c99 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 cc -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 clang -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 clang++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 clang++-18 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 clang-18 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 g++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 g++-14 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 gcc -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 gcc-14 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 i686-pc-linux-gnu-clang -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 i686-pc-linux-gnu-clang++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 i686-pc-linux-gnu-clang++-18 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 i686-pc-linux-gnu-clang-18 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 x86_64-pc-linux-gnu-c++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 x86_64-pc-linux-gnu-cc -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 x86_64-pc-linux-gnu-clang -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 x86_64-pc-linux-gnu-clang++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 x86_64-pc-linux-gnu-clang++-18 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 x86_64-pc-linux-gnu-clang-18 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 x86_64-pc-linux-gnu-g++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 x86_64-pc-linux-gnu-g++-14 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 x86_64-pc-linux-gnu-gcc -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 x86_64-pc-linux-gnu-gcc-14 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 zvbi-ntsc-cc -> /usr/bin/ccache

_________________
execctl --path exec filectl --current-directory list
Back to top
View user's profile Send private message
mi_unixbird
Tux's lil' helper
Tux's lil' helper


Joined: 24 Jul 2015
Posts: 130

PostPosted: Fri Sep 13, 2024 4:07 pm    Post subject: Reply with quote

In case anyone later stumbles upon this thread. I solved the issue, more or less, “ccache -sv” revealed that the error was “compiler check failed” so I removed the “compiler_check” configuration option in distcc.conf. This now does apparently mean that a re-installation of the same compiler version will invalidate the cache but I never had that.
_________________
execctl --path exec filectl --current-directory list
Back to top
View user's profile Send private message
logrusx
Advocate
Advocate


Joined: 22 Feb 2018
Posts: 2418

PostPosted: Fri Sep 13, 2024 5:12 pm    Post subject: Reply with quote

I spent a lot of time on that back when I had a dual core CPU with maximum limit of 8BG RAM which I didn't even have. Chromium doesn't benefit from ccache either way unless you recompile one and the same version.

Best Regards,
Georgi
Back to top
View user's profile Send private message
mi_unixbird
Tux's lil' helper
Tux's lil' helper


Joined: 24 Jul 2015
Posts: 130

PostPosted: Sat Sep 14, 2024 1:56 am    Post subject: Reply with quote

logrusx wrote:
I spent a lot of time on that back when I had a dual core CPU with maximum limit of 8BG RAM which I didn't even have. Chromium doesn't benefit from ccache either way unless you recompile one and the same version.

Best Regards,
Georgi
I remember Chromium benefiting immensely from ccache wen it still worked for me and it cutting down the compilation times from 8 hours to 2.5 hours:

Code:

     Mon Sep  4 05:40:19 2023 >>> www-client/chromium-116.0.5845.140
       merge time: 7 hours, 54 minutes and 13 seconds.

     Wed Sep 13 09:39:21 2023 >>> www-client/chromium-116.0.5845.187
       merge time: 2 hours, 19 minutes and 58 seconds.

     Wed Oct  4 07:36:08 2023 >>> www-client/chromium-117.0.5938.132
       merge time: 7 hours, 53 minutes and 23 seconds.

     Fri Oct 13 05:29:07 2023 >>> www-client/chromium-117.0.5938.149
       merge time: 2 hours, 47 minutes and 43 seconds.


I read a lot of things on the internet about ccache supposedly not doing much and only slowing down, all of those posts not giving any numbers but theoretical reasons like that cache misses slow things down, which should be true, but it should also be completely trivial compared to cache hits because hashes are incredibly cheap compared to actual compilation. The numbers on my system at least speak for themselves I might do an actual test later with a smaller package than chromium which is of course immense but these numbers above with chromium while a compiler update happened twice speak for tthemselves. In fact, ccache makes it so much faster oe could argue it's worth it to not update compilers for a considerable time merely for the sheer compilation speed benefits it brings.

Anyway, I found the ultimate source now. The compiler_check was set to “%compiler -v%” but on clang that does't return the version at all like on GCC but is the verbose output flag and it errors out when not given anything to copile of course; I wonder when that change happened. I honestly have been thinking for a while now that chromium actually got this ridiculously big with ccache that it took 9 hours now with it, not realizing that it wasn't uing it at all but still had to do a copiler check for every file and was only slowing it down.
_________________
execctl --path exec filectl --current-directory list
Back to top
View user's profile Send private message
logrusx
Advocate
Advocate


Joined: 22 Feb 2018
Posts: 2418

PostPosted: Sat Sep 14, 2024 3:23 am    Post subject: Reply with quote

I haven't compiled chromium for a long time, it still used GCC when I did. Back then it was written in a way ccache wasn't doing much. It used to take around two and a half hours to compile, but with ccache it was taking maybe 15 minutes more when the same version is recompiled and it had very high hit rates.

I then switched to ungoogled-chromium-bin, but ultimately I switched to Firefox.

Best Regards,
Georgi
Back to top
View user's profile Send private message
mi_unixbird
Tux's lil' helper
Tux's lil' helper


Joined: 24 Jul 2015
Posts: 130

PostPosted: Sat Sep 14, 2024 4:56 am    Post subject: Reply with quote

logrusx wrote:
I haven't compiled chromium for a long time, it still used GCC when I did. Back then it was written in a way ccache wasn't doing much. It used to take around two and a half hours to compile, but with ccache it was taking maybe 15 minutes more when the same version is recompiled and it had very high hit rates.
How does this work? Surely the number of cache hits is really the only thing that matters for time reduction?

We'll see what difference it brings with the next update though.

And yes, I still remember when Chromium took a mere two hours without cccache. I really wondner what happened. I've been considering switching to Firefox purely for the compilation speed advantage but from what I read it's less snappy and I would need to redo all my settings. The Chomium distfiles are also 7 GiB by now.
_________________
execctl --path exec filectl --current-directory list
Back to top
View user's profile Send private message
logrusx
Advocate
Advocate


Joined: 22 Feb 2018
Posts: 2418

PostPosted: Sat Sep 14, 2024 6:59 am    Post subject: Reply with quote

mi_unixbird wrote:
logrusx wrote:
I haven't compiled chromium for a long time, it still used GCC when I did. Back then it was written in a way ccache wasn't doing much. It used to take around two and a half hours to compile, but with ccache it was taking maybe 15 minutes more when the same version is recompiled and it had very high hit rates.
How does this work? Surely the number of cache hits is really the only thing that matters for time reduction?


Disk IO. Waiting for SSD is still an endless wait for the CPU. Ironically enough, it was faster to compile the files than looking them up from the cache on the disk.

mi_unixbird wrote:
And yes, I still remember when Chromium took a mere two hours without cccache.


Don't get me wrong, it took 2 and a half hours on a Ryzen 7 5800H with 32GB RAM in a cold room 3 and a half years ago. On my old dual core computer it took more like 7 hours with jumbo build back when they supported it and maybe 40 when they stopped supporting it more than 4 years ago. I used to use ccache as a means of resuming the build process because it was fragile back then and was breaking when resuming with keepwork feature enabled. This way I could compile for several consecutive nights while still being able to use my computer through the day. Later I discovered a not guaranteed to work fix was to delete .setuped in the temp directory, so that portage was forced to recreate the environment and successfully resume. I think they fix that too later.

I would test it now to see how it works but it wants to recompile nodejs as well and I don't want to go down that route and compile at least 6-7 hours just to test.

Best Regards,
Georgi
Back to top
View user's profile Send private message
mi_unixbird
Tux's lil' helper
Tux's lil' helper


Joined: 24 Jul 2015
Posts: 130

PostPosted: Sat Sep 14, 2024 8:57 am    Post subject: Reply with quote

logrusx wrote:
mi_unixbird wrote:
logrusx wrote:
I haven't compiled chromium for a long time, it still used GCC when I did. Back then it was written in a way ccache wasn't doing much. It used to take around two and a half hours to compile, but with ccache it was taking maybe 15 minutes more when the same version is recompiled and it had very high hit rates.
How does this work? Surely the number of cache hits is really the only thing that matters for time reduction?


Disk IO. Waiting for SSD is still an endless wait for the CPU. Ironically enough, it was faster to compile the files than looking them up from the cache on the disk.


This doesn't make any sense to me. New compilations have to write the files to the drive somehow which is far slower while ccache only compares hashes of identical compiler and compilation options stored for each cached file. I would assume that if it copy files, it's smart enough to use reflink=auto which is practically free on a copy-on-write filesystem.

Quote:
Don't get me wrong, it took 2 and a half hours on a Ryzen 7 5800H with 32GB RAM in a cold room 3 and a half years ago. On my old dual core computer it took more like 7 hours with jumbo build back when they supported it and maybe 40 when they stopped supporting it more than 4 years ago. I used to use ccache as a means of resuming the build process because it was fragile back then and was breaking when resuming with keepwork feature enabled. This way I could compile for several consecutive nights while still being able to use my computer through the day. Later I discovered a not guaranteed to work fix was to delete .setuped in the temp directory, so that portage was forced to recreate the environment and successfully resume. I think they fix that too later.
Two hours for me was when I first installed it, which was about a decade ago I'd say. On my machine now, it's about 8-9 hours now without the cache at least, but suffering the pœnalty of that it tries it at least. With the next update I will know how much time ccache cuts again but the numbers of even early this year look very promising with 2.5 hour compile times being the norm opposed to the 8 they take now.
_________________
execctl --path exec filectl --current-directory list
Back to top
View user's profile Send private message
Chiitoo
Administrator
Administrator


Joined: 28 Feb 2010
Posts: 2728
Location: Here and Away Again

PostPosted: Sat Sep 14, 2024 12:44 pm    Post subject: Reply with quote

I believe the last version of Chromium to have USE="jumbo-build" was 78.0.3904.108, which was removed back in 2019 [1].

That flag, when enabled, would usually cut down the build time to half, while using more RAM.

Aside from that, I think Chromium just got bigger.

(Qt still maintains 'jumbo-build' for their fork of Chromium, so one can someawhats test the difference using that still.)

Code:
$ genlop -t chromium
 * www-client/chromium

     Thu May  8 01:39:04 2014 >>> www-client/chromium-35.0.1916.86
       merge time: 42 minutes and 38 seconds.

     Mon Mar 16 09:00:10 2015 >>> www-client/chromium-41.0.2272.76
       merge time: 1 hour, 27 minutes and 29 seconds.

     Sat Jan 28 02:31:53 2023 >>> www-client/chromium-110.0.5481.38
       merge time: 6 hours, 44 minutes and 37 seconds.

     Sat May  6 16:19:52 2023 >>> www-client/chromium-113.0.5672.63
       merge time: 1 hour, 22 minutes and 12 seconds.

     Mon May 15 10:20:34 2023 >>> www-client/chromium-113.0.5672.63
       merge time: 11 seconds.

     Wed May 17 09:34:44 2023 >>> www-client/chromium-113.0.5672.92
       merge time: 1 hour, 33 minutes and 59 seconds.

     Fri May 26 10:58:24 2023 >>> www-client/chromium-113.0.5672.126
       merge time: 1 hour, 39 minutes and 19 seconds.

     Sat May 27 14:49:57 2023 >>> www-client/chromium-113.0.5672.126
       merge time: 8 minutes and 24 seconds.

     Sat May 27 21:07:17 2023 >>> www-client/chromium-113.0.5672.126
       merge time: 1 hour, 19 minutes and 29 seconds.

     Sun Jun 11 22:36:50 2023 >>> www-client/chromium-114.0.5735.110
       merge time: 1 hour, 41 minutes and 40 seconds.

The builds before 2023 were using an AMD Phenom II X6 1090T, while the later builds used AMD Ryzen 9 7950X (don't mind the 6-hour one; something special definitely was going on there).

I never really tested it with Chromium, but perhaps 'ccache' is still useful when moving between versions such as 116.0.5845.140 and 116.0.5845.187, but yeah, definitely seen that it has never been much, if any use between major versions.

1. https://gitweb.gentoo.org/repo/gentoo.git/commit/www-client/chromium?id=0cd82be6bd5b1b1de7a3a830dc5ef3517e346f15
_________________
Kindest of regardses.
Back to top
View user's profile Send private message
logrusx
Advocate
Advocate


Joined: 22 Feb 2018
Posts: 2418

PostPosted: Sat Sep 14, 2024 1:23 pm    Post subject: Reply with quote

Chiitoo wrote:

I never really tested it with Chromium, but perhaps 'ccache' is still useful when moving between versions such as 116.0.5845.140 and 116.0.5845.187, but yeah, definitely seen that it has never been much, if any use between major versions.


It hasn't been much even with minor versions. They used to make changes that would result in untouched files giving completely different result after being preprocessed so ccache was just a waste. I hesitate to argue it still is because although I don't think something changed in the development of chromium, I have seen lucky runs which decreased time significantly in the past but were one-offs. Who knows, they may really have improved their development process.

Best Regards,
Georgi
Back to top
View user's profile Send private message
mi_unixbird
Tux's lil' helper
Tux's lil' helper


Joined: 24 Jul 2015
Posts: 130

PostPosted: Sun Sep 15, 2024 10:30 am    Post subject: Reply with quote

Well, we'll see what changes with the next compile, but those three hour build times of Chromium I still had earlier this year before ccache failed do certainly look attractive. We'll see if it go back to that but it's hard to argue with theory against the statistics that in my genlop -t logs of chromium there are 3 hour build times [ccache succeeded], 8 hour build times [ccache didn't succeed due to say a compiler update or the recent problem], and pretty much nothing in between.
_________________
execctl --path exec filectl --current-directory list
Back to top
View user's profile Send private message
mi_unixbird
Tux's lil' helper
Tux's lil' helper


Joined: 24 Jul 2015
Posts: 130

PostPosted: Sat Sep 21, 2024 11:53 am    Post subject: Reply with quote

First cached update:

Code:

     Wed Sep 18 08:50:15 2024 >>> www-client/chromium-129.0.6668.42
       merge time: 10 hours, 29 minutes and 55 seconds.

     Sat Sep 21 13:32:19 2024 >>> www-client/chromium-129.0.6668.58
       merge time: 25 minutes and 58 seconds.


I would assume this was a very good case though since the version differences are very minor. I assume a major version update won't be quite this spectacular since as said, I remember 3 hour builds and I assume changing major dependencies in between also ruins quite a bit but at the very least the “best case” is absolutely spectacular.

In the meanwhile I looked at some statistics for ccache usage on big projects it seems to diverge from that 100% cache misses, as in first population increases compile time to 101% to 115% depending on the mode used while 100% cache hits, as in a direct rebuild, reduces it to 0.5%-5% for most big projects. In general the more preprocessor intensive a project is the lower the benefits of ccache are but it seems to me that even with 10% hits on average it should speed up.
_________________
execctl --path exec filectl --current-directory list
Back to top
View user's profile Send private message
logrusx
Advocate
Advocate


Joined: 22 Feb 2018
Posts: 2418

PostPosted: Fri Sep 27, 2024 12:18 pm    Post subject: Reply with quote

I still can't believe chromium build makes good use of ccache. There's a new 129 version available. Also 130. I'm curious what will happen with one of those.

Best Regards,
Georgi
Back to top
View user's profile Send private message
mi_unixbird
Tux's lil' helper
Tux's lil' helper


Joined: 24 Jul 2015
Posts: 130

PostPosted: Sun Sep 29, 2024 5:55 pm    Post subject: Reply with quote

logrusx wrote:
I still can't believe chromium build makes good use of ccache. There's a new 129 version available. Also 130. I'm curious what will happen with one of those.

Best Regards,
Georgi
I just had it updated. This one was disastrous:

Code:
     Sun Sep 29 19:32:00 2024 >>> www-client/chromium-129.0.6668.70
       merge time: 9 hours, 42 minutes and 35 seconds.


Code:
Cacheable calls:   39428 / 39474 (99.88%)
  Hits:             1962 / 39428 ( 4.98%)
    Direct:         1962 /  1962 (100.0%)
    Preprocessed:      0 /  1962 ( 0.00%)
  Misses:          37466 / 39428 (95.02%)
Uncacheable calls:    46 / 39474 ( 0.12%)


I actually zeroed the statistics before compiling this time and cache hit ratio was abysmal. I don't think a compiler update happened but clearly something happened that made the ratio so low. Maybe it has something to do with that I switched to depend mode in the meanwhile

It's still slightly faster than how it was when all calls where uncachable which was around 10~10:30 hours but it's not a pretty picture here. We'll see what the next update brings. I have been muckking around a bit and tuning some things and switched to a per-package cache originally just forked of the main cache by using a reflink copy of it so maybe that did something. Come to think of I didn't use “cp -a” to do this. Perhaps ccache really doesn't like it if file creation dates are updated. I definitely should've done that.
_________________
execctl --path exec filectl --current-directory list
Back to top
View user's profile Send private message
mi_unixbird
Tux's lil' helper
Tux's lil' helper


Joined: 24 Jul 2015
Posts: 130

PostPosted: Fri Oct 04, 2024 10:58 am    Post subject: Reply with quote

After getting a similarly abysmal result I did some research and it turns out that depend mode is entirely useless for upgrades because it by necessity needs to use the file pah in the hash then, not simply the content of the file, and since portage includes the version in the file path in the build dir it's of course entirely useless. I read somewhere that depend mode was generally faster but it seemed that only applied to rebuilding the same identical thing twice.
_________________
execctl --path exec filectl --current-directory list
Back to top
View user's profile Send private message
logrusx
Advocate
Advocate


Joined: 22 Feb 2018
Posts: 2418

PostPosted: Fri Oct 04, 2024 4:56 pm    Post subject: Reply with quote

mi_unixbird wrote:
I did some research and it turns out that depend mode is entirely useless for upgrades because it by necessity needs to use the file pah in the hash then, not simply the content of the file, and since portage includes the version in the file path in the build dir it's of course entirely useless.


Good job. However the common hash information includes the path for all modes, so they are all useless.

Common hashed information wrote:

The following information is always included in the hash:

the extension used by the compiler for a file with preprocessor output (normally .i for C code and .ii for C++ code)

the compiler’s size and modification time (or other compiler-specific information specified by compiler_check)

the name of the compiler

the current directory (if hash_dir is enabled)


So you may try with hash_dir disabled. If that works you will have given hundreds, if not thousands of Gentoo users new hope for compiling Chromium and its derivatives with ccache.

Or if you find a way to cut that part of the path that contains the version information, but that I think is not trivial and may need support in the ebuild itself.

Best Regards,
Georgi
Back to top
View user's profile Send private message
mi_unixbird
Tux's lil' helper
Tux's lil' helper


Joined: 24 Jul 2015
Posts: 130

PostPosted: Fri Oct 04, 2024 9:35 pm    Post subject: Reply with quote

logrusx wrote:
mi_unixbird wrote:
I did some research and it turns out that depend mode is entirely useless for upgrades because it by necessity needs to use the file pah in the hash then, not simply the content of the file, and since portage includes the version in the file path in the build dir it's of course entirely useless.


Good job. However the common hash information includes the path for all modes, so they are all useless.

Common hashed information wrote:

The following information is always included in the hash:

the extension used by the compiler for a file with preprocessor output (normally .i for C code and .ii for C++ code)

the compiler’s size and modification time (or other compiler-specific information specified by compiler_check)

the name of the compiler

the current directory (if hash_dir is enabled)


So you may try with hash_dir disabled. If that works you will have given hundreds, if not thousands of Gentoo users new hope for compiling Chromium and its derivatives with ccache.

Or if you find a way to cut that part of the path that contains the version information, but that I think is not trivial and may need support in the ebuild itself.

Best Regards,
Georgi
I already had hash_dir disabled of course.

It occurs to me that the people who had bad results, though they said it was despite getting high hit ratios might actually have it enabled. It's common knowledge to disable it I thought and I honestly don't understand why it's enabled by default. I would obviously not be getting those 3 hours and even 30 minute compile times if I had it enabled.

In fact, reading on about how it works. It might actually be worthwhile to completely disable direct mode so it doesn't even try for updates since it will almost always fail anyway, and have it always run in preprocessor mode. I'm not sure what the reason for including the current working directory is but using the current file names is because a file may contain a “__FILE__” macro and would thus get a different result with a different filename, perhaps this is also the reason the current working directory is included for relative filenames, but I don't see how that would still be relevant for files after preprocessing.

Quote:
Or if you find a way to cut that part of the path that contains the version information, but that I think is not trivial and may need support in the ebuild itself.

Honestly, this made me realize that the preprocessing for some files will still be different as they will contain debug information that contains the full path they are located in.I wonder if one can set up something with namespaces to begin with to make everything that compiles believe it's workdir is actually just under the same default directory while it isn't. This seems like a useful feature for more than one reason.

But even then many things will still contain versioning numbers I suppose that will be different in debug output. Also, I have distcc set up to rewrite all paths to relative but even the relative paths often still contain versioning numbers which I suppose is needed to provide accurate debugging.
_________________
execctl --path exec filectl --current-directory list
Back to top
View user's profile Send private message
mi_unixbird
Tux's lil' helper
Tux's lil' helper


Joined: 24 Jul 2015
Posts: 130

PostPosted: Sun Oct 06, 2024 10:34 pm    Post subject: Reply with quote

After some further digging and hacking I found that ccache is obviously heavily influenced by line marker metadata emitted by the preprocessor depending on the build system which may include a version number in the path. I now used a compiler wrapper which passes “-P” to the preprocessor to strip this. This led to mesa, a package I didn't use ccache on because hits were abysmal as in the 5% range to actually get over 90% hit ratio now on an upgrade cutting built time down from 9 to 1 minute.

As in: “prefix_command_cpp=/usr/local/libexec/ccache_pp_wrapper” with this simply executing “exec "$@" -P” and nothing more. It seems like this change makes a lot of packages which didn't work well with upgrades compile get a high hit ratio.
_________________
execctl --path exec filectl --current-directory list
Back to top
View user's profile Send private message
logrusx
Advocate
Advocate


Joined: 22 Feb 2018
Posts: 2418

PostPosted: Sat Nov 02, 2024 7:00 pm    Post subject: Reply with quote

Any news about that?

Best Regards,
Georgi
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum