| View previous topic :: View next topic |
| Author |
Message |
jpsollie
Guru
Joined: 17 Aug 2013
Posts: 319
|
 Posted: Sun Sep 29, 2024 5:04 am Post subject: how do you guys handle qtwebengine |
 |
|
I'm fed up ...
every time I perform a emerge --update --deep --newuse --with-bdeps=y @system @world, it fails at the package qtwebengine.
mostly (yes, murphy's law) at the end of this heavy package,
I can then start digging through, experimenting with whatever comes to mind.
every time, i mean since QT 5.10, it was qtwebengine which broke the upgrade process.
I can't imagine being the only one who's afraid of compiling qtwebengine,
so how do you people deal with it when using kde plasma?
_________________
The power of Gentoo optimization (not overclocked): [img]https://www.passmark.com/baselines/V10/images/503714802842.png[/img] |
|
| Back to top |
|
 |
asturm
Developer
Joined: 05 Apr 2007
Posts: 9202
|
| Posted: Sun Sep 29, 2024 6:25 am Post subject: |
|
|
KDE Plasma itself can do without qtwebengine.
If your system lacks resources as determined by your build params, adjust build parameters in make.conf or per-package via package.env. Or increase resources hardware-wise.
Last edited by asturm on Sun Sep 29, 2024 7:01 am; edited 1 time in total |
|
| Back to top |
|
|
dmpogo
Advocate
Joined: 02 Sep 2004
Posts: 3396
Location: Canada
|
| Posted: Sun Sep 29, 2024 6:51 am Post subject: |
|
|
| got rid of everything that depends on it and run USE=-webengine |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2250
|
| Posted: Sun Sep 29, 2024 7:12 am Post subject: |
|
|
Binhost
Binhost x86_64-v3 if your CPU supports it.
| /etc/portage/binrepos.conf/gentoobinhost.conf: |
[binhost-x86_64]
priority = 9998
#sync-uri = https://ftp.snt.utwente.nl/pub/os/linux/gentoo/releases/amd64/binpackages/23.0/x86-64/
sync-uri = https://ftp.snt.utwente.nl/pub/os/linux/gentoo/releases/amd64/binpackages/23.0/x86-64/
[binhost-x86_64-v3]
priority = 9999
#sync-uri = https://ftp.snt.utwente.nl/pub/os/linux/gentoo/releases/amd64/binpackages/23.0/x86-64-v3/
sync-uri = https://ftp.snt.utwente.nl/pub/os/linux/gentoo/releases/amd64/binpackages/23.0/x86-64-v3/
|
Here the x86_64-v3 will take precedence and this way if a package is not available in v3 it'll be taken from the base x86_64 binrepo. If it's not available there too, then portage builds it from source.
Then adjust use flags if necessary. I think qtwebengine particularly does not need anything other than default, but nodejs for example needs -inspector. And I hate that crap but firefox:rapid requires it to build. However if you build chromium, it requires nodejs[inspector] which does not satisfy the binary package. However I used to use ungoogled-chromium-bin::pf4public which saved the day.
If you want to use it only for qtwebengine you update i separately with --getbinpkg. There still isn't a way to selectively chose binary packages other than doing it manually, sometimes terminating the source ebuild and then running the --getbinpkg query for that package manually and then starting the update over.
Another option is:
This will fetch the binary package from the binhost (-g/--getbinpkg) and then use it locally (-K/--usepkgonly). Still you need to update it separately.
EDIT: I just came up with an idea based on the above: emerge -gFpv <package(s)>. This will only fetch the binary package(s). Then continue the update with --usepkg. It'll use whatever binary packages that are compatible from the cache.
p.s. a related thread where it turned out to be a low memory condition: https://forums.gentoo.org/viewtopic-t-1170988.html Chromium and qtwebengine are notorious for that. Even more after gcc support for building chromium was dropped. Clang seems to have higher memory usage peeks. I've observed that since long long ago.
Best Regards,
Georgi
Last edited by logrusx on Sun Sep 29, 2024 9:44 am; edited 3 times in total |
|
| Back to top |
|
|
Banana
Moderator
Joined: 21 May 2004
Posts: 1662
Location: Germany
|
|
| Back to top |
|
|
lars_the_bear
Guru
Joined: 05 Jun 2024
Posts: 474
|
| Posted: Sun Sep 29, 2024 8:43 am Post subject: |
|
|
I've adjusted the package-specific USE flags so that I can use the binary version. I'm not going to try to compile that monster again on the hardware I have. I'm not sure what the implications are of doing this; but I'm installing qtwebengine at all, because FreeCAD has a dependency on it. I have no idea how (or even if) FreeCAD uses qtwebengine, and I'm not particular bothered whether it works properly.
BR, Lars. |
|
| Back to top |
|
|
Josef.95
Advocate
Joined: 03 Sep 2007
Posts: 4648
Location: Germany
|
| Posted: Sun Sep 29, 2024 9:27 am Post subject: |
|
|
But, what is the real problem?
Can you post the output from "emerge --info" and the error message (the build.log)?
/edit:
As first step you can check | Code: | | grep -i Killed /path/to/the/build.log |
and check the dmesg output. |
|
| Back to top |
|
|
jpsollie
Guru
Joined: 17 Aug 2013
Posts: 319
|
| Posted: Sun Sep 29, 2024 3:18 pm Post subject: |
|
|
that's exactly my point:
it's always something else: the build system, libraries, the compiler, binutils, ...
every time it crashes, but at every update it finds a new update to crash on.
For 6.7.2 and 6.7.3, the error log simply states:
| Code: |
[78/106] : && /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/linker_ulimit.sh /usr/lib/distcc/bin/x86_64-pc-linux-gnu-g++ -fPIC -march=znver1 -mtune=znver2 -O3 -pipe -Wl,-O1 -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-Bsymbolic-functions -Wl,--no-undefined -Wl,--version-script,/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/api/WebEngineCore.version @/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/QtWebEngineCore_objects.rsp -Wl,--gc-sections -Wl,--enable-new-dtags -shared -Wl,-soname,libQt6WebEngineCore.so.6 -o lib64/libQt6WebEngineCore.so.6.7.3 src/core/api/CMakeFiles/WebEngineCore.dir/Unity/unity_1_cxx.cxx.o src/core/api/CMakeFiles/WebEngineCore.dir/Unity/unity_0_cxx.cxx.o -Wl,-rpath,::::::: -Wl,--start-group @/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/QtWebEngineCore_archives.rsp -Wl,--end-group -Wl,--no-fatal-warnings @/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/QtWebEngineCore_ldir.rsp @/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/QtWebEngineCore_libs.rsp -Wl,--no-fatal-warnings /usr/lib64/libxkbcommon.so /usr/lib64/libQt6Quick.so.6.7.3 /usr/lib64/libQt6OpenGL.so.6.7.3 /usr/lib64/libQt6Gui.so.6.7.3 /usr/lib64/libGLX.so /usr/lib64/libOpenGL.so /usr/lib64/libQt6QmlModels.so.6.7.3 /usr/lib64/libQt6WebChannel.so.6.7.3 /usr/lib64/libQt6Qml.so.6.7.3 /usr/lib64/libQt6Network.so.6.7.3 /usr/lib64/libQt6QmlBuiltins.a /usr/lib64/libQt6Core.so.6.7.3 && : FAILED: lib64/libQt6WebEngineCore.so.6.7.3 : && /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/linker_ulimit.sh /usr/lib/distcc/bin/x86_64-pc-linux-gnu-g++ -fPIC -march=znver1 -mtune=znver2 -O3 -pipe -Wl,-O1 -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-Bsymbolic-functions -Wl,--no-undefined -Wl,--version-script,/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/api/WebEngineCore.version @/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/QtWebEngineCore_objects.rsp -Wl,--gc-sections -Wl,--enable-new-dtags -shared -Wl,-soname,libQt6WebEngineCore.so.6 -o lib64/libQt6WebEngineCore.so.6.7.3 src/core/api/CMakeFiles/WebEngineCore.dir/Unity/unity_1_cxx.cxx.o src/core/api/CMakeFiles/WebEngineCore.dir/Unity/unity_0_cxx.cxx.o -Wl,-rpath,::::::: -Wl,--start-group @/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/QtWebEngineCore_archives.rsp -Wl,--end-group -Wl,--no-fatal-warnings @/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/QtWebEngineCore_ldir.rsp @/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/QtWebEngineCore_libs.rsp -Wl,--no-fatal-warnings /usr/lib64/libxkbcommon.so /usr/lib64/libQt6Quick.so.6.7.3 /usr/lib64/libQt6OpenGL.so.6.7.3 /usr/lib64/libQt6Gui.so.6.7.3 /usr/lib64/libGLX.so /usr/lib64/libOpenGL.so /usr/lib64/libQt6QmlModels.so.6.7.3 /usr/lib64/libQt6WebChannel.so.6.7.3 /usr/lib64/libQt6Qml.so.6.7.3 /usr/lib64/libQt6Network.so.6.7.3 /usr/lib64/libQt6QmlBuiltins.a /usr/lib64/libQt6Core.so.6.7.3 && :
ninja: build stopped: subcommand failed.
* ERROR: dev-qt/qtwebengine-6.7.3::gentoo failed (compile phase):
* ninja -v -j70 -l12 failed
*
* Call stack:
* ebuild.sh, line 136: Called src_compile
* environment, line 3579: Called cmake_src_compile * environment, line 1650: Called cmake_build
* environment, line 1617: Called eninja
* environment, line 2099: Called die
* The specific snippet of code:
* "$@" || die -n "${*} failed" *
* If you need support, post the output of `emerge --info '=dev-qt/qtwebengine-6.7.3::gentoo'`,
* the complete build log and the output of `emerge -pqv '=dev-qt/qtwebengine-6.7.3::gentoo'`.
* The complete build log is located at '/var/tmp/portage/dev-qt/qtwebengine-6.7.3/temp/build.log'.
* The ebuild environment file is located at '/var/tmp/portage/dev-qt/qtwebengine-6.7.3/temp/environment'. * Working directory: '/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build'
* S: '/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3'
|
so yes, that doesn't say anything
_________________
The power of Gentoo optimization (not overclocked): [img]https://www.passmark.com/baselines/V10/images/503714802842.png[/img] |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2250
|
| Posted: Sun Sep 29, 2024 3:28 pm Post subject: |
|
|
This is not relevant output. There must be something above. I see it's only 78 of 108 so I guess you could share the whole build.log. Or at least 100 lines above what you've shared.
Best Regards,
Georgi |
|
| Back to top |
|
|
jpsollie
Guru
Joined: 17 Aug 2013
Posts: 319
|
| Posted: Sun Sep 29, 2024 4:15 pm Post subject: |
|
|
| logrusx wrote: | This is not relevant output. There must be something above. I see it's only 78 of 108 so I guess you could share the whole build.log. Or at least 100 lines above what you've shared.
Best Regards,
Georgi |
You're too optimistic:
| Code: |
ebuild /var/db/repos/gentoo/dev-qt/qtwebengine/qtwebengine-6.7.3.ebuild compile
>>> Existing ${T}/environment for 'qtwebengine-6.7.3' will be sourced. Run
>>> 'clean' to start with a fresh environment.
>>> Checking qtwebengine-6.7-patchset-13.tar.xz's mtime...
>>> Checking qtwebengine-everywhere-src-6.7.3.tar.xz's mtime...
>>> WORKDIR is up-to-date, keeping...
* checking ebuild checksums ;-) ... [ ok ]
* checking auxfile checksums ;-) ... [ ok ]
* checking miscfile checksums ;-) ... [ ok ]
* Checking for at least 8 GiB disk space at "/var/tmp/portage/dev-qt/qtwebengine-6.7.3/temp" ... [ ok ]
* Checking for at least 360 MiB disk space at "/usr" ... [ ok ]
>>> It appears that 'setup' has already executed for 'qtwebengine-6.7.3'; skipping.
>>> Remove '/var/tmp/portage/dev-qt/qtwebengine-6.7.3/.setuped' to force setup.
>>> It appears that 'unpack' has already executed for 'qtwebengine-6.7.3'; skipping.
>>> Remove '/var/tmp/portage/dev-qt/qtwebengine-6.7.3/.unpacked' to force unpack.
>>> It appears that 'prepare' has already executed for 'qtwebengine-6.7.3'; skipping.
>>> Remove '/var/tmp/portage/dev-qt/qtwebengine-6.7.3/.prepared' to force prepare.
>>> It appears that 'configure' has already executed for 'qtwebengine-6.7.3'; skipping.
>>> Remove '/var/tmp/portage/dev-qt/qtwebengine-6.7.3/.configured' to force configure.
>>> Compiling source in /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3 ...
* Source directory (CMAKE_USE_DIR): "/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3"
* Build directory (BUILD_DIR): "/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build"
ninja -v -j70 -l12 WebEngineCore_sync_all_public_headers
[1/1] cd /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/api && /usr/lib64/qt6/libexec/syncqt @/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/api/WebEngineCore_syncqt_all_args
* Source directory (CMAKE_USE_DIR): "/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3"
* Build directory (BUILD_DIR): "/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build"
ninja -v -j70 -l12
[0/31] cd /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core && /usr/bin/ninja -j70 -l12 -v -C /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64 QtWebEngineCore ninja: Entering directory `/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64' ninja: no work to do.
[1/31] cd /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core && /usr/bin/ninja -j70 -l12 -v -C /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64 convert_dict
ninja: Entering directory `/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64'
ninja: no work to do.
[3/31] : && /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/linker_ulimit.sh /usr/lib/distcc/bin/x86_64-pc-linux-gnu-g++ -fPIC -march=znver1 -mtune=znver2 -O3 -pipe -Wl,-O1 -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-Bsymbolic-functions -Wl,--no-undefined -Wl,--version-script,/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/api/WebEngineCore.version @/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/QtWebEngineCore_objects.rsp -Wl,--gc-sections -Wl,--enable-new-dtags -shared -Wl,-soname,libQt6WebEngineCore.so.6 -o lib64/libQt6WebEngineCore.so.6.7.3 src/core/api/CMakeFiles/WebEngineCore.dir/Unity/unity_1_cxx.cxx.o src/core/api/CMakeFiles/WebEngineCore.dir/Unity/unity_0_cxx.cxx.o -Wl,-rpath,::::::: -Wl,--start-group @/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/QtWebEngineCore_archives.rsp -Wl,--end-group -Wl,--no-fatal-warnings @/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/QtWebEngineCore_ldir.rsp @/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/QtWebEngineCore_libs.rsp -Wl,--no-fatal-warnings /usr/lib64/libxkbcommon.so /usr/lib64/libQt6Quick.so.6.7.3 /usr/lib64/libQt6OpenGL.so.6.7.3 /usr/lib64/libQt6Gui.so.6.7.3 /usr/lib64/libGLX.so /usr/lib64/libOpenGL.so /usr/lib64/libQt6QmlModels.so.6.7.3 /usr/lib64/libQt6WebChannel.so.6.7.3 /usr/lib64/libQt6Qml.so.6.7.3 /usr/lib64/libQt6Network.so.6.7.3 /usr/lib64/libQt6QmlBuiltins.a /usr/lib64/libQt6Core.so.6.7.3 && :
FAILED: lib64/libQt6WebEngineCore.so.6.7.3
: && /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/linker_ulimit.sh /usr/lib/distcc/bin/x86_64-pc-linux-gnu-g++ -fPIC -march=znver1 -mtune=znver2 -O3 -pipe -Wl,-O1 -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-Bsymbolic-functions -Wl,--no-undefined -Wl,--version-script,/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/api/WebEngineCore.version @/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/QtWebEngineCore_objects.rsp -Wl,--gc-sections -Wl,--enable-new-dtags -shared -Wl,-soname,libQt6WebEngineCore.so.6 -o lib64/libQt6WebEngineCore.so.6.7.3 src/core/api/CMakeFiles/WebEngineCore.dir/Unity/unity_1_cxx.cxx.o src/core/api/CMakeFiles/WebEngineCore.dir/Unity/unity_0_cxx.cxx.o -Wl,-rpath,::::::: -Wl,--start-group @/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/QtWebEngineCore_archives.rsp -Wl,--end-group -Wl,--no-fatal-warnings @/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/QtWebEngineCore_ldir.rsp @/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/QtWebEngineCore_libs.rsp -Wl,--no-fatal-warnings /usr/lib64/libxkbcommon.so /usr/lib64/libQt6Quick.so.6.7.3 /usr/lib64/libQt6OpenGL.so.6.7.3 /usr/lib64/libQt6Gui.so.6.7.3 /usr/lib64/libGLX.so /usr/lib64/libOpenGL.so /usr/lib64/libQt6QmlModels.so.6.7.3 /usr/lib64/libQt6WebChannel.so.6.7.3 /usr/lib64/libQt6Qml.so.6.7.3 /usr/lib64/libQt6Network.so.6.7.3 /usr/lib64/libQt6QmlBuiltins.a /usr/lib64/libQt6Core.so.6.7.3 && :
ninja: build stopped: subcommand failed.
* ERROR: dev-qt/qtwebengine-6.7.3::gentoo failed (compile phase):
* ninja -v -j70 -l12 failed
*
* Call stack:
* ebuild.sh, line 136: Called src_compile
* environment, line 3579: Called cmake_src_compile
* environment, line 1650: Called cmake_build
* environment, line 1617: Called eninja
* environment, line 2099: Called die
* The specific snippet of code:
* "$@" || die -n "${*} failed"
*
* If you need support, post the output of `emerge --info '=dev-qt/qtwebengine-6.7.3::gentoo'`,
* the complete build log and the output of `emerge -pqv '=dev-qt/qtwebengine-6.7.3::gentoo'`.
* The complete build log is located at '/var/tmp/portage/dev-qt/qtwebengine-6.7.3/temp/build.log'.
* The ebuild environment file is located at '/var/tmp/portage/dev-qt/qtwebengine-6.7.3/temp/environment'.
* Working directory: '/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build'
* S: '/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3'
|
_________________
The power of Gentoo optimization (not overclocked): [img]https://www.passmark.com/baselines/V10/images/503714802842.png[/img] |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3640
Location: Rasi, Finland
|
| Posted: Sun Sep 29, 2024 4:24 pm Post subject: |
|
|
Hmm... | Quote: | | *snip*: | | ninja -v -j70 -l12 failed |
|
Ninja is running 70 concurrent compiler jobs at maximum, but limited if load average goes above 12.00.
There's quite a possibility for ninja to first spawn 70 jobs then eventually settling around 12 if nothing else is straining your system. I always assume each job takes one gig of RAM while compiling big packages. 70 concurrent jobs would surely kill any machines I own.
Maybe check your MAKEOPTS?
EDIT: Just saw you last post. MAKEOPTS may not be the problem here, but does not hurt to check.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2250
|
| Posted: Sun Sep 29, 2024 4:31 pm Post subject: |
|
|
| Code: | | dmesg | grep killed |
Best Regards,
Georgi |
|
| Back to top |
|
|
jpsollie
Guru
Joined: 17 Aug 2013
Posts: 319
|
| Posted: Sun Sep 29, 2024 4:56 pm Post subject: |
|
|
| logrusx wrote: | | Code: | | dmesg | grep killed |
Best Regards,
Georgi |
| Code: |
janpieter@desktopjp ~ $ dmesg | grep killed
janpieter@desktopjp ~ $
|
The desktop machine has 64GB of ram for 12 CPUs (and 256GB of swap), but links to a 64 core distcc server, so that's where the -j70 -l12 comes from
*edit* some additional info:
| Code: |
COMMON_FLAGS="-march=znver1 -mtune=znver2 -O3 -pipe"
#MACHINE_FLAGS="-march=znver1 -mabm -madx -maes -mavx -mavx2 -mbmi -mbmi2 -mclflushopt -mclzero -mcx16 -mf16c -mfma -mfsgsbase -mfxsr -mlzcnt -mmmx -mmovbe -mmwaitx -mpclmul -mpopcnt -mprfchw -mrdrnd -mrdseed -msahf -msha -msse -msse2 -msse3 -msse4.1 -msse4.2 -msse4a -mssse3 -mxsave -mxsavec -mxsaveopt -mxsaves --param=l1-cache-line-size=64 --param=l1-cache-size=32 --param=l2-cache-size=512"
CFLAGS="${COMMON_FLAGS} ${MACHINE_FLAGS}"
CXXFLAGS="${COMMON_FLAGS} ${MACHINE_FLAGS}"
FCFLAGS="${COMMON_FLAGS} ${MACHINE_FLAGS}"
FFLAGS="${COMMON_FLAGS} ${MACHINE_FLAGS}"
# NOTE: This stage was built with the bindist Use flag enabled
PORTDIR="/var/db/repos/gentoo"
DISTDIR="/var/cache/distfiles"
PKGDIR="/var/cache/binpkgs"
BINPKG_COMPRESS="lz4"
BINPKG_FORMAT="gpkg"
FEATURES="nostrip -network-sandbox -ebuild-locks -sandbox -ipc-sandbox -pid-sandbox buildpkg distcc" # -userfetch -userpriv"
EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --usepkg"
#DISTCC_IO_TIMEOUT=30
DISTCC_LOG="/tmp/distcc.log"
#DISTCC_VERBOSE=1
MAKEOPTS="-j70 -l12"
#MAKEOPTS="-j12"
|
... and the tail of distcc log:
| Code: |
distcc[3967] 11823232 bytes from /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/api/CMakeFiles/WebEngineCore.dir/Unity/unity_0_cxx.cxx compiled on linuxserver in 11.2919s, rate 1023kB/s distcc[3967] compile /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/api/CMakeFiles/WebEngineCore.dir/Unity/unity_0_cxx.cxx on linuxserver/70 completed ok
distcc[3967] elapsed compilation time 11.292266s
distcc[3967] (dcc_exit) exit: code 0; self: 0.004300 user 0.004300 sys; children: 0.607579 user 0.099740 sys
distcc[3973] (dcc_scan_args) called for link? i give up
distcc[3973] exec on localhost: x86_64-pc-linux-gnu-g++ -fPIC -march=znver1 -mtune=znver2 -O3 -pipe -Wl,-O1 -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-Bsymbolic-functions -Wl,--no-undefined -Wl,--version-script,/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/api/WebEngineCore.version /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/accessibility_tree_formatter_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/browser_accessibility_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/authentication_dialog_controller.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/autofill_client_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/autofill_popup_controller.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/browser_accessibility_manager_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/browser_main_parts_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/browser_message_filter_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/browsing_data_remover_delegate_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/certificate_error_controller.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/chromium_overrides.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/client_cert_select_controller.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/client_hints.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/clipboard_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/color_chooser_controller.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/color_chooser_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/qt_messages.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/compositor.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/content_gpu_client_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/display_overrides.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/display_software_output_surface.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/native_skia_output_device.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/content_browser_client_qt.o /var/tmp/p
distcc[3974] (dcc_execvp) ERROR: failed to exec x86_64-pc-linux-gnu-g++: Argument list too long
distcc[3974] (dcc_exit) exit: code 110; self: 0.000000 user 0.003660 sys; children: 0.000000 user 0.000000 sys
distcc[3973] (dcc_collect_child) cc times: user 0.000000s, system 0.000000s, 0 minflt, 0 majflt
distcc[3973] ERROR: compile (null) on localhost failed with exit code 110
distcc[3973] elapsed compilation time 0.011420s
distcc[3973] (dcc_exit) exit: code 110; self: 0.014649 user 0.000000 sys; children: 0.000000 user 0.003913 sys
distcc[4910] (dcc_scan_args) called for link? i give up
distcc[4910] exec on localhost: x86_64-pc-linux-gnu-g++ -fPIC -march=znver1 -mtune=znver2 -O3 -pipe -Wl,-O1 -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-Bsymbolic-functions -Wl,--no-undefined -Wl,--version-script,/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/api/WebEngineCore.version /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/accessibility_tree_formatter_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/browser_accessibility_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/authentication_dialog_controller.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/autofill_client_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/autofill_popup_controller.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/browser_accessibility_manager_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/browser_main_parts_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/browser_message_filter_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/browsing_data_remover_delegate_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/certificate_error_controller.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/chromium_overrides.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/client_cert_select_controller.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/client_hints.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/clipboard_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/color_chooser_controller.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/color_chooser_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/qt_messages.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/compositor.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/content_gpu_client_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/display_overrides.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/display_software_output_surface.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/native_skia_output_device.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/content_browser_client_qt.o /var/tmp/p distcc[4911] (dcc_execvp) ERROR: failed to exec x86_64-pc-linux-gnu-g++: Argument list too long
distcc[4911] (dcc_exit) exit: code 110; self: 0.000000 user 0.003651 sys; children: 0.000000 user 0.000000 sys
distcc[4910] (dcc_collect_child) cc times: user 0.000000s, system 0.000000s, 0 minflt, 0 majflt
distcc[4910] ERROR: compile (null) on localhost failed with exit code 110
distcc[4910] elapsed compilation time 0.011288s
distcc[4910] (dcc_exit) exit: code 110; self: 0.008630 user 0.005753 sys; children: 0.000000 user 0.003859 sys
|
_________________
The power of Gentoo optimization (not overclocked): [img]https://www.passmark.com/baselines/V10/images/503714802842.png[/img]
Last edited by jpsollie on Sun Sep 29, 2024 5:14 pm; edited 2 times in total |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2250
|
| Posted: Sun Sep 29, 2024 5:08 pm Post subject: |
|
|
| jpsollie wrote: | | distcc server |
I'm not very competent to analyze this setup. I hope others will come up with ideas of what might be going on. But what you describe is not normal. Even I didn't have such problems when I run a quad core CPU from 2007 with 6GB RAM.
But I strongly suggest using the binhost for that particular package. I have a 16T CPU with 32GB of RAM and I don't waste time compiling it although it would take 2 to 3 hours max. Compiling those(chromium derived pieces or crap) on whatever hardware is a waste of resource.
Best Regards,
Georgi |
|
| Back to top |
|
|
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1140
Location: Richmond Hill, Canada
|
| Posted: Sun Sep 29, 2024 5:15 pm Post subject: |
|
|
| jpsollie wrote: | | logrusx wrote: | This is not relevant output. There must be something above. I see it's only 78 of 108 so I guess you could share the whole build.log. Or at least 100 lines above what you've shared.
Best Regards,
Georgi |
You're too optimistic:
| Code: |
ebuild /var/db/repos/gentoo/dev-qt/qtwebengine/qtwebengine-6.7.3.ebuild compile
...
... |
|
You are picking up from an previous failed job.
So why you are using "ebuild" to run? Why not using emerge so we can see from beginning where the error? |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2250
|
| Posted: Sun Sep 29, 2024 5:24 pm Post subject: |
|
|
| pingtoo wrote: | | jpsollie wrote: | | logrusx wrote: | This is not relevant output. There must be something above. I see it's only 78 of 108 so I guess you could share the whole build.log. Or at least 100 lines above what you've shared.
Best Regards,
Georgi |
You're too optimistic:
| Code: |
ebuild /var/db/repos/gentoo/dev-qt/qtwebengine/qtwebengine-6.7.3.ebuild compile
...
... |
|
You are picking up from an previous failed job. |
Exactly what I meant.
| pingtoo wrote: | | So why you are using "ebuild" to run? Why not using emerge so we can see from beginning where the error? |
I hadn't noticed that. Maybe to save time, but this is exactly how not to share error output. That's why the real cause is not visible. Whatever has failed has already failed and now jobs counting on that to do their job fail too.
There may still be low memory condition but this should be checked on the distcc server.
p.s. I'm not sure if the command I posted above greps for the right thing. I haven't had it on this computer and the old one is not functional.
Best Regards,
Georgi |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3640
Location: Rasi, Finland
|
| Posted: Sun Sep 29, 2024 9:29 pm Post subject: |
|
|
| Quote: | | Code: | | distcc[3974] (dcc_execvp) ERROR: failed to exec x86_64-pc-linux-gnu-g++: Argument list too long |
|
I think this a is bug. Bug in the upstream build process or in distcc.
It's quite rare to reach this limit. I mean, I think it's something like 2 megabytes or more arguments until this error is triggered.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
eschwartz
Developer
Joined: 29 Oct 2023
Posts: 181
|
| Posted: Mon Sep 30, 2024 3:31 am Post subject: |
|
|
| Zucca wrote: | Hmm... | Quote: | | *snip*: | | ninja -v -j70 -l12 failed |
|
Ninja is running 70 concurrent compiler jobs at maximum, but limited if load average goes above 12.00.
There's quite a possibility for ninja to first spawn 70 jobs then eventually settling around 12 if nothing else is straining your system. I always assume each job takes one gig of RAM while compiling big packages. 70 concurrent jobs would surely kill any machines I own.
Maybe check your MAKEOPTS?
EDIT: Just saw you last post. MAKEOPTS may not be the problem here, but does not hurt to check. |
In the resumed failure, it failed on job 3 of 31 and attempted no further jobs (it would finish any other pending parallel jobs if they existed and we would see that status).
Given only 3 jobs were relevant, we can conclusively determine the issue wasn't clobbering the system with 70 parallel jobs.
| pingtoo wrote: | | jpsollie wrote: | | logrusx wrote: | This is not relevant output. There must be something above. I see it's only 78 of 108 so I guess you could share the whole build.log. Or at least 100 lines above what you've shared.
Best Regards,
Georgi |
You're too optimistic:
| Code: |
ebuild /var/db/repos/gentoo/dev-qt/qtwebengine/qtwebengine-6.7.3.ebuild compile
...
... |
|
You are picking up from an previous failed job.
So why you are using "ebuild" to run? Why not using emerge so we can see from beginning where the error? |
Because the correct thing to do is use "ebuild" to run -- if you don't use "ebuild" to run, you unfortunately start from the beginning, waste a ton of time, and then produce the same log file without useful information.
Conversely, using "ebuild" allows picking up from a previous failed job, which means you only see the parts that failed, which means you only see the errors you want to see.
| logrusx wrote: |
| pingtoo wrote: | | So why you are using "ebuild" to run? Why not using emerge so we can see from beginning where the error? |
I hadn't noticed that. Maybe to save time, but this is exactly how not to share error output. That's why the real cause is not visible. Whatever has failed has already failed and now jobs counting on that to do their job fail too.
There may still be low memory condition but this should be checked on the distcc server.
|
And still nope. This is exactly how to share error output the advanced way. Whatever has failed has FAILED and therefore jobs counting on that to do their job ***did not run to begin with***. They did not fail, they simply didn't run. They are in "build graph dependencies not satisfied mode".
Resuming a build will then restart with the last thing that failed, try it again, and show you the CAUSE of the error in the thing that originally failed but without noisy irrelevant surrounding context.
Any build system that fails to show correct error output on resuming a failed build, is a build system that does not work to begin with -- because if other jobs are counting on the first job to succeed, but end up running after the first job FAILED, then the other jobs are broken and don't actually depend on the first job at all.
If they don't depend on the first job, then it's actually up to the RNG to decide whether or not they try to run FIRST, without even attempting to wait until the actual real first job has run. They will then fail because, erm, they were assuming something would run without actually checking to see if that thing ran.  |
|
| Back to top |
|
|
eschwartz
Developer
Joined: 29 Oct 2023
Posts: 181
|
| Posted: Mon Sep 30, 2024 3:38 am Post subject: |
|
|
| jpsollie wrote: |
... and the tail of distcc log:
| Code: |
distcc[4910] exec on localhost: x86_64-pc-linux-gnu-g++ -fPIC -march=znver1 -mtune=znver2 -O3 -pipe -Wl,-O1 -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-Bsymbolic-functions -Wl,--no-undefined -Wl,--version-script,/var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/api/WebEngineCore.version /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/accessibility_tree_formatter_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/browser_accessibility_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/authentication_dialog_controller.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/autofill_client_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/autofill_popup_controller.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/browser_accessibility_manager_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/browser_main_parts_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/browser_message_filter_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/browsing_data_remover_delegate_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/certificate_error_controller.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/chromium_overrides.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/client_cert_select_controller.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/client_hints.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/clipboard_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/color_chooser_controller.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/color_chooser_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/qt_messages.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/compositor.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/content_gpu_client_qt.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/display_overrides.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/display_software_output_surface.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/native_skia_output_device.o /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/src/core/RelWithDebInfo/x86_64/obj/QtWebEngineCore/content_browser_client_qt.o /var/tmp/p distcc[4911] (dcc_execvp) ERROR: failed to exec x86_64-pc-linux-gnu-g++: Argument list too long
distcc[4911] (dcc_exit) exit: code 110; self: 0.000000 user 0.003651 sys; children: 0.000000 user 0.000000 sys
distcc[4910] (dcc_collect_child) cc times: user 0.000000s, system 0.000000s, 0 minflt, 0 majflt
distcc[4910] ERROR: compile (null) on localhost failed with exit code 110
distcc[4910] elapsed compilation time 0.011288s
distcc[4910] (dcc_exit) exit: code 110; self: 0.008630 user 0.005753 sys; children: 0.000000 user 0.003859 sys
|
|
Hi. The "argument list too long" thing indicates that seemingly distcc itself has failed to distribute jobs to other machines, for whatever reason, and as a result, ninja fails. Ninja is running
| Code: | | /var/tmp/portage/dev-qt/qtwebengine-6.7.3/work/qtwebengine-everywhere-src-6.7.3_build/linker_ulimit.sh /usr/lib/distcc/bin/x86_64-pc-linux-gnu-g++ ......... |
and this process produces no output or error message, but simply, silently, fails -- which is a distcc problem.
I will be honest -- I like the idea of distcc but in practice I feel like it does not work very well.  At least in this case it does seem to be causing problems. I would try temporarily disabling distcc and seeing if that helps.
P.S. I "handle" qtwebengine by installing it from the binhost. I'm a huge binhost shill, as it's the reason I agreed to try out gentoo in the first place and I'm currently one of the binhost project members. |
|
| Back to top |
|
|
jpsollie
Guru
Joined: 17 Aug 2013
Posts: 319
|
| Posted: Mon Sep 30, 2024 3:46 am Post subject: |
|
|
I also thought of distcc this time ...
It's hard: the PC itself is only a ryzen 5 device while the "compiling server" is a 64 core threadripper with no gui.
As a result, I can't compile qtwebengine on the compiiling server (which would obviously be a lot faster),
apparently can't compile distcc,
so then running on localhost would be the only way to fix this. Let's try ...
_________________
The power of Gentoo optimization (not overclocked): [img]https://www.passmark.com/baselines/V10/images/503714802842.png[/img] |
|
| Back to top |
|
|
jpsollie
Guru
Joined: 17 Aug 2013
Posts: 319
|
| Posted: Mon Sep 30, 2024 3:51 am Post subject: |
|
|
| eschwartz wrote: | | Zucca wrote: | Hmm... | Quote: | | *snip*: | | ninja -v -j70 -l12 failed |
|
Ninja is running 70 concurrent compiler jobs at maximum, but limited if load average goes above 12.00.
There's quite a possibility for ninja to first spawn 70 jobs then eventually settling around 12 if nothing else is straining your system. I always assume each job takes one gig of RAM while compiling big packages. 70 concurrent jobs would surely kill any machines I own.
Maybe check your MAKEOPTS?
EDIT: Just saw you last post. MAKEOPTS may not be the problem here, but does not hurt to check. |
In the resumed failure, it failed on job 3 of 31 and attempted no further jobs (it would finish any other pending parallel jobs if they existed and we would see that status).
Given only 3 jobs were relevant, we can conclusively determine the issue wasn't clobbering the system with 70 parallel jobs.
|
I saw this as well: even when running on localhost, the load never rises above 25, and the amount of swap used is only 0.5MB. waiting time is 0.0
If it would be a RAM or parallel issue, the load would be a lot higher and / or the swap used would be a lot higher (and I'd possible see some waiting time)
so it's definitely not running 70 compile jobs taking 2GB each at once
| eschwartz wrote: |
| pingtoo wrote: | | jpsollie wrote: | | logrusx wrote: | This is not relevant output. There must be something above. I see it's only 78 of 108 so I guess you could share the whole build.log. Or at least 100 lines above what you've shared.
Best Regards,
Georgi |
You're too optimistic:
| Code: |
ebuild /var/db/repos/gentoo/dev-qt/qtwebengine/qtwebengine-6.7.3.ebuild compile
...
... |
|
You are picking up from an previous failed job.
So why you are using "ebuild" to run? Why not using emerge so we can see from beginning where the error? |
Because the correct thing to do is use "ebuild" to run -- if you don't use "ebuild" to run, you unfortunately start from the beginning, waste a ton of time, and then produce the same log file without useful information.
Conversely, using "ebuild" allows picking up from a previous failed job, which means you only see the parts that failed, which means you only see the errors you want to see.
| logrusx wrote: |
| pingtoo wrote: | | So why you are using "ebuild" to run? Why not using emerge so we can see from beginning where the error? |
I hadn't noticed that. Maybe to save time, but this is exactly how not to share error output. That's why the real cause is not visible. Whatever has failed has already failed and now jobs counting on that to do their job fail too.
There may still be low memory condition but this should be checked on the distcc server.
|
And still nope. This is exactly how to share error output the advanced way. Whatever has failed has FAILED and therefore jobs counting on that to do their job ***did not run to begin with***. They did not fail, they simply didn't run. They are in "build graph dependencies not satisfied mode".
Resuming a build will then restart with the last thing that failed, try it again, and show you the CAUSE of the error in the thing that originally failed but without noisy irrelevant surrounding context.
Any build system that fails to show correct error output on resuming a failed build, is a build system that does not work to begin with -- because if other jobs are counting on the first job to succeed, but end up running after the first job FAILED, then the other jobs are broken and don't actually depend on the first job at all.
If they don't depend on the first job, then it's actually up to the RNG to decide whether or not they try to run FIRST, without even attempting to wait until the actual real first job has run. They will then fail because, erm, they were assuming something would run without actually checking to see if that thing ran. |
thank you for approving my thoughts. Couldn't have explained it better
_________________
The power of Gentoo optimization (not overclocked): [img]https://www.passmark.com/baselines/V10/images/503714802842.png[/img] |
|
| Back to top |
|
|
eschwartz
Developer
Joined: 29 Oct 2023
Posts: 181
|
| Posted: Mon Sep 30, 2024 4:16 am Post subject: |
|
|
| jpsollie wrote: |
It's hard: the PC itself is only a ryzen 5 device while the "compiling server" is a 64 core threadripper with no gui.
As a result, I can't compile qtwebengine on the compiiling server (which would obviously be a lot faster),
apparently can't compile distcc,
so then running on localhost would be the only way to fix this. Let's try ... |
Is it not possible to ssh into the "compiling server" and spawn an emerge job there? If necessary you can even unpack a fresh stage3 tarball, chroot into it, configure it with the same make.conf as you have on your PC, and then use emerge --buildpkg qtwebengine. This is actually more or less what the official binhost does too, except you get to choose exactly what CFLAGS and USE flags you want. Then simply sync (or serve over https) the resulting binaries in /var/cache/binpkgs and load them on the PC.
It's the same thing as distcc except that distcc can shard across multiple servers in tandem, whereas this still assumes you do all compiles on one machine, it's just a different machine.
If the problem is just that distcc is fragile then this will probably work fine. Just make sure you keep track of which binaries are built with which USE flags (and of course the same instructions for effective distcc use -- avoid -march=native and resolve to the actual flags such as -march=znver1 -- apply when using the compiling server to create whole binaries.) |
|
| Back to top |
|
|
Ionen
Developer
Joined: 06 Dec 2018
Posts: 2825
|
| Posted: Mon Sep 30, 2024 4:17 am Post subject: |
|
|
How I handle qtwebengine? Me I just frequently rebuild it before bumping it using the live ebuild in a temporary chroot, often with two different versions (6.7 and 6.8 right now), then build it again after I bumped it for my normal system, and another time on my VM I use for testing DEs.
Edit:
oh, also the llvm-musl chroot while praying it didn't break again, albeit I only do that one for new .0 branches. 6.8.0_rc seems ok, helps that Qt backported the clang19 fixes from upstream chromium.
I don't use distcc, do use ccache but that's mostly useless w/ qtwebengine unless rebuilding the same version (in which case it's amazing though). |
|
| Back to top |
|
|
jpsollie
Guru
Joined: 17 Aug 2013
Posts: 319
|
| Posted: Mon Sep 30, 2024 5:41 am Post subject: |
|
|
| eschwartz wrote: |
Is it not possible to ssh into the "compiling server" and spawn an emerge job there? If necessary you can even unpack a fresh stage3 tarball, chroot into it, configure it with the same make.conf as you have on your PC, and then use emerge --buildpkg qtwebengine. This is actually more or less what the official binhost does too, except you get to choose exactly what CFLAGS and USE flags you want. Then simply sync (or serve over https) the resulting binaries in /var/cache/binpkgs and load them on the PC.
|
ok ... but what about dependencies like eq qtcore?
*EDIT* : I assume you meant --builtpkgonly. It would be nice if you can pull everything into a chroot for compiling, otherwise you end up with a lot of use flags for dependencies compile-time dependencies on the no-GUI system you don't want...
_________________
The power of Gentoo optimization (not overclocked): [img]https://www.passmark.com/baselines/V10/images/503714802842.png[/img] |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2250
|
| Posted: Mon Sep 30, 2024 7:15 am Post subject: |
|
|
| jpsollie wrote: |
*EDIT* : I assume you meant --builtpkgonly. |
| man emerge: | --buildpkg [ y | n ], -b
Tells emerge to build binary packages for all ebuilds processed in addition to actually merging the packages. Useful for maintainers or if you administrate multiple Gentoo Linux systems (build
once, emerge tbz2s or gpkgs everywhere) as well as disaster recovery. The package will be created in the PKGDIR directory (see make.conf(5)). An alternative for already-merged packages is to use
quickpkg(1) which creates a tbz2 or gpkg from the live filesystem.
...
--buildpkgonly, -B
Creates binary packages for all ebuilds processed without actually merging the packages. This comes with the caveat that all build-time dependencies must already be emerged on the system.
|
Buildpkgonly will break because portage needs the packages merged to link against them. Essentially you'll have almost identical copy of your laptop's system in that chroot. I think you could even synchronize it with rsync but it would be easier if you used binary packages.
Also, why are you not able to run emerge or compile distcc on the build server? How did you built it first place? It doesn't require gui. Only ssh.
Best Regards,
Georgi |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
