| View previous topic :: View next topic |
| Author |
Message |
multix
Apprentice
Joined: 28 Nov 2004
Posts: 245
Location: Italy
|
 Posted: Fri Jun 21, 2024 8:20 am Post subject: on 686 memory exhausted while compiling clang 17.0.6 |
 |
|
When building | Code: | | sys-devel/clang-17.0.6 |
I get:
| Code: |
/usr/lib/gcc/i686-pc-linux-gnu/13/../../../../i686-pc-linux-gnu/bin/ld: can't relax section: memory exhausted
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
* ERROR: sys-devel/clang-17.0.6::gentoo failed (compile phase):
* ninja -v -j1 -l1 distribution failed
*
* Call stack:
* ebuild.sh, line 136: Called src_compile
* environment, line 4195: Called multilib-minimal_src_compile
* environment, line 3020: Called multilib_foreach_abi 'multilib-minimal_abi_src_compile'
* environment, line 3287: Called multibuild_foreach_variant '_multilib_multibuild_wrapper' 'multilib-minimal_abi_src_compile'
* environment, line 2980: Called _multibuild_run '_multilib_multibuild_wrapper' 'multilib-minimal_abi_src_compile'
* environment, line 2978: Called _multilib_multibuild_wrapper 'multilib-minimal_abi_src_compile'
* environment, line 679: Called multilib-minimal_abi_src_compile
* environment, line 3014: Called multilib_src_compile
* environment, line 3507: Called cmake_build 'distribution'
* environment, line 1462: Called eninja 'distribution'
* environment, line 1933: Called die
|
System has 2GB of RAM and 3GB of swap. I don't see any out of swap kernel messages in dmesg.
To be on the safe side, I added 1GB swap file and retried, same error. It doesn't support more RAM.
I wonder if I reached some 32bit addressing limit? This sytem has been going for years, but I bet now clang become fatter to compile and gcc more hungry at each cycle.
Perhaps there is a way to ease memory consumption?
| Quote: |
Portage 3.0.63 (python 3.12.3-final-0, default/linux/x86/23.0/i686/split-usr, gcc-13, glibc-2.39-r9, 6.6.30-gentoo i686)
=================================================================
System Settings
=================================================================
System uname: Linux-6.6.30-gentoo-i686-Intel-R-_Pentium-R-_M_processor_1.73GHz-with-glibc2.39
KiB Mem: 2059912 total, 1892384 free
KiB Swap: 4194296 total, 4185336 free
Timestamp of repository gentoo: Tue, 18 Jun 2024 00:45:00 +0000
Head commit of repository gentoo: 2bb21979b9dc9bd99c486971047593df4ff2ebda
sh bash 5.1_p16-r11
ld GNU ld (Gentoo 2.40 p7) 2.40.0
ccache version 4.9.1 [disabled]
app-misc/pax-utils: 1.3.7::gentoo
app-shells/bash: 5.1_p16-r11::gentoo
dev-build/autoconf: 2.13-r8::gentoo, 2.71-r7::gentoo
dev-build/automake: 1.16.5-r2::gentoo
dev-build/cmake: 3.28.5::gentoo
dev-build/libtool: 2.4.7-r4::gentoo
dev-build/make: 4.4.1-r1::gentoo
dev-build/meson: 1.4.0-r1::gentoo
dev-lang/perl: 5.38.2-r3::gentoo
dev-lang/python: 2.7.18_p16-r2::gentoo, 3.11.9::gentoo, 3.12.3-r1::gentoo
dev-lang/rust-bin: 1.77.1::gentoo
dev-util/ccache: 4.9.1-r1::gentoo
sys-apps/baselayout: 2.15::gentoo
sys-apps/openrc: 0.54::gentoo
sys-apps/sandbox: 2.38::gentoo
sys-devel/binutils: 2.40-r9::gentoo, 2.41-r5::gentoo, 2.42-r1::gentoo
sys-devel/binutils-config: 5.5::gentoo
sys-devel/clang: 17.0.6::gentoo
sys-devel/gcc: 13.2.1_p20240210::gentoo
sys-devel/gcc-config: 2.11::gentoo
sys-devel/llvm: 17.0.6::gentoo
sys-kernel/linux-headers: 6.6-r1::gentoo (virtual/os-headers)
sys-libs/glibc: 2.39-r9::gentoo
Repositories:
gentoo
location: /usr/portage
sync-type: rsync
sync-uri: rsync://rsync.gentoo.org/gentoo-portage
priority: -1000
volatile: True
sync-rsync-verify-jobs: 1
sync-rsync-verify-max-age: 3
sync-rsync-extra-opts:
sync-rsync-verify-metamanifest: yes
ACCEPT_KEYWORDS="x86"
ACCEPT_LICENSE="@FREE linux-firmware no-source-code ipw2200-fw MSttfEULA linux-fw-redistributable free-noncomm"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-O2 -march=native -pipe"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -march=native -pipe"
DISTDIR="/usr/portage/distfiles"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME"
FCFLAGS="-O2 -march=i686 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg-live config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync merge-wait multilib-strict network-sandbox news parallel-fetch pid-sandbox pkgdir-index-trusted preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -march=i686 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,-z,pack-relative-relocs"
LEX="flex"
LINGUAS="en it de"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
SHELL="/bin/bash"
USE="acl bindist bzip2 crypt gdbm iconv ipv6 libtirpc ncurses nls objc offensive openmp pam pcre readline seccomp split-usr ssl test-rust unicode x86 xattr zlib" ABI_X86="32" ADA_TARGET="gcc_12" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_anon authn_dbm authn_file authz_dbm authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir env expires ext_filter file_cache filter headers include info log_config logio mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 ntrip navcom oceanserver oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 tsip tripmate tnt ublox" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz glk hd44780 lb216 lcdm001 mtxorb text" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php8-2" POSTGRES_TARGETS="postgres15" PYTHON_SINGLE_TARGET="python3_12" PYTHON_TARGETS="python3_12" RUBY_TARGETS="ruby31 ruby32" VIDEO_CARDS="radeon r300" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipp2p iface geoip fuzzy condition tarpit sysrq proto logmark ipmark dhcpmac delude chaos account"
Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EMERGE_DEFAULT_OPTS, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LANG, LC_ALL, LD, LFLAGS, LIBTOOL, MAKE, MAKEFLAGS, MAKEOPTS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PYTHONPATH, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS
=================================================================
Package Settings
=================================================================
sys-devel/clang-17.0.6::gentoo was built with the following:
USE="extra (pie) static-analyzer -debug -doc (-ieee-long-double) -test -verify-sig -xml" LLVM_TARGETS="(AArch64) (AMDGPU) (ARM) (AVR) (BPF) (Hexagon) (Lanai) (LoongArch) (MSP430) (Mips) (NVPTX) (PowerPC) (RISCV) (Sparc) (SystemZ) (VE) (WebAssembly) (X86) (XCore) -ARC -CSKY -DirectX -M68k -SPIRV -Xtensa" PYTHON_SINGLE_TARGET="python3_11 -python3_10 -python3_12"
FEATURES="fixlafiles merge-sync unmerge-logs multilib-strict binpkg-docompress news assume-digests userpriv qa-unresolved-soname-deps userfetch ipc-sandbox buildpkg-live config-protect-if-modified ebuild-locks unknown-features-warn protect-owned pid-sandbox binpkg-dostrip distlocks xattr sandbox usersync network-sandbox preserve-libs sfperms parallel-fetch usersandbox binpkg-logs pkgdir-index-trusted unmerge-orphans strict"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
|
Removing targets is, as far as I understand, not possible anymore for llvm.
_________________
Unix is user friendly. It is only careful selecting its friends. |
|
| Back to top |
|
 |
Banana
Moderator
Joined: 21 May 2004
Posts: 1737
Location: Germany
|
|
| Back to top |
|
|
Ralphred
l33t
Joined: 31 Dec 2013
Posts: 653
|
| Posted: Sat Jun 22, 2024 2:30 am Post subject: Re: on 686 memory exhausted while compiling clang 17.0.6 |
|
|
| multix wrote: | | I wonder if I reached some 32bit addressing limit? |
As far as I can remember, this was fixed in linux years ago, there was a "PAE" patch/extension for the kernel that allowed you to go over the 4gig limit. I think it became kind of ubiquitous, so I'd just test it with an 8gig swapfile and see what happens... |
|
| Back to top |
|
|
RumpletonBongworth
Tux's lil' helper
Joined: 17 Jun 2024
Posts: 77
|
| Posted: Sat Jun 22, 2024 5:05 pm Post subject: |
|
|
| Even with PAE, I'm fairly sure that the virtual address space available to an individual process remains limited. For a system with 2GB of RAM, the Kconfig help text encourages the user to enable HIGHMEM4G, with disables PAE. Further, the default is to split the address space in such a way that there is a 3G/1G user/kernel split. If the EXPERT option is enabled, the user/kernel boundary can be adjusted, albeit only at the expense of userspace. Reportedly, there is (or was) a patch that allows for a 4G/4G split but it carries with it a performance impact and was never mainlined (probably also due to x86_64 taking over). |
|
| Back to top |
|
|
RumpletonBongworth
Tux's lil' helper
Joined: 17 Jun 2024
Posts: 77
|
| Posted: Sat Jun 22, 2024 5:37 pm Post subject: |
|
|
| I'm not sure whether it will make all that much difference but you could try without "-pipe" in your *FLAGS. |
|
| Back to top |
|
|
multix
Apprentice
Joined: 28 Nov 2004
Posts: 245
Location: Italy
|
| Posted: Mon Jun 24, 2024 9:23 am Post subject: |
|
|
| Banana wrote: | | https://wiki.gentoo.org/wiki/Knowledge_Base:Emerge_out_of_memory |
A check shows that I wasn't bad set there:MAKEOPTS and EMERGE_DEFAULT_OPTS empty
I do not try to run in parallel, this is a single-core CPU anyway, wouldn't make sense.
I tried adding swap through swapfiles. +2GB more for a total of 6GB. No help.
But it somehow makes sense, I htink the process exhausted addressable space, it wasn't killed because out of swap.
I will try
| Code: | LDFLAGS="${LDFLAGS} -Wl,--no-keep-memory"
|
And report back. I think this can affect other persons, there is nothing special in this sytem.
_________________
Unix is user friendly. It is only careful selecting its friends. |
|
| Back to top |
|
|
multix
Apprentice
Joined: 28 Nov 2004
Posts: 245
Location: Italy
|
| Posted: Mon Jun 24, 2024 10:04 am Post subject: |
|
|
| RumpletonBongworth wrote: | | I'm not sure whether it will make all that much difference but you could try without "-pipe" in your *FLAGS. |
I already have:
| Code: |
CFLAGS="-O2 -march=native -pipe"
CXXFLAGS="${CFLAGS}"
|
I suppose this should cover it?
_________________
Unix is user friendly. It is only careful selecting its friends. |
|
| Back to top |
|
|
RumpletonBongworth
Tux's lil' helper
Joined: 17 Jun 2024
Posts: 77
|
| Posted: Mon Jun 24, 2024 1:38 pm Post subject: |
|
|
| multix wrote: | | I suppose this should cover it? |
Without, not with. The reasoning is that, as a form of interprocess communication, using pipes will consume more memory than will using temporary files. |
|
| Back to top |
|
|
eccerr0r
Watchman
Joined: 01 Jul 2004
Posts: 9824
Location: almost Mile High in the USA
|
| Posted: Mon Jun 24, 2024 3:57 pm Post subject: |
|
|
probably worth to enable PAE because it allows more user mode data and keeps kernel mode in its own address space. There's a speed penalty but allows up to that 4GiB addressing space (so you can use the 2GiB RAM plus 2GiB swap per process) and also works around the meltdown bug.
Yes it's probably kicking the can down the road. Ensure you don't have LTO enabled. And there are some Pentium M's that seem to mark no PAE support but actually can run PAE just fine if you specify forcepae on your boot command line (and have it enabled in your kernel).
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
| Back to top |
|
|
multix
Apprentice
Joined: 28 Nov 2004
Posts: 245
Location: Italy
|
| Posted: Mon Jun 24, 2024 11:35 pm Post subject: |
|
|
| RumpletonBongworth wrote: | | multix wrote: | | I suppose this should cover it? |
Without, not with. The reasoning is that, as a form of interprocess communication, using pipes will consume more memory than will using temporary files. |
ah.. ok, will try without. Was done for speed back then.. but if it consumes memory...
The issue is linkning.
| Code: | | -Wl,--no-keep-memory -Wl,--no-keep-memory |
is present twice, so I suppose it was being added already
I also noticed that the amount of stuff being linked is humongous. Hundreds of object files.
| Code: | | [1226/1947] : && /usr/bin/i686-pc-linux-gnu-g++ -fPIC -O2 -march=native -pipe -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -fno-lifetime-dse -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -Wimplicit-fallthrough -Wno-nonnull -Wno-class-memaccess -Wno-redundant-move -Wno-pessimizing-move -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wno-misleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual -fno-strict-aliasing -pedantic -Wno-long-long -Wl,-O1 -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,--no-keep-memory -Wl,--no-keep-memory -Wl,-z,defs -Wl,-z,nodelete -Wl,-rpath-link,/var/tmp/portage/sys-devel/clang-17.0.6/work/x/y/clang-abi_x86_32.x86/./lib -Wl,--gc-sections -Wl,-Bsymbolic-functions -shared -Wl,-soname,libclang-cpp.so.17 -o lib/libclang-cpp.so.17 lib/Basic/CMakeFiles/obj.clangBasic.dir/Attributes.cpp.o lib/Basic/CMakeFiles/obj.clangBasic.dir/Builtins.cpp.o lib/Basic/CMakeFiles/obj.clangBasic.dir/CLWarnings.cpp.o lib/Basic/CMakeFiles/obj.clangBasic.dir/CharInfo.cpp.o lib/Basic/CMakeFiles/obj.clangBasic.dir/CodeGenOptions.cpp.o lib/Basic/CMakeFiles/obj.clangBasic.dir/Cuda.cpp.o lib/Basic/ |
Where to I see if LTO is enabled? I don't see options, do I need to specifically enable or disable it?
_________________
Unix is user friendly. It is only careful selecting its friends. |
|
| Back to top |
|
|
RumpletonBongworth
Tux's lil' helper
Joined: 17 Jun 2024
Posts: 77
|
| Posted: Tue Jun 25, 2024 11:04 pm Post subject: |
|
|
| multix wrote: | | Where to I see if LTO is enabled? I don't see options, do I need to specifically enable or disable it? |
As far as I am aware, LTO is one of those things that would need to be opted into (with the -flto flag). |
|
| Back to top |
|
|
multix
Apprentice
Joined: 28 Nov 2004
Posts: 245
Location: Italy
|
| Posted: Fri Jun 28, 2024 9:46 am Post subject: |
|
|
| eccerr0r wrote: | probably worth to enable PAE because it allows more user mode data and keeps kernel mode in its own address space. There's a speed penalty but allows up to that 4GiB addressing space (so you can use the 2GiB RAM plus 2GiB swap per process) and also works around the meltdown bug.
Yes it's probably kicking the can down the road. Ensure you don't have LTO enabled. And there are some Pentium M's that seem to mark no PAE support but actually can run PAE just fine if you specify forcepae on your boot command line (and have it enabled in your kernel). |
Isn't PAE available for all Pentium Pro's and successors? My CPU shows "pae" in cpuinfo flags, so I suppose it is supported and should work.
I wonder if it helps - it whould be Physical Address Exteionsion and also the kernel option to enable it has to be set to 64GB of Physical RAM.
Anyway, I built the new kernel, but it did not help. Still get out of memory... maybe it needs more than 2GB + 2GB per process or it is not working.
How can I confirm pae is correctly enabled and working?
_________________
Unix is user friendly. It is only careful selecting its friends. |
|
| Back to top |
|
|
eccerr0r
Watchman
Joined: 01 Jul 2004
Posts: 9824
Location: almost Mile High in the USA
|
| Posted: Sat Jun 29, 2024 12:20 am Post subject: |
|
|
I was able to build clang-17 and llvm-17 on 32-bit with 2GiB RAM (this is a VM with 2GiB RAM allocation). Something strange going on here...
If you built a PAE kernel it won't run on a machine without PAE, though I don't think my VM is using a PAE kernel actually...
---
I just failed a compile of clang-18 on a 32-bit machine with 2GiB RAM (baremetal) and HIGHMEM4G, so time to investigate...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
