| View previous topic :: View next topic |
| Author |
Message |
Dwosky
Tux's lil' helper
Joined: 07 Nov 2018
Posts: 141
|
 Posted: Mon Nov 21, 2022 11:54 am Post subject: Does dev-lang/rust-1.65.0 need more than 20G of tmp? |
 |
|
I've upgraded the SSD on my system and now that I had more space I wanted to swap between dev-lang/rust-bin and dev-lang/rust. I've given 20G of size to /var/tmp/portage, but it seems rust is using all the space and ends giving me a "No space left on device" error. Also, I know this can vary a lot from PC to PC, but I think the compilation times are way larger for this package, so not sure if there is any other issue that might be triggering. In my case, i3-3220T CPU @ 2.80GHz and 8Gb of RAM takes more than 24 hours to reach this state.
Any idea why it needs some much more space than the space checked on the pre build phase? Or if its just a matter of space to be able to compile it successfully?
| Code: | >>> Verifying ebuild manifests
>>> Running pre-merge checks for dev-lang/rust-1.65.0
* Checking for at least 11520 MiB disk space at "/var/tmp/portage/dev-lang/rust-1.65.0/temp [ ok ]
>>> Emerging (1 of 1) dev-lang/rust-1.65.0::gentoo |
This is the filesystem state once emerge fails:
| Code: | # df -h /var/tmp/portage
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/static-var_tmp 20G 19G 0 100% /var/tmp
# du -h --max-depth 1 /var/tmp/portage/dev-lang/rust-1.65.0
8.0K /var/tmp/portage/dev-lang/rust-1.65.0/homedir
19G /var/tmp/portage/dev-lang/rust-1.65.0/work
20M /var/tmp/portage/dev-lang/rust-1.65.0/temp
4.0K /var/tmp/portage/dev-lang/rust-1.65.0/empty
4.0K /var/tmp/portage/dev-lang/rust-1.65.0/.ipc
8.0K /var/tmp/portage/dev-lang/rust-1.65.0/distdir
28K /var/tmp/portage/dev-lang/rust-1.65.0/build-info
19G /var/tmp/portage/dev-lang/rust-1.65.0 |
Here's the package use flags:
| Code: | # emerge -pqv '=dev-lang/rust-1.65.0::gentoo'
[ebuild N ] dev-lang/rust-1.65.0 USE="-clippy -debug -dist -doc (-llvm-libunwind) (-miri) (-nightly) (-parallel-compiler) (-profiler) -rls -rust-analyzer -rust-src -rustfmt (-system-bootstrap) (-system-llvm) -test -verify-sig -wasm" ABI_X86="(64) -32 (-x32)" CPU_FLAGS_X86="sse2" LLVM_TARGETS="(X86) -AArch64 -AMDGPU -ARM -AVR -BPF -Hexagon -Lanai -MSP430 -Mips -NVPTX -PowerPC -RISCV -Sparc -SystemZ -WebAssembly -XCore" |
And the emerge info:
| Code: | # emerge --info
Portage 3.0.38.1 (python 3.10.8-final-0, default/linux/amd64/17.1, gcc-11.3.0, glibc-2.36-r5, 5.15.75-gentoo x86_64)
=================================================================
System uname: Linux-5.15.75-gentoo-x86_64-Intel-R-_Core-TM-_i3-3220T_CPU_@_2.80GHz-with-glibc2.36
KiB Mem: 7832368 total, 3292124 free
KiB Swap: 4194300 total, 3710360 free
Timestamp of repository gentoo: Fri, 18 Nov 2022 00:00:01 +0000
Head commit of repository gentoo: 1baab4b640ed925d97e91f0e768abe39c87777b8
sh bash 5.1_p16-r2
ld GNU ld (Gentoo 2.38 p4) 2.38
distcc[26333] (dcc_mkdir) ERROR: mkdir '/var/tmp/portage/.distcc/state' failed: No such file or directory [disabled]
ccache version 4.5.1 [enabled]
app-misc/pax-utils: 1.3.5::gentoo
app-shells/bash: 5.1_p16-r2::gentoo
dev-lang/perl: 5.36.0-r1::gentoo
dev-lang/python: 3.10.8_p3::gentoo, 3.11.0_p2::gentoo
dev-util/ccache: 4.5.1::gentoo
dev-util/cmake: 3.24.2::gentoo
dev-util/meson: 0.63.2-r1::gentoo
sys-apps/baselayout: 2.9::gentoo
sys-apps/openrc: 0.45.2-r1::gentoo
sys-apps/sandbox: 2.29::gentoo
sys-devel/autoconf: 2.71-r1::gentoo
sys-devel/automake: 1.16.5::gentoo
sys-devel/binutils: 2.38-r2::gentoo
sys-devel/binutils-config: 5.4.1::gentoo
sys-devel/gcc: 11.3.0::gentoo
sys-devel/gcc-config: 2.8::gentoo
sys-devel/libtool: 2.4.7::gentoo
sys-devel/llvm: 14.0.6-r2::gentoo, 15.0.3::gentoo
sys-devel/make: 4.3::gentoo
sys-kernel/linux-headers: 5.15-r3::gentoo (virtual/os-headers)
sys-libs/glibc: 2.36-r5::gentoo
Repositories:
dwosky
location: /var/lib/layman/dwosky
masters: gentoo
priority: -50
crossdev
location: /usr/local/portage-crossdev
masters: gentoo
priority: -10
gentoo
location: /usr/portage
sync-type: rsync
sync-uri: rsync://rsync.gentoo.org/gentoo-portage
priority: 1000
sync-rsync-verify-max-age: 24
sync-rsync-verify-metamanifest: yes
sync-rsync-extra-opts:
sync-rsync-verify-jobs: 1
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="@FREE"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=ivybridge -mno-rdrnd --param=l1-cache-line-size=64 --param=l1-cache-size=32 --param=l2-cache-size=3072"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php8.1/ext-active/ /etc/php/cgi-php8.1/ext-active/ /etc/php/cli-php8.1/ext-active/ /etc/php/fpm-php8.1/ext-active/ /etc/php/phpdbg-php8.1/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=ivybridge -mno-rdrnd --param=l1-cache-line-size=64 --param=l1-cache-size=32 --param=l2-cache-size=3072"
DISTDIR="/usr/portage/distfiles"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg-live ccache config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en es"
MAKEOPTS="-j4"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
SHELL="/bin/bash"
USE="acl amd64 avx bzip2 cli crypt dbus dri fortran gd gdbm iconv libglvnd libtirpc mmx mmxext multilib ncurses nls nptl openmp pam pcre png popcnt readline seccomp split-usr sse sse2 sse3 sse4_1 sse4_2 ssl ssse3 test-rust threads truetype udev unicode xattr zlib" ABI_X86="64" ADA_TARGET="gnat_2021" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="avx f16c mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" NGINX_MODULES_HTTP="access auth_basic browser fastcgi geo geoip grpc gzip limit_conn limit_req map memcached mirror proxy rewrite scgi upstream_keepalive userid stub_status" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-4 php8-0" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_10" PYTHON_TARGETS="python3_10" RUBY_TARGETS="ruby27" USERLAND="GNU" VIDEO_CARDS="i915 intel" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EMERGE_DEFAULT_OPTS, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LEX, LFLAGS, LIBTOOL, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS |
|
|
| Back to top |
|
 |
fedeliallalinea
Administrator
Joined: 08 Mar 2003
Posts: 31478
Location: here
|
| Posted: Mon Nov 21, 2022 12:18 pm Post subject: |
|
|
I don't know how the rust build system works, but it takes up a lot of space.
That is why there is also the rust-bin package.
See bug #783768.
_________________
Questions are guaranteed in life; Answers aren't. |
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 5396
Location: Bavaria
|
| Posted: Mon Nov 21, 2022 12:29 pm Post subject: |
|
|
Dwosky,
your system has only 8 GB ram; this is not much for compiling rust with "-j4" ... an additional usage of your ram for a tmpfs is clearly not possible. So, I would recommend you these settings:
| Code: | # mkdir /etc/portage/env
# nano -w /etc/portage/env/monster.conf
=>
MAKEOPTS="-j2"
<=
# mkdir -p /var/tmp/notmpfs
# chown portage:portage /var/tmp/notmpfs
# nano -w /etc/portage/env/notmpfs.conf
=>
PORTAGE_TMPDIR="/var/tmp/notmpfs"
<=
# nano -w /etc/portage/package.env
=>
# app-office/libreoffice monster.conf
dev-lang/rust monster.conf notmpfs.conf
# dev-qt/qtwebengine monster.conf
# sys-devel/gcc monster.conf
<= |
(taken from my two articles here: https://forums.gentoo.org/viewtopic-p-8595924.html#8595924 ) |
|
| Back to top |
|
|
Dwosky
Tux's lil' helper
Joined: 07 Nov 2018
Posts: 141
|
| Posted: Mon Nov 21, 2022 12:37 pm Post subject: |
|
|
Right now I'm running it with -j3:
| Code: | $ cat /etc/portage/env/lower_makeops.conf
MAKEOPTS="-j3"
$ cat /etc/portage/package.env/lower_makeops
sys-devel/llvm lower_makeops.conf
dev-lang/rust lower_makeops.conf |
But what's the point on having another directory in the same filesystem of /var/tmp (in my case)? |
|
| Back to top |
|
|
mike155
Advocate
Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany
|
| Posted: Mon Nov 21, 2022 12:56 pm Post subject: |
|
|
Below are the results on my machine (AMD Zen 3 processor, x86_64, no-multilib, -j8 )
| Code: | # ebuild /var/db/repos/gentoo/dev-lang/rust/rust-1.65.0.ebuild compile
24K /var/tmp/portage/dev-lang/rust-1.65.0/build-info
0 /var/tmp/portage/dev-lang/rust-1.65.0/distdir
11G /var/tmp/portage/dev-lang/rust-1.65.0/work
0 /var/tmp/portage/dev-lang/rust-1.65.0/.ipc
23M /var/tmp/portage/dev-lang/rust-1.65.0/temp
0 /var/tmp/portage/dev-lang/rust-1.65.0/homedir
12G /var/tmp/portage/dev-lang/rust-1.65.0 |
| Code: | # ebuild /var/db/repos/gentoo/dev-lang/rust/rust-1.65.0.ebuild install
286M /var/tmp/portage/dev-lang/rust-1.65.0/image
0 /var/tmp/portage/dev-lang/rust-1.65.0/homedir
184K /var/tmp/portage/dev-lang/rust-1.65.0/build-info
0 /var/tmp/portage/dev-lang/rust-1.65.0/distdir
12G /var/tmp/portage/dev-lang/rust-1.65.0/work
0 /var/tmp/portage/dev-lang/rust-1.65.0/.ipc
45M /var/tmp/portage/dev-lang/rust-1.65.0/temp
13G /var/tmp/portage/dev-lang/rust-1.65.0 |
What ebuild estimated:
| Code: | | Checking for at least 11904 MiB disk space at "/var/tmp/portage/dev-lang/rust-1.65.0/temp" ... |
My USE flags:
| Code: | # emerge -pqv '=dev-lang/rust-1.65.0::gentoo'
[ebuild R ] dev-lang/rust-1.65.0 USE="clippy rustfmt -debug -dist -doc (-llvm-libunwind) (-miri) (-nightly) (-parallel-compiler) (-profiler) -rls -rust-analyzer -rust-src (-system-bootstrap) (-system-llvm) -test -verify-sig -wasm" CPU_FLAGS_X86="sse2" LLVM_TARGETS="(X86) -AArch64 -AMDGPU -ARM -AVR -BPF -Hexagon -Lanai -MSP430 -Mips -NVPTX -PowerPC -RISCV -Sparc -SystemZ -WebAssembly -XCore" |
|
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 5396
Location: Bavaria
|
| Posted: Mon Nov 21, 2022 1:57 pm Post subject: |
|
|
| Dwosky wrote: | | But what's the point on having another directory in the same filesystem of /var/tmp (in my case)? |
The point is: Not using ram for /var/tmp/portage because /var/tmp/notmpfs is not a tmpfs, just a simple directory on your hd (or sdd). |
|
| Back to top |
|
|
Dwosky
Tux's lil' helper
Joined: 07 Nov 2018
Posts: 141
|
| Posted: Mon Nov 21, 2022 2:15 pm Post subject: |
|
|
| mike155 wrote: | | Below are the results on my machine (AMD Zen 3 processor, x86_64, no-multilib, -j8 ) |
Got another PC with an i7 and same MAKEOPTS that more or less fits in your example, it has a filesystem in /var/tmp of 20G and I can compile rust there without issues, so despite not having measured, I suppose it doesn't use as much as disk space as this other PC. Seeing the bug pointed out in the thread makes me think it might vary from different CPU types/configs (CFLAGS).
| pietinger wrote: | | The point is: Not using ram for /var/tmp/portage because /var/tmp/notmpfs is not a tmpfs, just a simple directory on your hd (or sdd). |
I understand the point, the thing is that I'm not using a tmpfs filesystem for /var/tmp/portage, the portage folder is inside a filesystem in the SSD of 20G, that's why I don't understand the recommendation, since creating the folder /var/tmp/notmpfs would end writing to the same filesystem.
| Code: | # df -h /var/tmp/portage
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/static-var_tmp 20G 19G 0 100% /var/tmp |
|
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 5396
Location: Bavaria
|
| Posted: Mon Nov 21, 2022 2:54 pm Post subject: |
|
|
| Dwosky wrote: | | I understand the point, the thing is that I'm not using a tmpfs filesystem for /var/tmp/portage, the portage folder is inside a filesystem in the SSD of 20G, that's why I don't understand the recommendation, since creating the folder /var/tmp/notmpfs would end writing to the same filesystem. |
Sorry for my misunderstanding. I thought you are using a tmpfs because you wrote in your first post "I've given 20G of size to /var/tmp/portage" ... I thought it is a tmpfs ...  |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 23103
|
| Posted: Mon Nov 21, 2022 3:13 pm Post subject: Re: Does dev-lang/rust-1.65.0 need more than 20G of tmp? |
|
|
| Dwosky wrote: | | Any idea why it needs some much more space than the space checked on the pre build phase? |
The space checked in the setup phase is an estimate set by the Gentoo maintainer based on observed results and previously observed space exhaustion reports. It could be grossly optimistic or pessimistic, and may rot with newer rust versions if upstream changes how they build. The maintainer likely tries to keep it accurate, but based on the current code, the correct value depends heavily on the user's chosen configuration options, so it's quite possible that you have some combination for which the maintainer did not account. Once you get this to work, please note what was actually used, and consider filing a bug to have the requirement revised. |
|
| Back to top |
|
|
mike155
Advocate
Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany
|
| Posted: Mon Nov 21, 2022 3:58 pm Post subject: |
|
|
It is probably related to CFLAGS and CXXFLAGS.
With CFLAGS=CXXFLAGS="-march=native -O2 -pipe":
| Code: | # ebuild /var/db/repos/gentoo/dev-lang/rust/rust-1.65.0.ebuild compile
24K /var/tmp/portage/dev-lang/rust-1.65.0/build-info
0 /var/tmp/portage/dev-lang/rust-1.65.0/distdir
11G /var/tmp/portage/dev-lang/rust-1.65.0/work
0 /var/tmp/portage/dev-lang/rust-1.65.0/.ipc
23M /var/tmp/portage/dev-lang/rust-1.65.0/temp
0 /var/tmp/portage/dev-lang/rust-1.65.0/homedir
12G /var/tmp/portage/dev-lang/rust-1.65.0 |
With CFLAGS=CXXFLAGS="-march=ivybridge -mno-rdrnd --param=l1-cache-line-size=64 --param=l1-cache-size=32 --param=l2-cache-size=3072":
| Code: | # ebuild /var/db/repos/gentoo/dev-lang/rust/rust-1.65.0.ebuild compile
OSError: [Errno 28] No space left on device
du -h --max-depth 1 /var/tmp/portage/dev-lang/rust-1.65.0
0 /var/tmp/portage/dev-lang/rust-1.65.0/empty
24K /var/tmp/portage/dev-lang/rust-1.65.0/build-info
0 /var/tmp/portage/dev-lang/rust-1.65.0/distdir
16G /var/tmp/portage/dev-lang/rust-1.65.0/work
0 /var/tmp/portage/dev-lang/rust-1.65.0/.ipc
67M /var/tmp/portage/dev-lang/rust-1.65.0/temp
0 /var/tmp/portage/dev-lang/rust-1.65.0/homedir
16G /var/tmp/portage/dev-lang/rust-1.65.0 |
|
|
| Back to top |
|
|
fedeliallalinea
Administrator
Joined: 08 Mar 2003
Posts: 31478
Location: here
|
| Posted: Mon Nov 21, 2022 4:24 pm Post subject: |
|
|
| mike155 wrote: | | It is probably related to CFLAGS and CXXFLAGS. |
Couldn't it also take up more space during compilation? I guess the values you give is at the end of compilation.
_________________
Questions are guaranteed in life; Answers aren't. |
|
| Back to top |
|
|
Christian99
Veteran
Joined: 28 May 2009
Posts: 1721
|
| Posted: Mon Nov 21, 2022 4:25 pm Post subject: |
|
|
| mike155 wrote: | It is probably related to CFLAGS and CXXFLAGS.
With CFLAGS=CXXFLAGS="-march=native -O2 -pipe":
| Code: | # ebuild /var/db/repos/gentoo/dev-lang/rust/rust-1.65.0.ebuild compile
24K /var/tmp/portage/dev-lang/rust-1.65.0/build-info
0 /var/tmp/portage/dev-lang/rust-1.65.0/distdir
11G /var/tmp/portage/dev-lang/rust-1.65.0/work
0 /var/tmp/portage/dev-lang/rust-1.65.0/.ipc
23M /var/tmp/portage/dev-lang/rust-1.65.0/temp
0 /var/tmp/portage/dev-lang/rust-1.65.0/homedir
12G /var/tmp/portage/dev-lang/rust-1.65.0 |
With CFLAGS=CXXFLAGS="-march=ivybridge -mno-rdrnd --param=l1-cache-line-size=64 --param=l1-cache-size=32 --param=l2-cache-size=3072":
| Code: | # ebuild /var/db/repos/gentoo/dev-lang/rust/rust-1.65.0.ebuild compile
OSError: [Errno 28] No space left on device
du -h --max-depth 1 /var/tmp/portage/dev-lang/rust-1.65.0
0 /var/tmp/portage/dev-lang/rust-1.65.0/empty
24K /var/tmp/portage/dev-lang/rust-1.65.0/build-info
0 /var/tmp/portage/dev-lang/rust-1.65.0/distdir
16G /var/tmp/portage/dev-lang/rust-1.65.0/work
0 /var/tmp/portage/dev-lang/rust-1.65.0/.ipc
67M /var/tmp/portage/dev-lang/rust-1.65.0/temp
0 /var/tmp/portage/dev-lang/rust-1.65.0/homedir
16G /var/tmp/portage/dev-lang/rust-1.65.0 |
|
Interesting find. Really surprised that it makes such a big difference. From the flags alone, I wouldn't have guessed that they increase the size so much.
I would guess, that the missing O2 flag makes the biggest differnce in build, though I wouldn't have guessed that leaving this flag out increases the object files so much.
@Dwosky I would try building again with O2 enabled and see what it changes.
BTW, did you mean to leave out O2 on purpose, or is this just an oversight? |
|
| Back to top |
|
|
Christian99
Veteran
Joined: 28 May 2009
Posts: 1721
|
| Posted: Mon Nov 21, 2022 4:28 pm Post subject: |
|
|
| fedeliallalinea wrote: | | mike155 wrote: | | It is probably related to CFLAGS and CXXFLAGS. |
Couldn't it also take up more space during compilation? I guess the values you give is at the end of compilation. |
I would think, that the compilation just produces object files that are written to disk and just stay there. Don't think, that a build system cleans up object files during build process, so I would expected the diskusage to be monotonically growing. |
|
| Back to top |
|
|
fedeliallalinea
Administrator
Joined: 08 Mar 2003
Posts: 31478
Location: here
|
| Posted: Mon Nov 21, 2022 4:36 pm Post subject: |
|
|
| Christian99 wrote: | | I would think, that the compilation just produces object files that are written to disk and just stay there. Don't think, that a build system cleans up object files during build process, so I would expected the diskusage to be monotonically growing. |
You're probably right.
_________________
Questions are guaranteed in life; Answers aren't. |
|
| Back to top |
|
|
Ionen
Developer
Joined: 06 Dec 2018
Posts: 2893
|
| Posted: Mon Nov 21, 2022 4:42 pm Post subject: |
|
|
Still not sure what's up with this, seen a few rare users get to ridiculous sizes (e.g. 65GB), but for most it's more a reasonable 12-13GB afaik, around what the ebuild checks for. Think specific to 1.65? Not sure I heard reports for older.
My own last build was tiny, but I cheat with system-llvm+bootstrap | Code: | >>> Completed installing dev-lang/rust-1.65.0 into /var/tmp/portage/dev-lang/rust-1.65.0/image
* Final size of build directory: 6637064 KiB ( 6.3 GiB)
* Final size of installed tree: 400444 KiB (391.0 MiB) |
|
|
| Back to top |
|
|
sdauth
l33t
Joined: 19 Sep 2018
Posts: 679
Location: Ásgarðr
|
| Posted: Mon Nov 21, 2022 7:08 pm Post subject: |
|
|
| Ionen wrote: | | My own last build was tiny, but I cheat with system-llvm+bootstrap |
How to enable that ? I have set this in my package.use : dev-lang/rust system-bootstrap system-llvm rustfmt
but when running equery u on dev-lang/rust, these USE flags are not showing up. (although they do exist in the ebuild, I don't understand) |
|
| Back to top |
|
|
mike155
Advocate
Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany
|
| Posted: Mon Nov 21, 2022 7:32 pm Post subject: |
|
|
| Christian99 wrote: | | I would guess, that the missing O2 flag makes the biggest difference in build, though I wouldn't have guessed that leaving this flag out increases the object files so much. |
You're right!
With CFLAGS = CXXFLAGS = "-march=native -O2 -pipe":
| Code: | # ebuild /var/db/repos/gentoo/dev-lang/rust/rust-1.65.0.ebuild compile
11G /var/tmp/portage/dev-lang/rust-1.65.0/work
|
With CFLAGS = CXXFLAGS = "-march=native -pipe":
| Code: | # ebuild /var/db/repos/gentoo/dev-lang/rust/rust-1.65.0.ebuild compile
20G /var/tmp/portage/dev-lang/rust-1.65.0/work
|
Without -O2, the build process takes much longer: 150 minutes instead of 30 minutes.
Something is wrong without -O2.
Below is a list of the 30 largest files (without -O2):
| Code: | # cd /var/tmp/portage/dev-lang/rust-1.65.0/work/rustc-1.65.0-src/build/x86_64-unknown-linux-gnu
# find . -type f -ls | sort -n -k 7 | tail -n 30
144.245.696 Nov 21 18:12 ./llvm/bin/llc
146.292.320 Nov 21 18:13 ./llvm/build/bin/llvm-opt-fuzzer
157.125.424 Nov 21 18:12 ./llvm/build/bin/llvm-lto
157.125.424 Nov 21 18:12 ./llvm/bin/llvm-lto
157.239.840 Nov 21 18:13 ./llvm/build/bin/llvm-lto2
157.239.840 Nov 21 18:13 ./llvm/bin/llvm-lto2
159.116.312 Nov 21 18:12 ./llvm/build/lib/libLTO.so.15-rust-1.65.0-stable
159.116.312 Nov 21 18:12 ./llvm/lib/libLTO.so.15-rust-1.65.0-stable
160.663.976 Nov 21 18:12 ./llvm/build/bin/bugpoint
160.663.976 Nov 21 18:12 ./llvm/bin/bugpoint
163.118.620 Nov 21 18:11 ./llvm/build/lib/libLLVMCodeGen.a
163.118.620 Nov 21 18:11 ./llvm/lib/libLLVMCodeGen.a
174.442.656 Nov 21 18:14 ./llvm/build/bin/opt
174.442.656 Nov 21 18:14 ./stage1/lib/rustlib/x86_64-unknown-linux-gnu/bin/opt
174.442.656 Nov 21 18:14 ./stage2/lib/rustlib/x86_64-unknown-linux-gnu/bin/opt
174.442.656 Nov 21 18:14 ./llvm/bin/opt
294.712.928 Nov 21 19:35 ./stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-ef00e77318b56011.so
294.712.928 Nov 21 19:35 ./stage1-rustc/x86_64-unknown-linux-gnu/release/librustc_driver.so
294.712.928 Nov 21 19:35 ./stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_driver-ef00e77318b56011.so
294.712.928 Nov 21 19:35 ./stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_driver-ef00e77318b56011.so
294.712.928 Nov 21 19:35 ./stage2/lib/librustc_driver-ef00e77318b56011.so
300.189.864 Nov 21 18:18 ./stage0-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_driver-18613274f1c25ece.so
300.189.864 Nov 21 18:18 ./stage0-rustc/x86_64-unknown-linux-gnu/release/librustc_driver.so
300.189.864 Nov 21 18:18 ./stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_driver-18613274f1c25ece.so
300.189.864 Nov 21 18:18 ./stage1/lib/librustc_driver-18613274f1c25ece.so
1.080.426.762 Nov 21 18:29 ./stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_llvm-3c1fd4e318bccfa2.rlib
1.080.426.762 Nov 21 18:29 ./stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_llvm-3c1fd4e318bccfa2.rlib
1.080.426.762 Nov 21 18:29 ./stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_llvm-3c1fd4e318bccfa2.rlib
1.080.427.866 Nov 21 18:15 ./stage0-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_llvm-a82f5dfbab002307.rlib
1.080.427.866 Nov 21 18:15 ./stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_llvm-a82f5dfbab002307.rlib
|
Compare that to the size of the same files compiled with -O2:
| Code: | 198.221.900 Nov 21 20:59 ./stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_llvm-3c1fd4e318bccfa2.rlib
198.221.900 Nov 21 20:59 ./stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_llvm-3c1fd4e318bccfa2.rlib
198.221.900 Nov 21 20:59 ./stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_llvm-3c1fd4e318bccfa2.rlib
198.223.004 Nov 21 20:55 ./stage0-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_llvm-a82f5dfbab002307.rlib
198.223.004 Nov 21 20:55 ./stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_llvm-a82f5dfbab002307.rlib
|
The file sizes differ more than a factor of 5!
Last edited by mike155 on Mon Nov 21, 2022 8:25 pm; edited 2 times in total |
|
| Back to top |
|
|
Ionen
Developer
Joined: 06 Dec 2018
Posts: 2893
|
| Posted: Mon Nov 21, 2022 7:58 pm Post subject: |
|
|
| sdauth wrote: | | Ionen wrote: | | My own last build was tiny, but I cheat with system-llvm+bootstrap |
How to enable that ? I have set this in my package.use : dev-lang/rust system-bootstrap system-llvm rustfmt
but when running equery u on dev-lang/rust, these USE flags are not showing up. (although they do exist in the ebuild, I don't understand) |
They are masked in stable.
rust uses a slightly modified llvm making bundled typically preferred for stability and minor extras, not that there's been real issues with that in a while, also the llvm it needs may sometime be a bit older than system's latest (albeit at worst you can keep two copies of slotted llvm, 1.65.0 is okay with llvm-15 now, 1.64.x wanted llvm-14).
system-bootstrap requires a recent rust, but if your last built rust is too old it'll just pull rust-bin to bootstrap itself (can depclean after). Doesn't change a whole lot beside not downloading and unpacking rust to bootstrap itself every time. Mostly just masked in case your system's rust is broken and it can't build itself (not really been an issue lately).
That aside, you can un-stable mask if you want by adding to /etc/portage/profile/package.use.stable.mask (note the /profile/): | Code: | | dev-lang/rust -system-bootstrap -system-llvm |
Alternatively, having rust in package.accept_keywords would count as no longer being stable.. but mixing stable and testing may cause more problems. |
|
| Back to top |
|
|
Ionen
Developer
Joined: 06 Dec 2018
Posts: 2893
|
| Posted: Mon Nov 21, 2022 8:17 pm Post subject: |
|
|
And oh, hadn't noticed that missing -O2.
It's possible that without optimizations it even assumes you want debugging in places, albeit I could see llvm grow large from lack of optimization alone -- plus add rust on top. |
|
| Back to top |
|
|
mike155
Advocate
Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany
|
| Posted: Mon Nov 21, 2022 10:13 pm Post subject: |
|
|
| Ionen wrote: | | It's possible that without optimizations it even assumes you want debugging in places, albeit I could see llvm grow large from lack of optimization alone -- plus add rust on top. |
I've analyzed the build logs with -O2 and without -O2 to find out what's going on. But I can't find a difference. Strange.
It's not only that without optimization it takes longer and the files are much larger. rustc also seems to be slower.
Would it make sense to have a warning in the ebuild that warns users if they emerge rust without -O2? |
|
| Back to top |
|
|
sdauth
l33t
Joined: 19 Sep 2018
Posts: 679
Location: Ásgarðr
|
| Posted: Tue Nov 22, 2022 5:47 am Post subject: |
|
|
| Ionen wrote: | | They are masked in stable. [...] |
Oh I see, I totally forgot about profile mask  I will add that to /etc/portage/profile/package.use.stable.mask to see how much time I can spare with system-llvm and system-bootstrap enabled then. Thanks !
EDIT :
Before :
| Code: | Fri Nov 18 03:25:59 2022 >>> dev-lang/rust-1.65.0
merge time: 1 hour, 51 minutes and 20 seconds. |
After :
| Code: | Fri Nov 22 11:25:25 2022 >>> dev-lang/rust-1.65.0
merge time: 1 hour, 14 minutes and 10 seconds. |
| Code: | * Final size of build directory: 6235896 KiB ( 5.9 GiB)
* Final size of installed tree: 292724 KiB (285.8 MiB) |
Nice |
|
| Back to top |
|
|
Dwosky
Tux's lil' helper
Joined: 07 Nov 2018
Posts: 141
|
| Posted: Tue Nov 22, 2022 8:15 pm Post subject: |
|
|
| Christian99 wrote: | @Dwosky I would try building again with O2 enabled and see what it changes.
BTW, did you mean to leave out O2 on purpose, or is this just an oversight? |
I updated the CFLAGS to the current processor in order to use distcc, but not sure why I left O2 out, I think I might just copied what app-misc/resolve-march-native outputs and didn't add the pipe or O2 at the end.
I was able to emerge rust increasing the filesystem to 30G, but since the emerge was successfull and I did that over app-misc/screen, I'm not aware of the size it really took to compile it. Is there a way to avoid clearing /var/tmp/portage once the package is built and its successfull?
EDIT: Did another run, shrinking the filesystem back to 20G and adding -O2 -pipe to the CFLAGS and not only compiled without issues, but was a lot more faster that without those flags (second time is the one with O2):
| Code: | # genlop -t dev-lang/rust
* dev-lang/rust
Tue Nov 22 10:01:30 2022 >>> dev-lang/rust-1.65.0
merge time: 20 hours, 54 minutes and 30 seconds.
Wed Nov 23 02:53:10 2022 >>> dev-lang/rust-1.65.0
merge time: 5 hours, 22 minutes and 32 seconds.
|
|
|
| Back to top |
|
|
Christian99
Veteran
Joined: 28 May 2009
Posts: 1721
|
| Posted: Wed Nov 23, 2022 8:16 am Post subject: |
|
|
| man make.conf wrote: |
[...]
FEATURES =
[...]
keeptemp
Do not delete the ${T} directory after the merge process.
[...]
|
|
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2709
|
| Posted: Wed Nov 23, 2022 2:55 pm Post subject: |
|
|
I don't think rust is worth compiling. Bottom line is what code it produces and that doesn't depend on the compiler options it was compiled with. Do not create a problem you don't have. I recently switched to rust-bin, and it already feels like a whole eternity free of troubles.
Regards,
Georgi |
|
| Back to top |
|
|
|
Portage & Programming
