| View previous topic :: View next topic |
| Author |
Message |
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2390
|
 Posted: Fri Jul 19, 2024 7:31 pm Post subject: |
 |
|
| eccerr0r wrote: | Lol I was inspecting
| Code: | | # qlop -mvt virtual/* |
and I notice
| Code: | | 2023-02-25T21:23:15 >>> virtual/editor-0-r4: 19'19" |
I don't think it really took 20 minutes, but it was probably waiting on a lock for 19 minutes... |
Quite possible. Or if this was a laptop, you might have suspended it.
Best Regards,
Georgi |
|
| Back to top |
|
 |
therik
n00b
Joined: 14 Jul 2024
Posts: 7
|
| Posted: Sat Jul 20, 2024 5:18 pm Post subject: |
|
|
| wanne32 wrote: | If you set your load average to 31 and your parallel processes per CPU also to 31 it is more or less clear that it rarely starts a second process. This will hamper your CPU-utilisation but maybe not so much your compile time since more cache hits will compensate for the idle time.
But also compiling it on only 32G tmpfs sounds scarce for bigger things like firefox/kde/chromium. What are you compiling? Can it be that you are satisfying your internet connection since you are compiling a lot of big things without much compiletime?
Also llvm/clang are these packages that usually also stop my CPU running on max. The compilation process doesn't seems very parallel and a lot of other packages depend on it. So you can not go on with compiling other packages. |
I was recompiling everything, it was #emerge --emptytree @world and I was looking at the load numbers and memory usage from free and top, SSD read/write speeds and network statistics in KDE system monitor, I dunno where the KDE data is fundamentally pulled from. df -h was reporting that the tmpfs in /var/tmp/portage was rarely above 2.5 gigs, despite having 32 gigs available, which also seems a bit suspicious. You can see on the outputs I copied in the first post what the typical values were. Compiling LLVM, gcc and few others packages did hit the load limit from makeopts, but portage is also set to 30-ish jobs and it was never even remotely close. Obviously, when make spawns 32 processes for a big package, it's expected that portage won't do anything until that's done, but vast majority of the time, there isn't anything big compiling in the background.
That's kind of the gist of what I'm trying to say, that the compilation can easily max out the hardware limits on big packages, but when emerging 1000 small packages, it's nowhere near that.
Basically, considering that most packages are small, the compilation itself is not even remotely the bottleneck.
If you indulge me, just to really nail this point down, if I use 6.6.30 as a crude benchmark, it looks like this:
| Code: | Gentoo /usr/src/linux # make distclean --silent
Gentoo /usr/src/linux # cp /tmp/.config .config
Gentoo /usr/src/linux # time make --silent -j31
real 2m42.899s
user 50m37.359s
sys 8m27.441s
|
A minute in I see | Code: | top - 17:35:27 up 6:28, 1 user, load average: 29.19, 9.97, 3.93
Tasks: 579 total, 34 running, 545 sleeping, 0 stopped, 0 zombie
%Cpu(s): 81.8 us, 18.0 sy, 0.0 ni, 0.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 64223.8 total, 53052.7 free, 7078.1 used, 5054.8 buff/cache
MiB Swap: 16312.0 total, 16312.0 free, 0.0 used. 57145.8 avail Mem |
in a parallel terminal.
It baloons up in in memory, does its thing and it's done before the kettle boils the tea. But if I need to do 1000 small packages, I'll have to let it run over night, which is frankly a little bit silly.
Just as logrusx's post shows:
| logrusx wrote: | | It took me ~8 hours to emerge ~1600 packages using the binary packages host. |
This is not a compiliation thing. This is portage twiddling its thumbs. |
|
| Back to top |
|
|
wanne32
n00b
Joined: 11 Nov 2023
Posts: 69
|
| Posted: Sat Jul 20, 2024 11:22 pm Post subject: |
|
|
| Quote: | | df -h was reporting that the tmpfs in /var/tmp/portage was rarely above 2.5 gigs, despite having 32 gigs available, which also seems a bit suspicious. |
Indeed. Sounds to me like some things are not compiled in /var/tmp/portage. But be aware that only chrome, firefox and khtml are realy big packages and finished packages are removed. So as long as you are not compiling these low usage is normal.
| Quote: | | but portage is also set to 30-ish jobs and it was never even remotely close. Obviously, when make spawns 32 processes for a big package, it's expected that portage won't do anything until that's done, but vast majority of the time, there isn't anything big compiling in the background. |
This is very similar for me I set load-average to 64 for that reason.
| Quote: | | That's kind of the gist of what I'm trying to say, that the compilation can easily max out the hardware limits on big packages, but when emerging 1000 small packages, it's nowhere near that. |
That is kind of normal. You like to have a transaction based system for installing to minimize the risk of ending in an inconsistent state witch will result in a lot of waiting processes. yum/dnf is even worse in that regard. But usually compiling the bigger packages will outwight that by far. In an optimal scenario you would try to always have an big package compiling while there is one downloading and one installing. urpmi was able to do something like that by piping downloads directly to cpio. Similarly you could do btrfs snapshots and run your stuff on eatmydata and roll back if the update fails. But portage has no such intelligent scheduling and usually starts with downloading a lot of small packages at the same time to later compile bigger packages in parallel. Since the whole emerge time is usually dominated by compile times I doubt that anyone is willing to invest time in optimizing that.
Because like I said: You should be much faster than 8h on an modern day nvme/DDR5/tmpfs computer.
Her on my PC:
| Code: | #time emerge virtual/ssh
Calculating dependencies... done!
Dependency resolution took 0.82 s (backtrack: 0/20).
>>> Emerging binary (1 of 1) virtual/ssh-0-r2::gentoo
>>> Installing (1 of 1) virtual/ssh-0-r2::gentoo
>>> Completed (1 of 1) virtual/ssh-0-r2::gentoo
>>> Jobs: 1 of 1 complete Load avg: 1.23, 1.17, 1.10
* GNU info directory index is up-to-date.
real 0m5.437s
user 0m3.700s
sys 0m1.044s
# time emerge help
Dependency resolution took 0.56 s (backtrack: 0/20).
emerge: there are no ebuilds to satisfy "help".
emerge: searching for similar names...
emerge: Maybe you meant any of these: dev-qt/qthelp, gnome-extra/yelp, media-sound/helm, app-emacs/helm, app-admin/helm?
real 0m1.709s
user 0m1.131s
sys 0m0.142s |
So virtual/ssh without starting emerge takes 3.7s on a modern day mid-range PC virtual/editor takes 3.3s. For my 1029 packages that would amount to a full hour. This is still a lot. But be aware that there also where real things happening. It used in average more than a core. So if you have a few compiled packages the will easily saturate your CPU for that time. |
|
| Back to top |
|
|
therik
n00b
Joined: 14 Jul 2024
Posts: 7
|
| Posted: Tue Jul 23, 2024 1:19 pm Post subject: |
|
|
| wanne32 wrote: | | Since the whole emerge time is usually dominated by compile times I doubt that anyone is willing to invest time in optimizing that. |
If the report in this thread is correct, and emerging 1600 packages from binary took 8 hours, it would seem that emerge time is not dominated by compile times at all.
| wanne32 wrote: | | You should be much faster than 8h on an modern day nvme/DDR5/tmpfs computer |
| Code: | # time emerge --ask=n -1 virtual/ssh
Calculating dependencies... done!
Dependency resolution took 1.29 s (backtrack: 0/20).
>>> Verifying ebuild manifests
>>> Emerging (1 of 1) virtual/ssh-0-r2::gentoo
>>> Installing (1 of 1) virtual/ssh-0-r2::gentoo
>>> Completed (1 of 1) virtual/ssh-0-r2::gentoo
>>> Jobs: 1 of 1 complete Load avg: 0.65, 0.61, 0.40
* GNU info directory index is up-to-date.
* IMPORTANT: 3 config files in '/etc' need updating.
* See the CONFIGURATION FILES and CONFIGURATION FILES UPDATE TOOLS
* sections of the emerge man page to learn how to update config files.
real 0m8.012s
user 0m5.269s
sys 0m2.094s
|
| Code: | # time emerge --ask=n -1 virtual/editor
Calculating dependencies... done!
Dependency resolution took 1.38 s (backtrack: 0/20).
>>> Verifying ebuild manifests
>>> Emerging (1 of 1) virtual/editor-0-r7::gentoo
>>> Installing (1 of 1) virtual/editor-0-r7::gentoo
>>> Completed (1 of 1) virtual/editor-0-r7::gentoo
>>> Jobs: 1 of 1 complete Load avg: 0.79, 0.64, 0.41
* GNU info directory index is up-to-date.
* IMPORTANT: 3 config files in '/etc' need updating.
* See the CONFIGURATION FILES and CONFIGURATION FILES UPDATE TOOLS
* sections of the emerge man page to learn how to update config files.
real 0m8.118s
user 0m5.276s
sys 0m2.220s
|
| Code: | # time emerge help
These are the packages that would be merged, in order:
Calculating dependencies... done!
Dependency resolution took 0.80 s (backtrack: 0/20).
emerge: there are no ebuilds to satisfy "help".
emerge: searching for similar names...
emerge: Maybe you meant any of these: dev-qt/qthelp, gnome-extra/yelp, app-admin/helm, app-emacs/helm, media-sound/helm?
real 0m1.419s
user 0m1.189s
sys 0m0.133s
|
Why are my numbers so different? |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 2390
|
| Posted: Tue Jul 23, 2024 3:28 pm Post subject: |
|
|
| therik wrote: |
If the report in this thread is correct, and emerging 1600 packages from binary took 8 hours, it would seem that emerge time is not dominated by compile times at all. |
Those were binary packages!
Precisely it took 6:25, I checked and it's a few posts above. I think one big package and a few smaller compiled for about an hour or so. I can check that too but I don't think it's important.
EDIT: noticeably longer compile times were for two packages, one 15 and one 20 minutes, I haven't counted the vast majority of packages which took below or around a minute. I don'r temember statistics and I sure don't want to count how many were not binary packages.
p.s. | therik wrote: |
Why are my numbers so different? |
Because you have a different tree of installed packages and dependency resolution takes different time.
p.s.2 I'm out.
Best Regards,
Georgi |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9605
Location: beyond the rim
|
| Posted: Tue Jul 23, 2024 4:52 pm Post subject: |
|
|
| therik wrote: | | Why are my numbers so different? |
Obviously because your system hardware and configuration is different. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
