| View previous topic :: View next topic |
| Author |
Message |
lordoftheworld
n00b
Joined: 16 Jun 2004
Posts: 19
|
 Posted: Tue Apr 10, 2018 3:47 am Post subject: |
 |
|
Great info, I added a sort to see which packages might be taking up more time:
qlist -I | xargs qlop -t |sort -t" " -rnk2 |
|
| Back to top |
|
 |
krinn
Watchman
Joined: 02 May 2003
Posts: 7470
|
| Posted: Tue Apr 10, 2018 12:35 pm Post subject: |
|
|
| Code: | qlist -I | xargs qlop -t | awk '{ if ($2 > 5400) printf("%s %dd:%dh:%dm:%ds\n", $1, $2 / 86400, $2 / 3600, ($2 % 3600) / 60, $2 % 60); }'
gzip: 1d:26h:52m:15s
perl: 0d:7h:48m:44s
libxslt: 0d:1h:41m:7s
gimp: 0d:6h:0m:47s
libpng: 0d:8h:18m:2s
wireshark: 0d:5h:19m:57s
diffutils: 4d:114h:42m:38s
findutils: 3d:81h:57m:18s
gawk: 0d:21h:10m:11s
grep: 0d:19h:58m:8s
kbd: 4d:110h:17m:48s
fuse: 14233598822306d:341606371735361h:55m:12s
e2fsprogs-libs: 0d:18h:23m:13s
perl-version: 0d:1h:50m:40s
lynx: 1d:25h:54m:59s
|
yep something is weird, thanks for the update |
|
| Back to top |
|
|
rudregues
Apprentice
Joined: 29 Jan 2013
Posts: 231
Location: Brazil
|
| Posted: Tue Apr 10, 2018 5:36 pm Post subject: |
|
|
| krinn wrote: | | Code: | qlist -I | xargs qlop -t | awk '{ if ($2 > 5400) printf("%s %dd:%dh:%dm:%ds\n", $1, $2 / 86400, $2 / 3600, ($2 % 3600) / 60, $2 % 60); }'
gzip: 1d:26h:52m:15s
perl: 0d:7h:48m:44s
libxslt: 0d:1h:41m:7s
gimp: 0d:6h:0m:47s
libpng: 0d:8h:18m:2s
wireshark: 0d:5h:19m:57s
diffutils: 4d:114h:42m:38s
findutils: 3d:81h:57m:18s
gawk: 0d:21h:10m:11s
grep: 0d:19h:58m:8s
kbd: 4d:110h:17m:48s
fuse: 14233598822306d:341606371735361h:55m:12s
e2fsprogs-libs: 0d:18h:23m:13s
perl-version: 0d:1h:50m:40s
lynx: 1d:25h:54m:59s
|
yep something is weird, thanks for the update |
Thanks for pointing out my mistake krinn, there was really an error. Since the format of the output from this command is d:h:m:s hours should be always less than 24 (remainder from 86400 seconds division), like minutes (remainder from 3600 seconds division) and seconds (remainder from 60 seconds division) are less than 60. $2 / 3600 should be ($2 % 86400) / 3600
I updated again the first post.
_________________
Emerging en gentoo |
|
| Back to top |
|
|
rudregues
Apprentice
Joined: 29 Jan 2013
Posts: 231
Location: Brazil
|
| Posted: Tue Apr 10, 2018 6:58 pm Post subject: |
|
|
| <3 wrote: | | can we make an ebuild of this to put in portage or add this to gentoolkit? |
I think it would be suitable for qlop from q applets.
I can open a bug report asking for this feature if people want to.
_________________
Emerging en gentoo |
|
| Back to top |
|
|
josephg
l33t
Joined: 10 Jan 2016
Posts: 783
Location: usually offline
|
| Posted: Fri Apr 13, 2018 10:18 pm Post subject: Re: [TIP] Calculate your world compile time |
|
|
| rudregues wrote: | EDIT2: Mitigating/troubleshooting the outlier problem. People with absurd build times can use two commands to adapt the situation.
1) Calculating total sum excluding outliers defined by the user. I want everything bigger than 5400 seconds (1h30min) to be removed from calculation |
Why would you want to remove bigger packages? Those are the ones which matter for this purpose.
Something still doesn't look right 
| Code: | $ qlist -I | xargs qlop -t | awk '{ if ($2 > 5400) printf("%s %dd:%dh:%dm:%ds\n", $1, $2 / 86400, ($2 % 86400) / 3600, ($2 % 3600) / 60, $2 % 60); }'
sudo: 1d:18h:35m:44s
libreoffice: 0d:12h:38m:34s
libnl: 0d:1h:56m:27s
libpcre: 0d:1h:46m:3s
qtdbus: 0d:1h:51m:35s
qtprintsupport: 0d:2h:42m:36s
qtwebengine: 0d:19h:4m:1s
qtwebkit: 6d:18h:9m:2s
qtxmlpatterns: 0d:2h:0m:40s
terminus-font: 1d:13h:15m:10s
opus: 0d:2h:22m:54s
ffmpeg: 0d:10h:3m:37s
libpcap: 6d:2h:7m:37s
busybox: 0d:2h:4m:8s
man-db: 0d:8h:59m:9s
man-pages: 0d:1h:47m:35s
gcc: 0d:3h:46m:6s
llvm: 0d:2h:16m:34s
gentoo-sources: 0d:16h:49m:9s
kergen: 4d:3h:17m:40s
e2fsprogs-libs: 0d:2h:11m:54s
package-manager: 1d:22h:54m:13s
service-manager: 1d:22h:31m:27s
ssh: 1d:22h:50m:52s
firefox: 0d:8h:38m:9s
adwaita-icon-theme: 0d:3h:51m:9s |
_________________
"Growth for the sake of growth is the ideology of the cancer cell." Edward Abbey |
|
| Back to top |
|
|
<3
Veteran
Joined: 21 Oct 2004
Posts: 1084
|
| Posted: Sat Apr 21, 2018 8:05 am Post subject: |
|
|
| rudregues wrote: | | <3 wrote: | | can we make an ebuild of this to put in portage or add this to gentoolkit? |
I think it would be suitable for qlop from q applets.
I can open a bug report asking for this feature if people want to. |
that would be great |
|
| Back to top |
|
|
edgar_uriel84
Guru
Joined: 21 Dec 2008
Posts: 485
Location: Tultitlan, México
|
| Posted: Tue May 01, 2018 7:40 pm Post subject: |
|
|
It's funny
| Code: | ┌─ [4966][14:24:27][genomorro@genomorro-xps-15-9550][~]
└ᐅ qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
7h:31m:19s
┌─ [4966][14:29:10][genomorro@genomorro-xps-15-9550][~]
└ᐅ qlist -I | xargs qlop -t | awk '{ if ($2 > 1800) printf("%s %dd:%dh:%dm:%ds\n", $1, $2 / 86400, ($2 % 86400) / 3600, ($2 % 3600) / 60, $2 % 60); }'
libreoffice: 0d:1h:4m:54s
┌─ [4966][14:29:19][genomorro@genomorro-xps-15-9550][~]
└ᐅ uname -a
Linux genomorro-xps-15-9550 4.9.76-gentoo-r1 #1 SMP PREEMPT Mon Apr 2 01:56:45 CDT 2018 x86_64 Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz GenuineIntel GNU/Linux
|
I'm using DWM as WM.
_________________
Algún día habrá una persona mejor que yo, pero esa persona no eres tú ni ese momento es ahora. |
|
| Back to top |
|
|
Nreal
Apprentice
Joined: 06 Jan 2009
Posts: 289
|
| Posted: Sat Jun 16, 2018 12:43 pm Post subject: |
|
|
Two of those bastards Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz is together 72 threads, (36 actual cores)
| Code: | qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
1024819115206107h:1m:20s |
Waiting for more ram as 16gt swaps with anything above -j30.. Its fast though 
| Code: | boxi /home/markus # uname -a
Linux boxi 4.17.1-gentoo #6 SMP PREEMPT Sat Jun 16 12:55:42 EEST 2018 x86_64 Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz GenuineIntel GNU/Linux
processor : 71
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz
stepping : 2
microcode : 0x3c
cpu MHz : 2793.850
cache size : 46080 KB
physical id : 1
siblings : 36
core id : 27
cpu cores : 18
apicid : 119
initial apicid : 119
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips : 4590.71
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management: |
[Moderator edit: added [code] tags to preserve output layout. -Hu] |
|
| Back to top |
|
|
rudregues
Apprentice
Joined: 29 Jan 2013
Posts: 231
Location: Brazil
|
| Posted: Sat Jun 23, 2018 1:57 pm Post subject: |
|
|
I posted a bug report asking for add this functionality into qlop (app-portage/portage-utils) https://bugs.gentoo.org/658824
_________________
Emerging en gentoo |
|
| Back to top |
|
|
phelix
n00b
Joined: 28 Oct 2018
Posts: 2
|
| Posted: Sun Oct 28, 2018 2:16 am Post subject: |
|
|
Hi there,
have you tried genlop?
| Code: |
$ emerge --pretend --emptytree @world | genlop --pretend
[...]
Estimated update time: 5 hours, 39 minutes.
|
cheers
-- edited for formatting |
|
| Back to top |
|
|
rudregues
Apprentice
Joined: 29 Jan 2013
Posts: 231
Location: Brazil
|
| Posted: Fri Nov 16, 2018 1:21 pm Post subject: |
|
|
| phelix wrote: | Hi there,
have you tried genlop?
| Code: |
$ emerge --pretend --emptytree @world | genlop --pretend
[...]
Estimated update time: 5 hours, 39 minutes.
|
cheers
-- edited for formatting |
Your tip worked, but it has some caveats:
1) It took a big while to complete (maybe 1 minute?);
2) The output is very verbose because of emerge (using --quiet hasn't effect to minimize this);
3) Emerge will look into new packages to satisfy dependencies, not only the already installed, maybe increasing the time in the answer.
Other than these problems, the final answer was Estimated update time: 1 day, 2 hours, 4 minutes., which is pretty close to my script answer of 25h:1m:32s
_________________
Emerging en gentoo |
|
| Back to top |
|
|
ppurka
Advocate
Joined: 26 Dec 2004
Posts: 3256
|
| Posted: Wed Jan 02, 2019 2:36 pm Post subject: |
|
|
qlop seems unreliable for some cases. My script (see my signature) provides an estimated time | Code: | quietemerge -pe1 @world
...
Total ETA: 1d 16h 24m 10s |
The command in the main post provides an estimated time of over 10 days | Code: | ~» qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
290h:22m:38s |
It seems that some of the estimates made by qlop are incorrect: | Code: | » qlop -v -g -H qtcurve
qtcurve-1.8.17-r1: Sat Mar 22 11:19:38 2014: 1 minute, 19 seconds
qtcurve-1.8.17-r1: Thu Apr 24 04:27:08 2014: 39 seconds
qtcurve-1.8.17-r1: Sun Oct 19 22:44:02 2014: 51 seconds
qtcurve-1.8.18-r1: Sat Feb 20 19:07:04 2016: 2 minutes, 16 seconds
qtcurve-1.8.18-r1: Sun May 8 14:10:10 2016: 2 minutes, 8 seconds
qtcurve-1.8.18-r1: Sun May 7 22:41:07 2017: 104 days, 13 hours, 34 minutes, 53 seconds
qtcurve-1.9.0_rc1-r1: Sun Oct 1 16:37:44 2017: 2 minutes, 20 seconds
qtcurve-1.9.0_rc1-r1: Sat Nov 11 17:14:05 2017: 1 minute, 32 seconds
qtcurve-1.9.0_rc1-r1: Mon Jul 2 19:38:20 2018: 1 minute, 7 seconds
qtcurve-1.9.0_rc1-r2: Sun Jul 15 19:13:07 2018: 1 minute, 50 seconds
qtcurve-1.9.0: Sun Oct 21 20:23:40 2018: 2 minutes, 20 seconds
qtcurve: 11 times |
On the other hand, the estimates provided by the internal computations in the quietemerge script gives the following (in reverse order to above): | Code: | ~ » mygenlop qtcurve
* Printing the times in reverse order (recent emerge appears first)
Run no. Time Taken
1 2m 20s
2 1m 50s
3 1m 7s
4 1m 32s
5 2m 20s
6 1m 2s
7 2m 8s
8 2m 16s
9 51s
10 39s
11 1m 19s
* Average time taken over 11 runs: 1m 34s |
_________________
emerge --quiet redefined | E17 vids: I, II | Now using kde5 | e is unstable :-/ |
|
| Back to top |
|
|
pcmaster
Advocate
Joined: 04 Dec 2003
Posts: 2234
Location: Barcelona
|
| Posted: Wed Jan 30, 2019 7:06 pm Post subject: |
|
|
In my box:
| Code: | Core2Duo ~ # qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
42h:19m:8s
Core2Duo ~ # |
Intel Core2 Duo 6550
4 GB RAM
2 TB WD black HDD
_________________
pcmaster |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3747
Location: Rasi, Finland
|
| Posted: Wed Jan 30, 2019 11:07 pm Post subject: |
|
|
I don't know if some else has already mentioned this, but the estimation the command gives is almost always larger than the actual world compile time, unless one disallows parallel merging of packages (emerge --jobs <n>).
_________________
..: Zucca :..
| My gentoo installs: | init=/sbin/openrc-init
-systemd -logind -elogind seatd |
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
proxy
Apprentice
Joined: 20 Apr 2002
Posts: 260
Location: Chantilly, VA
|
| Posted: Thu Feb 14, 2019 3:10 pm Post subject: |
|
|
| Zucca wrote: | | I don't know if some else has already mentioned this, but the estimation the command gives is almost always larger than the actual world compile time, unless one disallows parallel merging of packages (emerge --jobs <n>). |
I was thinking the same thing. Though, you may be able to get a sensible estimate by dividing the result by the number of cores you have (assuming they are all the same spec). Won't be precise, but would probably be close enough to be informative as a real world estimate. |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3747
Location: Rasi, Finland
|
| Posted: Fri Feb 15, 2019 1:36 pm Post subject: |
|
|
Any bigger package that can run make -j<N> effectively at compile phase would normally limit emerge --jobs mto just one.
For example I have --jobs 8 --load-average 10.00 on my EMERGE_DEFAULT_OPTS.
So maximum of 8 concurrent package installs/compiles, but only is load average is under 10.00.
So sometimes for example Firefox and libreoffice both get to be compiled at the same time, effectively lenghtening the compile time of both on my 8-core CPU (would result around load average of 16). In this case compile times recorded for these packages are 2x at worst.
On the other hand some small packages that really can't fully utilize make -j8 (on my system), are almost always being run at 1x speed, but those can be "emerged" parallel (at most of 8 packages being compiled/installed at the same time).
In this case it's the opposite to the previous one. Since on average all those smaller packages are "emerged" in "time * 1/8". But the method counting the world compile time gives each 1x time instead of 1/8.
So you can't just divide the time by the cores on your system... unfortunately.
_________________
..: Zucca :..
| My gentoo installs: | init=/sbin/openrc-init
-systemd -logind -elogind seatd |
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
grobian
Developer
Joined: 31 May 2006
Posts: 67
|
| Posted: Thu Jun 13, 2019 6:52 pm Post subject: |
|
|
For what is worth, with portage-utils-0.80_pre*:
| Code: |
% qlop --summary
...
sync: 1′56″ average for 1 sync
total: 9:38:35 for 352 merges, 1 sync
|
_________________
Gentoo on a different level |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3747
Location: Rasi, Finland
|
| Posted: Thu Jun 13, 2019 6:57 pm Post subject: |
|
|
I'm tempted to ACCEPT some KEYWORDS now.
Thanks for the work, btw.
_________________
..: Zucca :..
| My gentoo installs: | init=/sbin/openrc-init
-systemd -logind -elogind seatd |
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
brokendreams
n00b
Joined: 11 Nov 2018
Posts: 22
Location: clinton,ia US
|
| Posted: Sun Jul 14, 2019 5:17 pm Post subject: |
|
|
| rudregues wrote: | | phelix wrote: | Hi there,
have you tried genlop?
| Code: |
$ emerge --pretend --emptytree @world | genlop --pretend
[...]
Estimated update time: 5 hours, 39 minutes.
|
cheers
-- edited for formatting |
Your tip worked, but it has some caveats:
1) It took a big while to complete (maybe 1 minute?);
2) The output is very verbose because of emerge (using --quiet hasn't effect to minimize this);
3) Emerge will look into new packages to satisfy dependencies, not only the already installed, maybe increasing the time in the answer.
Other than these problems, the final answer was Estimated update time: 1 day, 2 hours, 4 minutes., which is pretty close to my script answer of 25h:1m:32s |
The other issue when genlop is you have to still have all the logs for the last build you did of every single package that's installed or the time is off. It must read the log files to get the time. Because it will warn you if there are log files missing |
|
| Back to top |
|
|
brokendreams
n00b
Joined: 11 Nov 2018
Posts: 22
Location: clinton,ia US
|
| Posted: Sun Jul 14, 2019 5:26 pm Post subject: |
|
|
| Zucca wrote: | I'm tempted to ACCEPT some KEYWORDS now.
Thanks for the work, btw. |
I run several systems that are fully ~amd64 keyworded. Have never had what I'd call serious problems. Sometimes something will fail or an issue will arise so I have to mask something. But it doesnt happen too often. My package.mask is maybe 15 lines. Of those half of them were by choice. And I could probably get rid of half of the rest as they are older packages. |
|
| Back to top |
|
|
Tony0945
Watchman
Joined: 25 Jul 2006
Posts: 5127
Location: Illinois, USA
|
| Posted: Sun Jul 14, 2019 6:38 pm Post subject: |
|
|
| Code: | tony@MSI ~ $ emerge --pretend --emptytree @world | genlop --pretend
!!! Error: no merge found for '-p' |
| Code: | ony@MSI ~ $ emerge --pretend --emptytree @world
These are the packages that would be merged, in order:
Calculating dependencies -... |
Something is wrong with genlop --pretend |
|
| Back to top |
|
|
brokendreams
n00b
Joined: 11 Nov 2018
Posts: 22
Location: clinton,ia US
|
| Posted: Sun Jul 14, 2019 6:57 pm Post subject: |
|
|
| Tony0945 wrote: | | Code: | tony@MSI ~ $ emerge --pretend --emptytree @world | genlop --pretend
!!! Error: no merge found for '-p' |
| Code: | ony@MSI ~ $ emerge --pretend --emptytree @world
These are the packages that would be merged, in order:
Calculating dependencies -... |
Something is wrong with genlop --pretend |
Do you have color forced on? Like in emerge_default_opts or something? I know having --color=y will totally confuse genlop -p |
|
| Back to top |
|
|
Anon-E-moose
Watchman
Joined: 23 May 2008
Posts: 6169
Location: Dallas area
|
| Posted: Sun Jul 14, 2019 7:08 pm Post subject: |
|
|
| Tony0945 wrote: | | Code: | tony@MSI ~ $ emerge --pretend --emptytree @world | genlop --pretend
!!! Error: no merge found for '-p' |
|
Worked for me.
| Code: | emerge --pretend --emptytree @world | genlop --pretend
These are the pretended packages: (this may take a while; wait...)
...
Estimated update time: 7 hours, 58 minutes. |
It did indeed take a while. 
_________________
UM780, 6.1 zen kernel, gcc 13, profile 17.0 (custom bare multilib), openrc, wayland |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3747
Location: Rasi, Finland
|
| Posted: Sun Jul 14, 2019 8:05 pm Post subject: |
|
|
| brokendreams wrote: | | I run several systems that are fully ~amd64 keyworded. Have never had what I'd call serious problems. Sometimes something will fail or an issue will arise so I have to mask something. But it doesnt happen too often. My package.mask is maybe 15 lines. Of those half of them were by choice. And I could probably get rid of half of the rest as they are older packages. |
I've been on the ~ -side. The problem isn't so much the "unstabliness" of the packages, but rather the update frequency. And of course it's nice to have the feeling of stable system on a home server.
_________________
..: Zucca :..
| My gentoo installs: | init=/sbin/openrc-init
-systemd -logind -elogind seatd |
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
brokendreams
n00b
Joined: 11 Nov 2018
Posts: 22
Location: clinton,ia US
|
| Posted: Sun Jul 14, 2019 8:23 pm Post subject: |
|
|
| Zucca wrote: | | brokendreams wrote: | | I run several systems that are fully ~amd64 keyworded. Have never had what I'd call serious problems. Sometimes something will fail or an issue will arise so I have to mask something. But it doesnt happen too often. My package.mask is maybe 15 lines. Of those half of them were by choice. And I could probably get rid of half of the rest as they are older packages. |
I've been on the ~ -side. The problem isn't so much the "unstabliness" of the packages, but rather the update frequency. And of course it's nice to have the feeling of stable system on a home server. |
I can certainly agree with that. I do run stable on my media server simply because that box is lower specs and would take forever to update if it was on ~arch. Still runs an old amd phenom 955.
Although unstable is currently a little "hairy" too. The switch fro. Cblas blas and lapack reference packages to a single lapack package that includes all three blas cblas and lapack reference libraries has tons of stuff failing with undefined references to various cblas_* (cant remember the exact error) functions or whatever. That revdep-rebuild wont catch. Something just wasnt rebuilt right I just don't know what yet. I'll prob just end up emerge -e @world to fix it if I cant figure out exactly what package it is. So it does get crazy sometimes too but I expected this since moving to that single package and allowing it to be selected and picked up by ld.so was a very radical change. |
|
| Back to top |
|
|
|
Documentation, Tips & Tricks
