| View previous topic :: View next topic |
| Author |
Message |
rudregues
Apprentice
Joined: 29 Jan 2013
Posts: 231
Location: Brazil
|
 Posted: Sat Mar 10, 2018 4:50 pm Post subject: [TIP] Calculate your world compile time |
 |
|
I was struggling very hard to find an utility that calculates the total time I'd compiled all my packages, so I developed a command. Here are the explanation:
1. Get a list of all installed packages without the version at the end.
2. Redirect this to show building times in seconds.
3. Print only building times.
4. Sum all lines of building times.
5. Redirect the sum to a file.
6. Evalute the sum from the file.
7. Print the evaluated sum in HH:MM:SS format.
8. Delete the file to keep the system clean 
| Code: | | qlist -I | xargs qlop -t | cut -f2 -d" " | awk '{s+=$1} END {print "secs="s}' > /tmp/btime && eval $(cat /tmp/btime) && printf '%dh:%dm:%ds\n' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60)) && rm /tmp/btime |
Mine is 15h:51m:28s right now.
EDIT: steveL did a big enhancement over my command, eliminating steps 3, 5, 6 and 8:
| Code: | | qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }' |
EDIT2: Mitigating/troubleshooting the outlier problem. People with absurd build times can use two commands to adapt the situation.
1) Calculating total sum excluding outliers defined by the user. I want everything bigger than 5400 seconds (1h30min) to be removed from calculation: | Code: | | qlist -I | xargs qlop -t | awk '{ if ($2 < 5400) secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }' |
2) Discovering the problematic build times (maybe report a bug?). I want to point out which packages build times are bigger than 5400 seconds (1h30min): | Code: | | qlist -I | xargs qlop -t | awk '{ if ($2 > 5400) printf("%s %dd:%dh:%dm:%ds\n", $1, $2 / 86400, ($2 % 86400) / 3600, ($2 % 3600) / 60, $2 % 60); }' |
_________________
Emerging en gentoo
Last edited by rudregues on Tue Apr 10, 2018 5:20 pm; edited 5 times in total |
|
| Back to top |
|
 |
albright
Advocate
Joined: 16 Nov 2003
Posts: 2588
Location: Near Toronto
|
| Posted: Sun Mar 11, 2018 12:56 am Post subject: |
|
|
OK, that is fun
my desktop i7-6700: 16h:36m:10s
my thinkpad x1 carbon (4th gen / i7-6600U): 21h:42m:15s
very similar installations (but distcc probably skews result)
_________________
.... there is nothing - absolutely nothing - half so much worth
doing as simply messing about with Linux ...
(apologies to Kenneth Graeme) |
|
| Back to top |
|
|
fedeliallalinea
Administrator
Joined: 08 Mar 2003
Posts: 31354
Location: here
|
| Posted: Sun Mar 11, 2018 8:22 am Post subject: |
|
|
You can skip to write in a file
| Code: | | qlist -I | xargs qlop -t | cut -f2 -d" " | awk '{s+=$1} END {print "secs="s}' | eval && printf '%dh:%dm:%ds\n' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60)) |
_________________
Questions are guaranteed in life; Answers aren't. |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Sun Mar 11, 2018 11:18 am Post subject: Re: [TIP] Calculate your world compile time |
|
|
Luke.. use the awk, Luke ;) | Code: | | qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }' |
(untested, as not in gentoo atm.) |
|
| Back to top |
|
|
Juippisi
Developer
Joined: 30 Sep 2005
Posts: 757
Location: /home
|
| Posted: Sun Mar 11, 2018 5:30 pm Post subject: |
|
|
Is this one of those cases where "I copied a bash line from the internet and now my / is empty"?
Just kidding,
| Code: |
# qlist -I | xargs qlop -t | cut -f2 -d" " | awk '{s+=$1} END {print "secs="s}' > /tmp/btime && eval $(cat /tmp/btime) && printf '%dh:%dm:%ds\n' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60)) && rm /tmp/btime
11h:50m:56s
|
Sure felt a lot longer when I did -e world a while back. i7-2700k with '-march=native -O3 -fgraphite-identity -ftree-loop-distribution -floop-nest-optimize -flto=8 -fuse-linker-plugin -pipe -fstack-protector-all -mindirect-branch=thunk' |
|
| Back to top |
|
|
Fitzcarraldo
Advocate
Joined: 30 Aug 2008
Posts: 2056
Location: United Kingdom
|
| Posted: Sun Mar 11, 2018 6:59 pm Post subject: |
|
|
It's a rather academic exercise, though, isn't it? It doesn't take into account one's time spent investigating between failed merges due to blockers and bugs in packages and/or ebuilds. In many cases I have had to spend hours trying to fix a problem and get a package to merge, and its build time may be a few minutes. Over time I have spent more time debugging than the actual time it took to build the packages.
| Code: | clevow230ss /home/fitzcarraldo # qlist -I | xargs qlop -t | cut -f2 -d" " | awk '{s+=$1} END {print "secs="s}' > /tmp/btime && eval $(cat /tmp/btime) && printf '%dh:%dm:%ds\n' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60)) && rm /tmp/btime
40h:18m:44s |
_________________
Clevo W230SS: amd64, VIDEO_CARDS="intel modesetting nvidia".
Compal NBLB2: ~amd64, xf86-video-ati. Dual boot Win 7 Pro 64-bit.
OpenRC systemd-utils[udev] elogind KDE on both.
My blog |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 22870
|
| Posted: Sun Mar 11, 2018 7:10 pm Post subject: |
|
|
| Juippisi wrote: | | Is this one of those cases where "I copied a bash line from the internet and now my / is empty"? |
Any time you see an eval in the program, that's a good question to ask. Although sometimes useful, incautious use of eval is extremely dangerous. |
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Sun Mar 11, 2018 7:34 pm Post subject: |
|
|
| Code: | ~ $ qlist -I | xargs qlop -t | cut -f2 -d" " | awk '{s+=$1} END {print "secs="s}' > /tmp/btime && eval $(cat /tmp/btime) && printf '%dh:%dm:%ds\n' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60)) && rm /tmp/btime
5055h:12m:42s |
My output's a bit skewed by this one entry...
| Code: | | openscenegraph: 18068512 seconds average for 1 merges |
Probably related to this bug in qlop |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3777
Location: Rasi, Finland
|
| Posted: Sun Mar 11, 2018 8:26 pm Post subject: Re: [TIP] Calculate your world compile time |
|
|
| steveL wrote: | Luke.. use the awk, Luke ;) | Code: | | qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }' |
(untested, as not in gentoo atm.) |
Works. Returns the exact same time.
My time is 76h:19m:7s... FX-8350 shows its age.
_________________
..: Zucca :..
| My gentoo installs: | init=/sbin/openrc-init
-systemd -logind -elogind seatd |
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3477
|
| Posted: Sun Mar 11, 2018 9:36 pm Post subject: |
|
|
My poor hamster powered pc 
I bought this box nearly a decade (sic!) ago and it was dirt cheap already.... And I don't really feel like replacing it any time soon. That APU A8-3870 works well enough. |
|
| Back to top |
|
|
Goverp
Advocate
Joined: 07 Mar 2007
Posts: 2187
|
| Posted: Mon Mar 12, 2018 10:08 am Post subject: |
|
|
What would be more useful is similar arithmetic in emerge --ask to estimate the time for the command in question. Then this script becomes essentially emerge -ea world.
_________________
Greybeard |
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3477
|
| Posted: Mon Mar 12, 2018 8:31 pm Post subject: |
|
|
Goverp, this script completed in 20 seconds, which is significantly faster than 4 days emerge -aev @world would take.
Also, each of us has different @world, different set of USE flags and different hardware which means we simply _cant_ compare results anyway. There is no way to get any meaningful data from it besides "how badly screwed I am when my hard drive dies". And the answer is "you better verify your backups" |
|
| Back to top |
|
|
Goverp
Advocate
Joined: 07 Mar 2007
Posts: 2187
|
| Posted: Tue Mar 13, 2018 10:40 am Post subject: |
|
|
szatox, I'm not sure what you think I was suggesting, but nothing that would take 4 days. My point was that emerge could do the same arithmetic, and tell you how long the command would take if you say "Yes", thereby at least telling you whether to take a coffee, go out to the cinema, or take a small vacation while portage recompiled whatever.
_________________
Greybeard |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3777
Location: Rasi, Finland
|
| Posted: Tue Mar 13, 2018 11:56 am Post subject: |
|
|
That would be useful... Also the estimations for each package in the listing, since by calcutaling total time the time for each packge is already known and wouldne add much overhead.
This can be done wia processing --pretend output, but wastes time.
_________________
..: Zucca :..
| My gentoo installs: | init=/sbin/openrc-init
-systemd -logind -elogind seatd |
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
tuggbuss
Apprentice
Joined: 20 Mar 2017
Posts: 222
|
| Posted: Fri Mar 30, 2018 9:28 am Post subject: |
|
|
i7 - 6900K
64GB RAM
NVMe
Nvidia blob
Xfce4 |
|
| Back to top |
|
|
rudregues
Apprentice
Joined: 29 Jan 2013
Posts: 231
Location: Brazil
|
| Posted: Fri Apr 06, 2018 9:17 pm Post subject: |
|
|
| tuggbuss wrote: |
i7 - 6900K
64GB RAM
NVMe
Nvidia blob
Xfce4 |
This is humiliation...
_________________
Emerging en gentoo |
|
| Back to top |
|
|
krinn
Watchman
Joined: 02 May 2003
Posts: 7470
|
| Posted: Sat Apr 07, 2018 6:30 am Post subject: |
|
|
| Ant P. wrote: | My output's a bit skewed by this one entry...
| Code: | | openscenegraph: 18068512 seconds average for 1 merges |
|
Can't beat me 
| Code: | qlop -vgH fuse
fuse-2.7.4: Sat Sep 19 00:01:48 2009: 1158050441 days, 5 hours, 1 minute, 10 seconds
fuse-2.7.4: Sat Sep 19 03:21:32 2009: 22 seconds
fuse-2.8.1: Wed Nov 11 00:44:36 2009: 18 seconds
fuse-2.8.1: Wed Jun 2 23:19:18 2010: 14 seconds
fuse-2.8.1: Wed Sep 15 21:20:45 2010: 34 seconds
fuse-2.8.5: Mon Oct 11 05:01:42 2010: 23 seconds
fuse-2.8.6: Wed Sep 28 02:18:14 2011: 18 seconds
fuse-2.8.6: Mon Dec 5 17:28:47 2011: 23 seconds
fuse-2.8.7: Mon Apr 16 02:45:26 2012: 38 seconds
fuse-2.9.2: Sun Dec 2 17:55:34 2012: 58 seconds
fuse-2.9.3: Sun Dec 22 10:38:25 2013: 15 seconds
fuse-2.9.5: Thu Apr 14 12:07:27 2016: 16 seconds
fuse-2.9.7: Sun Jan 29 21:19:25 2017: 25 seconds
fuse-2.9.7-r1: Sun Mar 25 03:44:15 2018: 9 seconds
fuse-2.9.7-r1: Sun Mar 25 08:28:48 2018: 14 seconds
fuse: 15 times |
|
|
| Back to top |
|
|
Irre
Guru
Joined: 09 Nov 2013
Posts: 434
Location: Stockholm
|
| Posted: Sat Apr 07, 2018 8:17 pm Post subject: |
|
|
| Code: | | qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }' |
502h:0m:44s 
Linux localhost 4.12.10-gentoo #2 SMP Thu Aug 31 09:20:09 GMT 2017 x86_64 Intel(R) Core(TM) i7-2670QM CPU @ 2.20GHz GenuineIntel GNU/Linux
Something is wrong!
[Moderator edit: added [code] tags to preserve output layout. -Hu] |
|
| Back to top |
|
|
josephg
l33t
Joined: 10 Jan 2016
Posts: 783
Location: usually offline
|
| Posted: Sat Apr 07, 2018 9:41 pm Post subject: |
|
|
think i just topped everyone  and i run a tight ship too!
| Code: | $ qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
777h:50m:29s
$ uname -a
Linux localhost 4.9.76-gentoo-r1-josephg #7 SMP Sun Feb 18 20:38:56 GMT 2018 i686 Intel(R) Core(TM)2 Duo CPU T5870 @ 2.00GHz GenuineIntel GNU/Linux |
and this is my fastest machine  sigh i don't keep up with the rest of the world
atleast it's faster runtime than my neighbours' shiny new windoze laptops
_________________
"Growth for the sake of growth is the ideology of the cancer cell." Edward Abbey |
|
| Back to top |
|
|
krinn
Watchman
Joined: 02 May 2003
Posts: 7470
|
| Posted: Sat Apr 07, 2018 10:06 pm Post subject: |
|
|
i didn't first output my result because it is just bork, but here it is to make everyone a few more happy with their "tiny" 500-700hours to build 
| Code: | qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
341606371735829h:20m:0s
Linux beleg 4.14.11 #2 SMP PREEMPT Wed Jan 10 15:11:16 CET 2018 x86_64 Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz GenuineIntel GNU/Linux
|
|
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Sat Apr 07, 2018 10:47 pm Post subject: |
|
|
| krinn wrote: | | Code: | qlop -vgH fuse
fuse-2.7.4: Sat Sep 19 00:01:48 2009: 1158050441 days, 5 hours, 1 minute, 10 seconds |
|
krinn ... hehe, just over 3 million years, that makes you gentoo's oldest user, hows that beard of yours? :)
| Code: | % printf "%'.f\n" $((1158050441 / 364))
3,181,457 |
... well, if you don't calculate leap years.
best ... khay |
|
| Back to top |
|
|
Tony0945
Watchman
Joined: 25 Jul 2006
Posts: 5127
Location: Illinois, USA
|
| Posted: Sat Apr 07, 2018 11:03 pm Post subject: |
|
|
Athlon X4 (bulldozer) Not bad for a $50 cpu | Code: | qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
2h:17m:38s
|
EDIT:Can't believe Kaveri is this fast. | Code: | # qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
0h:0m:0s
|
|
|
| Back to top |
|
|
rudregues
Apprentice
Joined: 29 Jan 2013
Posts: 231
Location: Brazil
|
| Posted: Mon Apr 09, 2018 9:59 pm Post subject: |
|
|
I edited my first post in order to help people with huge build times.
Don't know though what to do with Tony0945's Kaveri...
_________________
Emerging en gentoo |
|
| Back to top |
|
|
<3
Veteran
Joined: 21 Oct 2004
Posts: 1085
|
| Posted: Tue Apr 10, 2018 12:04 am Post subject: |
|
|
| can we make an ebuild of this to put in portage or add this to gentoolkit? |
|
| Back to top |
|
|
ali3nx
l33t
Joined: 21 Sep 2003
Posts: 732
Location: Winnipeg, Canada
|
| Posted: Tue Apr 10, 2018 2:29 am Post subject: |
|
|
dual cpu 10 core hyperthreaded xeon
Total: 1153 packages (1153 reinstalls), Size of downloads: 0 KiB
| Code: | fenrir ~ # qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
9h:46m:11s
fenrir ~ # uname -a
Linux fenrir 4.16.0-gentoo #3 SMP Tue Apr 3 03:50:21 CDT 2018 x86_64 Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz GenuineIntel GNU/Linux
|
_________________
Compiling Gentoo since version 1.4
Thousands of Gentoo Installs Completed
Emerged on every continent but Antarctica
Compile long and Prosper! |
|
| Back to top |
|
|
|
Documentation, Tips & Tricks
