| View previous topic :: View next topic |
| Author |
Message |
dreamer3
Guru
Joined: 24 Sep 2002
Posts: 553
|
 Posted: Tue Feb 25, 2003 5:32 am Post subject: Download files and compile/emerge simultaneously |
 |
|
Ok, this is hot off the presses. I've heard some of the mods say that downloading files while compiling can really glitch things up but I think they are referring to the "emerge -f kde & ", wait a little while then "emerge kde" and crossing your fingering hoping the downloads stay ahead of the compiles (otherwise you'll have two emerges trying to download the same file, messy).
I for one have had 4 or 5 different emerge processes running all download/compiling different things in the past in seperate windows with NO problems whatsoever.
This BASH script simply makes sure that the compiles don't get ahead of the file downloads.
Known Problems (as of v0.16*)
Each package that you pass is emerged invididually and added to the world file, dependencies are NOT handled as emerge itself handles them (ie, not adding them to world file). This is a side-effect of the looping that I just discovered (ie, it's a bug in qemerge) but haven't though of an easy fix for yet. Meanwhile if you're only emerging dependencies you can pass the --oneshot parameter which qemerge now does understand... Sorry for this, any thoughts on an easy way to fix this would be welcome. Update 3-06: Solution is in my head... just need to turn it into code...
Updates
v0.02 - now will die if the compiling emerge process ends with an error
v0.10 - added status bar to top of screen detailing emerge and download progress
v0.15added name and version to status bar
can now be used as a wrapper script for emerge (ie, figures out if you're wanting emerge for some of it's options and calls it, otherwises does what it does best, emerge and download)
v0.151 - fix bug in sed editing of emerge output
v0.152 - add "info" to options that need to go directly to emerge
v0.16 - understands --oneshot option now
v0.161 - understand all options in man emerge and when to handle compiles/downloads and when to let emerge do the work (ie, when you've passed options that don't really have to do with compiling or downloading and using semerge as a full-fledged wrapper for emerge)
Everyone, let me know if you like it, want more, etc...
semerge v0.161
| Code: | #!/bin/bash
source /sbin/functions.sh
# the folder to store working files
TEMP=~/.semerge
EMERGE=/usr/bin/emerge
set_scroll()
{
#echo "from: $1 to: $2"
echo -en "\e[${1};${2}r\e[?4h";
}
save_cursor() { echo -en "\e7"; }
restore_cursor() { echo -en "\e8"; }
cursor_to() { echo -en "\e[$2;$1H"; }
erase_line() { echo -en "\e[2K"; }
download_files()
{
i=1;
total=`count_ebuilds $@`;
for ebuild in $@; do
update_dl_status "($i of $total) Fetching files for ${ebuild}..."
$EMERGE -f "=${ebuild}" 2>/dev/null >/dev/null
touch $TEMP/${ebuild/\//_}.complete
i=$(($i+1))
done
update_dl_status "Downloads complete..."
}
is_downloaded()
{
#convert the slash into an underscore
local ebuild=${1/\//_};
if [ -e $TEMP/${ebuild}.complete ]; then
return 0;
else
return 1;
fi
}
count_ebuilds()
{
i=0;
for x in $@; do
i=$(($i+1));
done
echo "$i"
}
setup_status_bar()
{
echo; echo;
save_cursor
set_scroll 4 80
cursor_to 1 1; erase_line
cursor_to 1 2; erase_line
cursor_to 1 3; erase_line
echo -en "--------------------------------\e[7m[ semerge v0.16 ]\e[0m--------------------------------"
restore_cursor
}
update_dl_status()
{
save_cursor
cursor_to 1 1
erase_line
einfo $@
restore_cursor
}
update_emerge_status()
{
save_cursor
cursor_to 1 2
erase_line
einfo $@
restore_cursor
}
start_compiles()
{
i=1;
total=`count_ebuilds $@`;
for ebuild in $@; do
# are the files for the ebuild downloaded
is_downloaded ${ebuild}
while [ $? != 0 ]; do
sleep 2s;
is_downloaded ${ebuild}
done
update_emerge_status "($i of $total) Emerging ${ebuild}..."
einfo "$EMERGE $EMERGE_OPTS \"=${ebuild}\""
einfo "($i of $total) Emerging ${ebuild}..."
$EMERGE $EMERGE_OPTS "=${ebuild}"
if [ $? != 0 ]; then
eerror "emerge ${ebuild}"
eerror " - has exited with an error"
echo
eerror "qemerge is exiting as well"
exit 1
fi
i=$(($i+1));
done
}
get_ebuilds()
{
all=`$EMERGE $@ -p | grep "\[ebuild " | sed -e "s/\[ebuild[^]]*] //" | awk '{print $1}'`
echo ${all}
return ${result}
}
clean_up()
{
for ebuild in $@; do
rm -f $TEMP/${ebuild/\//_}.complete
done
save_cursor; set_scroll 1 1000; restore_cursor
}
EMERGE_OPTS=""
is_emerge_options()
{
result=0
while [ ${#} -gt 0 ]; do
a=${1}
shift
case "${a}" in
--oneshot|--nodeps|--buildpkg|-b|--usepkg|-k|-verbose|-v)
# these options should be passed each time emerge is executed
EMERGE_OPTS="$EMERGE_OPTS ${a}"
;;
--deep|--emptytree|-e|--onlydeps|-o)
# these simply possibly add or change which packages
# emerge returns, these are allowed
;;
clean|depclean|help|info|inject|prune|rsync|search|sync|unmerge)
result=1
;;
-*)
result=1
;;
esac
done
return ${result}
}
is_emerge_options $@
if [ $? != 0 ]; then
$EMERGE $@
exit
fi
# make sure working folder it exists
mkdir -p $TEMP
# get the ebuilds we'll be building
ebuilds=`get_ebuilds $@`
if [ -z "${ebuilds}" ]; then
$EMERGE $@
exit
fi
# startup stuff
clean_up ${ebuilds}
setup_status_bar
# start the downloads in the background (one at a time)
download_files ${ebuilds} &
# start compiling
start_compiles ${ebuilds}
# clean up our state tracking files
clean_up ${ebuilds} |
Last edited by dreamer3 on Fri Mar 07, 2003 4:21 am; edited 8 times in total |
|
| Back to top |
|
 |
AlterEgo
Veteran
Joined: 25 Apr 2002
Posts: 1619
|
| Posted: Tue Feb 25, 2003 3:38 pm Post subject: |
|
|
Great, I'm trying it now.
Really stupid question: how do you keep track of both processes (comping and downloading). I use prozilla, so the compiling process happens where I cannot see it.
The only indication I have that both things are running, comes from gkrellm (great tool BTW). |
|
| Back to top |
|
|
vaevictus
n00b
Joined: 24 Feb 2003
Posts: 20
|
| Posted: Tue Feb 25, 2003 10:13 pm Post subject: glad I could be of inspiration. |
|
|
think there's any chance of someone pythonizing this type of setup into the portage standard way of doing things?
n8
_________________
Use FDISK for WIN32 today! |
|
| Back to top |
|
|
Kijutsu
n00b
Joined: 17 Feb 2003
Posts: 16
Location: Romulus, MI
|
| Posted: Tue Feb 25, 2003 10:33 pm Post subject: n33t |
|
|
When I start compiling minor stuff I'll give it a runthru... great idea dude!  |
|
| Back to top |
|
|
dreamer3
Guru
Joined: 24 Sep 2002
Posts: 553
|
| Posted: Wed Feb 26, 2003 11:37 pm Post subject: Re: glad I could be of inspiration. |
|
|
| vaevictus wrote: | think there's any chance of someone pythonizing this type of setup into the portage standard way of doing things?
n8 |
It's been on the list of "wanted" Portage features for some time I think, but until then I think it can be done just as well with this type of approach...
Now, if I could just come up with a cool way to allow the user to see BOTH output streams (downloading and compiling) I seriously think that with better error handling this approach is perfect.
I will think about it... seems like one could nohup the wget and then read the file, pass it through grep and sed and print download "status" lines every once in a while during the buildling... would anyone even like this kind of download status though? |
|
| Back to top |
|
|
vaevictus
n00b
Joined: 24 Feb 2003
Posts: 20
|
| Posted: Thu Feb 27, 2003 3:37 pm Post subject: yeah |
|
|
I'd certainly enjoy any status updates you can provide...
I'd love it if portage would have a curses tagline on the bottom with:
"Emerge package X out of Y total" and a download line would be good too.
also ... if there was a way of generating a "finished compilation time" estimate, that'd be the best. I'm sure it'd be rather difficult to code...
but if you could maybe parse a (make -n), it might be feasible to get a "currently .o'ing #597 out of #1438 objects"
oh well... :)
_________________
Use FDISK for WIN32 today! |
|
| Back to top |
|
|
dreamer3
Guru
Joined: 24 Sep 2002
Posts: 553
|
| Posted: Fri Feb 28, 2003 1:27 am Post subject: Re: yeah |
|
|
| vaevictus wrote: | I'd certainly enjoy any status updates you can provide...
I'd love it if portage would have a curses tagline on the bottom with:
"Emerge package X out of Y total" and a download line would be good too.
also ... if there was a way of generating a "finished compilation time" estimate, that'd be the best. I'm sure it'd be rather difficult to code...
but if you could maybe parse a (make -n), it might be feasible to get a "currently .o'ing #597 out of #1438 objects"
oh well...  |
Could probably do all that with BASH  In one of my CS classes in college we had to write a file viewer (kinda like nano but without edit functionality) with a toolbar and I learned how to do all that kinda of stuff with VT100 escape sequences... not sure how ncurses does it, but I bet what we learned would work with most consoles...
I will look into those sorts of glitzy things if I see people start using the script... |
|
| Back to top |
|
|
dreamer3
Guru
Joined: 24 Sep 2002
Posts: 553
|
| Posted: Sat Mar 01, 2003 1:31 pm Post subject: Re: yeah |
|
|
| vaevictus wrote: | | I'd love it if portage would have a curses tagline on the bottom with: "Emerge package X out of Y total" and a download line would be good too. |
Ask you shall recieve... semerge now has both (at the top cause I like it that way). (uses VT100 terminal escape codes which work fine with eterm at least, should work with most term emus).
Give it a try... i use semerge hdparm hddtemp unzip to test it...
Edit: Ok, it works fine in Eterm, but not from a console... evidentally Eterm doesn't mind my inflating the # of lines in the VT100 code that sets the scroll area, but the console does... it only works when you specify the correct # of lines that are actually VISIBLE on the screen. Anyone know how to get this easily with a command line utility? Of course if you ALWAYS used only the console you can just hack the utility... |
|
| Back to top |
|
|
wyrickre
n00b
Joined: 10 Jul 2002
Posts: 27
Location: Colorado Springs, CO
|
| Posted: Wed Mar 05, 2003 2:07 am Post subject: |
|
|
try
or
_________________
Geek used to be a 4 letter word; Now it's a 6 figure one. |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20476
|
|
| Back to top |
|
|
dreamer3
Guru
Joined: 24 Sep 2002
Posts: 553
|
| Posted: Fri Mar 07, 2003 3:48 am Post subject: |
|
|
Looking into using this. Also, just updated the version to 0.16 to allow it to work with --oneshot. |
|
| Back to top |
|
|
dreamer3
Guru
Joined: 24 Sep 2002
Posts: 553
|
| Posted: Fri Mar 07, 2003 4:22 am Post subject: Re: yeah |
|
|
| vaevictus wrote: | I'd certainly enjoy any status updates you can provide...
I'd love it if portage would have a curses tagline on the bottom with:
"Emerge package X out of Y total" and a download line would be good too. |
Still waiting to see what you think of recent updates. |
|
| Back to top |
|
|
Sastraxi
Apprentice
Joined: 25 Feb 2003
Posts: 258
|
| Posted: Fri Mar 07, 2003 4:45 am Post subject: |
|
|
| Is there any way to pause part of the console? Aka. have the top 4 lines have the wget status (ex. 1 of 8, pbar, etc.), and the rest the compile output. |
|
| Back to top |
|
|
dreamer3
Guru
Joined: 24 Sep 2002
Posts: 553
|
| Posted: Fri Mar 07, 2003 8:57 am Post subject: |
|
|
| Sastraxi wrote: | | Is there any way to pause part of the console? Aka. have the top 4 lines have the wget status (ex. 1 of 8, pbar, etc.), and the rest the compile output. |
Pause???
I think what you're asking for is something like "split screen"... and no, I don't think that is possible... what I'm doing is redefining the "scroll" window of the screen and then moving the cursor to the top, updating status, and moving it back quickly... just thinking about it it seems that eventually it should "screw up" when in the middle of a big emerge or something and put some emerge output on the top, but I haven't seen that happen yet.
Anyone have any ideas for "split screen" mode? I was just wondering about logging both the download and the emerge (running in the BG) and having the BASH script read the output files very often and draw the screen itself... sounds like a lot of work though if it would work at all
Would be cool to see an implimentation though.
Edit: Is it possible to "pause" a process in BASH from script? |
|
| Back to top |
|
|
TheCoop
Veteran
Joined: 15 Jun 2002
Posts: 1814
Location: Where you least expect it
|
| Posted: Fri Mar 07, 2003 9:29 am Post subject: |
|
|
can you do a split screen thing with framebuffers? (like the gentoo logo on bootup)
_________________
95% of all computer errors occur between chair and keyboard (TM)
"One World, One web, One program" - Microsoft Promo ad.
"Ein Volk, Ein Reich, Ein Führer" - Adolf Hitler
Change the world - move a rock |
|
| Back to top |
|
|
dreamer3
Guru
Joined: 24 Sep 2002
Posts: 553
|
| Posted: Fri Mar 07, 2003 9:50 am Post subject: |
|
|
| TheCoop wrote: | | can you do a split screen thing with framebuffers? (like the gentoo logo on bootup) |
Maybe, but that would be outside the scope of what I want to accomplish as I don't want a certain thing to have to be compiled into your kernel for semerge to work properly... |
|
| Back to top |
|
|
Ari Rahikkala
Guru
Joined: 02 Oct 2002
Posts: 370
Location: Finland
|
| Posted: Wed Aug 20, 2003 10:30 am Post subject: |
|
|
| dreamer3 wrote: | | Edit: Is it possible to "pause" a process in BASH from script? |
Well, it's possible to `kill -STOP` one, but I think you mean the output, not the whole process...
_________________
<laurentius> gentoo linux?
<ari> Yesh.
<laurentius> they look horny |
|
| Back to top |
|
|
dreamer3
Guru
Joined: 24 Sep 2002
Posts: 553
|
| Posted: Thu Aug 21, 2003 8:37 pm Post subject: |
|
|
| It's been so long I have no idea what I was asking... |
|
| Back to top |
|
|
Hackeron
Guru
Joined: 01 Nov 2002
Posts: 307
|
| Posted: Sat Feb 28, 2004 2:53 am Post subject: |
|
|
| why is script not being updated? |
|
| Back to top |
|
|
dreamer3
Guru
Joined: 24 Sep 2002
Posts: 553
|
| Posted: Sat Feb 28, 2004 6:32 am Post subject: |
|
|
Becuase I have a dial-up connection and get tired of downloading/compiling source all the time so I switched to Debian. *ducks*
That being said, my intended answer to the problem:
The get_ebuilds step needs to be broken into two... the first command run is exected with --no-deps and this list is stored as packages that should be emerged and ADDED to the world file.
Then the same emerge needs to be called with --deps-only to return a list of which ebuilds should be emerged with --one-time-only (ie, not added to the world file cause they are only depends).
Now that one has BOTH these lists it might be necessary to call emerge again with neither parameter so that a COMPLETE list is created with the correct installation order. Incase one of the dependencies requires one of the non-deps of a different package, etc...
Only problem with this is it might significantly increase the startup time of the script as emerge needs to be run 3 times... If emerge outputed more detailed info (say if package was gonna be added to world or not) you could just parse that, but it doesn't.
If I still ran Gentoo I'd be happy to write this up but I've honestly seen little interest in this script except for the occasoinal passerby, so I'll leave it as an excersize for the reader... Sorry.
Feel free to hack something together and paste here and I"ll come look at it... I love BASH scripting, but I really don't have the time or inclination to do more at this point. |
|
| Back to top |
|
|
Hackeron
Guru
Joined: 01 Nov 2002
Posts: 307
|
| Posted: Sat Feb 28, 2004 11:58 am Post subject: |
|
|
Thanks for giving this info, I'm going to try to make a hack for what you just said. PS, debian is a nice distro and has many advantages to gentoo, no need to duck, there are many simultaneous debian/gentoo users here
EDIT: Think I'm going to turn this into a full size wrapper for portage actually, I have a way in my head for something like the 'ehush' script, but with better output, and shows information about how much lines compiled, with ability to make a bug report and display both compile and download information. |
|
| Back to top |
|
|
meowsqueak
Veteran
Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand
|
| Posted: Sat Jul 03, 2004 11:15 pm Post subject: |
|
|
| What's the current status of this enhancement please? Is anyone still actively working on it? |
|
| Back to top |
|
|
kamagurka
Veteran
Joined: 25 Jan 2004
Posts: 1026
Location: /germany/munich
|
| Posted: Thu Jul 08, 2004 1:55 pm Post subject: Re: yeah |
|
|
| vaevictus wrote: | I'd certainly enjoy any status updates you can provide...
I'd love it if portage would have a curses tagline on the bottom with:
"Emerge package X out of Y total" and a download line would be good too.
also ... if there was a way of generating a "finished compilation time" estimate, that'd be the best. I'm sure it'd be rather difficult to code...
but if you could maybe parse a (make -n), it might be feasible to get a "currently .o'ing #597 out of #1438 objects"
oh well... :) |
yes please. want please.
_________________
If you loved me, you'd all kill yourselves today.
--Spider Jerusalem, the Word |
|
| Back to top |
|
|
|
Unsupported Software
