an interesting portage improvment

Gandalf_Grey_ · Apprentice Joined: 19 Apr 2003 Posts: 151

As I was emergeing kde a thought occured to me. A significant amount of time spent emerging packages is spent downloading, moreso if you are on dialup. I was thinking it might be an interesting idea to have portage do two separate threads so to speak. With the subsequent packages downloading while the first is compiling. OK perhaps I am being quite vague. I will give an example with KDE.

For my use flags on a new install there is ~120 packages. The first package is gnuconfig. Lets say I type emerge kde, portage then grabs gnuconfig and begins compiling, however it waits until gnuconfig is finished compiling before grabing the next package, libtool. Why not have libtool and all the packages after it be downloaded while gnuconfig is being compiled. Since it is usually faster to download something than it is to compile it, this would probably save over an hour on a stage 1 install.

This would also be helpful for dialup users whos bottleneck is the download of larger packages.

This feature may already have been suggested (when I searched the bug database and gentoo forums for stuff relating to backgrounding downloads nothing came up), and I don't mean to sound like I am an ungrateful SOB whining about other peoples hard work. I think the gentoo developers have done a great job, I only mak ethis suggestion to help them make gentoo even better.

What does everyone think?
_________________
Really, I'm not out to destroy Microsoft. That will just be a completely unintentional side effect.
-- Linus Torvalds

Gandalf_Grey_ · Apprentice Joined: 19 Apr 2003 Posts: 151

Oh, I forgot to mention my workaround for this, about 5 minutes before the emerge get an emerge -f started, this will give it a neccesary head start so you don't waste bandwidth downloading two copies of the same file (plus portage would probably get confused), then in another console do the regular emerge. In a bigger package like KDE there is no way in hell the compiling will ever catch the downloading.

This only works if you are on broadband though :/
_________________
Really, I'm not out to destroy Microsoft. That will just be a completely unintentional side effect.
-- Linus Torvalds

tomcf · n00b Joined: 11 Sep 2003 Posts: 10

hmm.. Good idea, but I belive its been mentioned

Not completely sure though..

However, I would like to see emerge setup a bit like apt-get. It would grab more than one package at a time for people with the bandwidth. I could get Debian installed (over FTP) with a basic X, KDE, and GCC, in >500MB and in less than 2 hours. Having to get the packages singally would have slowed down things somewhat... basically how it worked, it would get the package you wanted, and its dependencies, then install.

Genone · Posted: Thu Sep 11, 2003 4:14 am Post subject:

Mayn people would like that feature, unfortunately it's a bit complicated, not impossible but a bit tricky:
- portage isn't threaded by default
- we need a good locking mechanism if the compile of package A is finished before the download of package B
- without some modification to the print code the messages from compiling and fetching are completely mixed

In short: all the problems you get when you introduce threads to a non-threaded program.
Also I think we should fix a lot of the existing bugs before we include this feature, so it probably won't happen before portage 2.1. That said, there is a patch for that in bugzilla for some time, but I don't know if it still works with current portage versions.

If you just want to disconnect as fast as possible use

plate · Posted: Thu Sep 11, 2003 12:43 pm Post subject:

I'm just wondering what you've searched for. This has been asked so many times I lost count, and an ample choice of answers are here, both in the forums and Bugzilla.

Want to download packages while other packages build?
Parallelising the download > compile > download > compile cycle
Download files and compile/emerge simultaneously