| View previous topic :: View next topic |
| Author |
Message |
Kethinov
Apprentice
Joined: 20 Aug 2003
Posts: 166
Location: ESU, Student Teaching 5th Grade
|
 Posted: Thu Jan 08, 2004 2:35 am Post subject: Simultaneous portage - a possible way to speed it up? |
 |
|
I've done a few gentoo installations now and the one thing I always wish portage had is what I'm call simultaneous portage. Example, I run the command emerge kde gnome and I have 172 packages to download and compile. But a great deal of that time is wasted because you're either downloading or you're compiling. Never both. Imagine if portage had a feature to do both simultaneously?
As soon as package1 finished downloading, it would begin to compile. But during its compilation, package2 would begin downloading, instead of waiting for the compilation to finish. Imagine how much more time-efficient portage would be, especially for modem users!
It probably wouldnt be a hard feature to add. emerge -simul kde gnome  |
|
| Back to top |
|
 |
CheshireCat
Guru
Joined: 25 Aug 2002
Posts: 572
|
| Posted: Thu Jan 08, 2004 2:40 am Post subject: |
|
|
| You could start emerge -f <whatever> and then start emerge <whatever when the first package or two are done downloading. This could get screwed up badly if the build process catches up to the fetch process, though, and they try to fetch the same file. |
|
| Back to top |
|
|
adaptr
Watchman
Joined: 06 Oct 2002
Posts: 6730
Location: Rotterdam, Netherlands
|
| Posted: Thu Jan 08, 2004 2:47 am Post subject: Re: Simultaneous portage - a possible way to speed it up? |
|
|
| Kethinov wrote: | | Imagine how much more time-efficient portage would be, especially for modem users! |
Only for modem users, actually.
I have Gentoo on a P-3 733 MHz box with a 100K pipe, so downloading all of KDE will cost me about... ooo say 20 minutes.
Compiling KDE, however, will take upwards of 6 hours - making parallel downloading completely useless to me.
| Kethinov wrote: | | It probably wouldnt be a hard feature to add. |
EDIT:
Hmm...yes, if an emerge first started the download thread, and just kept that going until all was downloaded, then start compiling as soon as the first package is in - that should keep it ahead of the queue.
And even if not - it would be easier to build the cross-checking of the d/l and build queues into portage than to run emerge twice and hope they don't clobber eachother... |
|
| Back to top |
|
|
gsfgf
Veteran
Joined: 08 May 2002
Posts: 1266
|
| Posted: Thu Jan 08, 2004 3:21 am Post subject: |
|
|
I think thhiis is in the works. however, since it's only a big deal for dialup users and very few gentooers use dialup, it keeps getting pushed to the back burner.
_________________
Aim:gsfgf0 |
|
| Back to top |
|
|
CheshireCat
Guru
Joined: 25 Aug 2002
Posts: 572
|
| Posted: Thu Jan 08, 2004 3:33 am Post subject: Re: Simultaneous portage - a possible way to speed it up? |
|
|
| adaptr wrote: | Hmm...yes, if an emerge first started the download thread, and just kept that going until all was downloaded, then start compiling as soon as the first package is in - that should keep it ahead of the queue.
And even if not - it would be easier to build the cross-checking of the d/l and build queues into portage than to run emerge twice and hope they don't clobber eachother... |
Actually, on a fast machine with dialup, it wouldn't be too hard for compile to catch up with download. A quick hack solution for this: have portage lock the file while downloading, and lock it before checksumming it if it finds the file already exists. Build thread would see that a file is already there, and stall waiting to get the lock until the dl completes. With this in place, you could also easily add a -F/--fetchforked option to fork a --fetchonly emerge. |
|
| Back to top |
|
|
Kethinov
Apprentice
Joined: 20 Aug 2003
Posts: 166
Location: ESU, Student Teaching 5th Grade
|
| Posted: Thu Jan 08, 2004 4:14 am Post subject: |
|
|
If the compile caught up with the download it would simply wait for the download to complete like under the current system.  |
|
| Back to top |
|
|
CheshireCat
Guru
Joined: 25 Aug 2002
Posts: 572
|
| Posted: Thu Jan 08, 2004 4:23 am Post subject: |
|
|
This won't work without some extra checking that is not currently being done by portage. If you start emerge -f for a package, and then start emerge as soon as the first process has started retrieving the file, they will both attempt to fetch it, and you get a corrupted file. I tested using wget for fetching.
You could probably make this work without modifying portage, if you wrote a fetch script that checked for an in-progress fetch, and simply waited for it to finish if one was found. Writing one that was "good enough" would be fairly easy, "perfect" might be a bit more work.
EDIT: I've put up a proof-of-concept fetch script. This is definitely only "good enough" quality, if even that. Add the following to your make.conf to use it:
| Code: | | FETCHCOMMAND='/path/to/lockfetchtest.py ${DISTDIR} ${FILE} ${URI}' |
It has received only minimal testing, it does not perform any error handling, other than making sure that two instances never write the same file, and it does not provide any form of status reporting during download. If something goes wrong, expect python to dump a stack trace, don't expect a neat error message. It "works for me" but it's not my fault if your installation gets trashed, or your CPU gives up the magic smoke, or something else unexpected or unpleasant happens. |
|
| Back to top |
|
|
X
Apprentice
Joined: 30 Apr 2002
Posts: 192
Location: Lexington KY
|
|
| Back to top |
|
|
|
Gentoo Chat
