| View previous topic :: View next topic |
| Author |
Message |
duckz
Apprentice
Joined: 26 Jun 2007
Posts: 158
|
 Posted: Wed Feb 06, 2008 1:13 pm Post subject: Why these improvement is not in implemented? |
 |
|
I've been using gentoo for almost 2 years, and keep wondering why these good tricks (improvement) is not made default by gentoo dev?
1. Squashed /usr/portage. normal emerge --sync will take hours and lots of harddisk space and activity, while if the portage tree is distributed in squashed fs format will be very small in size with less disk activity in the server and user.
2. axel download instead of wget, its clearly a major improvement in download speed using axel compares to wget.
so why the good improvement in not implemented by default?.
just a thought though.
duckz
_________________
emerge -va techygadget.com <- install my blog in gentoo
./configure anime wallpaper
make install premium drupal theme |
|
| Back to top |
|
 |
Veldrin
Veteran
Joined: 27 Jul 2004
Posts: 1945
Location: Zurich, Switzerland
|
| Posted: Wed Feb 06, 2008 1:27 pm Post subject: |
|
|
Fair question.
1. the problem is not the portage tree itself, but rather the whole package "database" which lies in /var/db.
2. My guess is that wget is pretty old and stable, and for a package manager you need something that "just works" (TM).
OTOH I don't really see and advantage of having multiple connection open. On a slow connection (56k, ISDN and alike) it is more likely that you create more overhead, thus slowing down the entire download process. And on a faster connection (ADSL, Universities) the download time is not really an issue.
| Quote: | | so why the good improvement in not implemented by default?. |
And as always, those who code decide.
just my .02$
V. |
|
| Back to top |
|
|
JeliJami
Veteran
Joined: 17 Jan 2006
Posts: 1086
Location: Belgium
|
| Posted: Wed Feb 06, 2008 2:35 pm Post subject: Re: Why these improvement is not in implemented? |
|
|
| duckz wrote: | 2. axel download instead of wget, its clearly a major improvement in download speed using axel compares to wget.
|
If you want to use axel, just instruct portage to use it: see /etc/make.conf.example, search for FETCHCOMMAND.
_________________
Unanswered Post Initiative | Search | FAQ
Former username: davjel |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9555
Location: beyond the rim
|
| Posted: Wed Feb 06, 2008 3:10 pm Post subject: |
|
|
| duckz wrote: | | 1. Squashed /usr/portage. normal emerge --sync will take hours and lots of harddisk space and activity, while if the portage tree is distributed in squashed fs format will be very small in size with less disk activity in the server and user. |
Except that now on every sync every user would have to fetch the whole tree, which even compressed is somewhere around 30-40MB, while the current setup on average needs only a fraction of that, so you're just replacing one resource with another. Also the squashfs setup requires quite a bit more configuration on the user side.
| Veldrin wrote: | | 1. the problem is not the portage tree itself, but rather the whole package "database" which lies in /var/db. |
What problem are you talking about? |
|
| Back to top |
|
|
duckz
Apprentice
Joined: 26 Jun 2007
Posts: 158
|
| Posted: Wed Feb 06, 2008 3:15 pm Post subject: |
|
|
| Quote: | | If you want to use axel, just instruct portage to use it: see /etc/make.conf.example, search for FETCHCOMMAND. |
Thanks, I did that long time ago, If you have fast connection then I guess everything is good. but if you have slow connection, moving from wget to axel improves a lot of downloading time.
| Quote: | | Also the squashfs setup requires quite a bit more configuration on the user side. |
That's why its not set by default from the start?.
30 - 40 mb of download in one single big file is better than having 1000+ small file download. IMHO. 
anyway this is just my opinion.
[/code]
_________________
emerge -va techygadget.com <- install my blog in gentoo
./configure anime wallpaper
make install premium drupal theme |
|
| Back to top |
|
|
Veldrin
Veteran
Joined: 27 Jul 2004
Posts: 1945
Location: Zurich, Switzerland
|
| Posted: Wed Feb 06, 2008 3:18 pm Post subject: |
|
|
| Quote: | | Quote: | | 1. the problem is not the portage tree itself, but rather the whole package "database" which lies in /var/db. |
What problem are you talking about? |
I was talking about the excessive disk access which is caused by the emerge --metadata which is part of the emerge --sync (Or am I wrong). Yes downloading the portage tree does some diskaccess, but IIRC metadata part is worse. |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9555
Location: beyond the rim
|
| Posted: Wed Feb 06, 2008 4:27 pm Post subject: |
|
|
| Veldrin wrote: | | Quote: | | Quote: | | 1. the problem is not the portage tree itself, but rather the whole package "database" which lies in /var/db. |
What problem are you talking about? |
I was talking about the excessive disk access which is caused by the emerge --metadata which is part of the emerge --sync (Or am I wrong). Yes downloading the portage tree does some diskaccess, but IIRC metadata part is worse. |
Then you've confused /var/db/pkg with /var/cache/edb/dep  |
|
| Back to top |
|
|
Monkeh
Veteran
Joined: 06 Aug 2005
Posts: 1656
Location: England
|
| Posted: Wed Feb 06, 2008 4:44 pm Post subject: |
|
|
| duckz wrote: | | 30 - 40 mb of download in one single big file is better than having 1000+ small file download. IMHO. |
Except the servers are paying by used bandwidth. Which means it'll cost them much, much more. |
|
| Back to top |
|
|
i92guboj
Bodhisattva
Joined: 30 Nov 2004
Posts: 10315
Location: Córdoba (Spain)
|
| Posted: Wed Feb 06, 2008 4:49 pm Post subject: Re: Why these improvement is not in implemented? |
|
|
| duckz wrote: | I've been using gentoo for almost 2 years, and keep wondering why these good tricks (improvement) is not made default by gentoo dev?
1. Squashed /usr/portage. normal emerge --sync will take hours and lots of harddisk space and activity, while if the portage tree is distributed in squashed fs format will be very small in size with less disk activity in the server and user.
|
First, it doesn't take hours, it's more like minutes on a 0.5 mbit connection, which is pretty low for today standards. Second, squashfs will not help with that. Third, partition your portage sanely with a 512 bytes or 1024 block size, and you don't need all the squashfs hassle. Fourth, compression in squashfs adds an overhead on cpu, which might not be suitable for lower end machines, as such, it is not suitable for a default.
| Quote: |
2. axel download instead of wget, its clearly a major improvement in download speed using axel compares to wget.
|
Unless you isp limits the speed per connection, using axel over wget will not give you any benefit. Plus when I use axel on amd64 I get random segfaults on my log messages. It doesn't seem to cause problems, but it certainly is not a good thing for a default tool.
Just my view of the thing. |
|
| Back to top |
|
|
Rad
Guru
Joined: 11 Feb 2004
Posts: 401
Location: Bern, Switzerland
|
| Posted: Thu Feb 07, 2008 12:16 am Post subject: |
|
|
About 1: It is already happening that overlays migrate to GIT. Keep an eye on that, it does pretty much solve most sync performance / efficiency issues - more so than squashfs can. The next thing that could be happen is that someone codes something that eliminates the need to check out the files onto a regular filesystem (allowing just use the .git/ files). If that happens, then it'll really be pretty and sweet.
Other, currently available enhancements are deltup and emerge-webrsync. Have fun with them!
2: Well, you can already do that if you want to. Note download accelerators generate a little network overhead and -worse- additional read accesses on the server side. The latter is probably the main problem; I figure only so few boxes from the Gentoo mirrors actually have all files cached on RAM (random access media in the broader sense). |
|
| Back to top |
|
|
duckz
Apprentice
Joined: 26 Jun 2007
Posts: 158
|
| Posted: Thu Feb 07, 2008 11:44 am Post subject: |
|
|
| Quote: |
Except the servers are paying by used bandwidth. Which means it'll cost them much, much more. |
I am confused a 40 mb file in single file total will cost more compares to 1000+ files in 200 mb total?
| Quote: | | compression in squashfs adds an overhead on cpu, which might not be suitable for lower end machines, as such, it is not suitable for a default. |
but If its distributed from default, isnt the one that compress them is gentoo dev? not user?.
I dont know I dont setup my box correctly or its normal practice, but everytime I do emerge --sync I got disconnected halfway because of time out. and have to redo the whole process again from another mirror. is this a waste of bandwidth?.
[/code]
_________________
emerge -va techygadget.com <- install my blog in gentoo
./configure anime wallpaper
make install premium drupal theme |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9555
Location: beyond the rim
|
| Posted: Thu Feb 07, 2008 12:04 pm Post subject: |
|
|
| duckz wrote: | | Quote: |
Except the servers are paying by used bandwidth. Which means it'll cost them much, much more. |
I am confused a 40 mb file in single file total will cost more compares to 1000+ files in 200 mb total? |
rsync only transfers the differences, not the whole tree. |
|
| Back to top |
|
|
|
Gentoo Chat
