| View previous topic :: View next topic |
| Author |
Message |
DArtagnan
l33t
Joined: 30 Apr 2002
Posts: 942
Location: Israel, Jerusalem
|
 Posted: Mon Jul 29, 2002 5:52 am Post subject: wget optimizations |
 |
|
Do you have a large bandwidth like me?
If you do:
1) # vim /etc/make.conf
2) add this line:
FETCHCOMMAND='/usr/bin/wget --limit-rate=1000k ${URI} -P ${DISTDIR}'
3) replace the 1000k (--limit-rate=1000k) with your possible value
My results:
[code]
overseas voicu # emerge -f evolution
Calculating dependencies ...done!
>>> emerge dev-util/indent-2.2.6 to /
>>> emerge gnome-base/oaf-0.6.8-r1 to /
>>> emerge dev-util/guile-1.4-r3 to /
>>> emerge gnome-base/gconf-1.0.8-r3 to /
>>> emerge gnome-base/gnome-vfs-1.0.5-r2 to /
>>> Resuming download...
>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gnome-vfs-1.0.5.tar.gz
--08:57:14-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gnome-vfs-1.0.5.tar.gz
=> `/usr/portage/distfiles/gnome-vfs-1.0.5.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 206 Partial Content
Length: 1,048,954 (12,031 to go) [application/x-tar]
100%[======================================================>] 1,048,954 11.47M/s ETA 00:00
08:57:14 (11.47 MB/s) - `/usr/portage/distfiles/gnome-vfs-1.0.5.tar.gz' saved [1048954/1048954]
>>> emerge gnome-base/gnome-print-0.35-r3 to /
>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gnome-print-0.35.tar.gz
--08:57:15-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gnome-print-0.35.tar.gz
=> `/usr/portage/distfiles/gnome-print-0.35.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 991,657 [application/x-tar]
100%[======================================================>] 991,657 860.05K/s ETA 00:00
08:57:16 (860.05 KB/s) - `/usr/portage/distfiles/gnome-print-0.35.tar.gz' saved [991657/991657]
>>> emerge gnome-base/libglade-0.17-r6 to /
>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/libglade-0.17.tar.gz
--08:57:16-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/libglade-0.17.tar.gz => `/usr/portage/distfiles/libglade-0.17.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 418,921 [application/x-tar]
100%[======================================================>] 418,921 845.25K/s ETA 00:00
08:57:17 (845.25 KB/s) - `/usr/portage/distfiles/libglade-0.17.tar.gz' saved [418921/418921]
>>> emerge gnome-extra/gal-0.19.3 to /
>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gal-0.19.3.tar.bz2--08:57:17-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gal-0.19.3.tar.bz2
=> `/usr/portage/distfiles/gal-0.19.3.tar.bz2'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 404 Not Found
08:57:17 ERROR 404: Not Found.
>>> Downloading ftp://ftp.gnome.org/pub/GNOME/unstable/sources/gal/gal-0.19.3.tar.bz2
--08:57:17-- ftp://ftp.gnome.org/pub/GNOME/unstable/sources/gal/gal-0.19.3.tar.bz2
=> `/usr/portage/distfiles/gal-0.19.3.tar.bz2'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
[ <=> ] 903,527 106.69K/s
08:57:29 (106.69 KB/s) - `/usr/portage/distfiles/gal-0.19.3.tar.bz2' saved [903527]
>>> emerge gnome-base/control-center-1.4.0.5-r1 to /
>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/control-center-1.4.0.5.tar.bz2
--08:57:29-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/control-center-1.4.0.5.tar.bz2
=> `/usr/portage/distfiles/control-center-1.4.0.5.tar.bz2'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 3,382,798 [application/x-tar]
100%[======================================================>] 3,382,798 851.20K/s ETA 00:00
08:57:33 (851.20 KB/s) - `/usr/portage/distfiles/control-center-1.4.0.5.tar.bz2' saved [3382798/3382798]
>>> emerge gnome-base/libghttp-1.0.9-r2 to /
>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/libghttp-1.0.9.tar.gz
--08:57:33-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/libghttp-1.0.9.tar.gz
=> `/usr/portage/distfiles/libghttp-1.0.9.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 147,094 [application/x-tar]
100%[======================================================>] 147,094 886.71K/s ETA 00:00
08:57:33 (886.71 KB/s) - `/usr/portage/distfiles/libghttp-1.0.9.tar.gz' saved [147094/147094]
>>> emerge gnome-base/bonobo-1.0.19-r1 to /
>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/bonobo-1.0.19.tar.gz
--08:57:33-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/bonobo-1.0.19.tar.gz => `/usr/portage/distfiles/bonobo-1.0.19.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 1,556,542 [application/x-tar]
100%[======================================================>] 1,556,542 840.28K/s ETA 00:00
08:57:35 (840.28 KB/s) - `/usr/portage/distfiles/bonobo-1.0.19.tar.gz' saved [1556542/1556542]
>>> emerge gnome-extra/gtkhtml-1.0.2-r2 to /
>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gtkhtml-1.0.2.tar.gz
--08:57:35-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gtkhtml-1.0.2.tar.gz => `/usr/portage/distfiles/gtkhtml-1.0.2.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 1,303,882 [application/x-tar]
100%[======================================================>] 1,303,882 89.99K/s ETA 00:00
08:57:50 (89.99 KB/s) - `/usr/portage/distfiles/gtkhtml-1.0.2.tar.gz' saved [1303882/1303882]
>>> emerge app-text/gnome-spell-0.4.1-r2 to /
>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gnome-spell-0.4.1.tar.gz
--08:57:50-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gnome-spell-0.4.1.tar.gz
=> `/usr/portage/distfiles/gnome-spell-0.4.1.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 185,220 [application/x-tar]
100%[======================================================>] 185,220 918.17K/s ETA 00:00
08:57:50 (918.17 KB/s) - `/usr/portage/distfiles/gnome-spell-0.4.1.tar.gz' saved [185220/185220]
>>> emerge gnome-extra/bonobo-conf-0.14 to /
>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/bonobo-conf-0.14.tar.gz
--08:57:50-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/bonobo-conf-0.14.tar.gz
=> `/usr/portage/distfiles/bonobo-conf-0.14.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 438,259 [application/x-tar]
100%[======================================================>] 438,259 839.19K/s ETA 00:00
08:57:51 (839.19 KB/s) - `/usr/portage/distfiles/bonobo-conf-0.14.tar.gz' saved [438259/438259]
>>> emerge dev-util/gob-1.0.12 to /
>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gob-1.0.12.tar.gz
--08:57:51-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gob-1.0.12.tar.gz
=> `/usr/portage/distfiles/gob-1.0.12.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 171,225 [application/x-tar]
100%[======================================================>] 171,225 950.07K/s ETA 00:00
08:57:51 (950.07 KB/s) - `/usr/portage/distfiles/gob-1.0.12.tar.gz' saved [171225/171225]
>>> emerge net-mail/evolution-1.0.8-r1 to /
>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/evolution-1.0.8.tar.gz
--08:57:51-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/evolution-1.0.8.tar.gz
=> `/usr/portage/distfiles/evolution-1.0.8.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 13,929,927 [application/x-tar]
100%[======================================================>] 13,929,927 853.09K/s ETA 00:00
08:58:07 (853.09 KB/s) - `/usr/portage/distfiles/evolution-1.0.8.tar.gz' saved [13929927/13929927]
>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/db-3.1.17.tar.gz
--08:58:07-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/db-3.1.17.tar.gz
=> `/usr/portage/distfiles/db-3.1.17.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 1,954,117 [application/x-tar]
100%[======================================================>] 1,954,117 872.57K/s ETA 00:00
08:58:10 (872.57 KB/s) - `/usr/portage/distfiles/db-3.1.17.tar.gz' saved [1954117/1954117]
_________________
All for one and one for All
--
MACPRO machine...
Last edited by DArtagnan on Mon Jul 29, 2002 6:54 am; edited 2 times in total |
|
| Back to top |
|
 |
delta407
Bodhisattva
Joined: 23 Apr 2002
Posts: 2876
Location: Chicago, IL
|
| Posted: Mon Jul 29, 2002 5:55 am Post subject: |
|
|
...how did it perform without the rate limiting?
_________________
I don't believe in witty sigs. |
|
| Back to top |
|
|
DArtagnan
l33t
Joined: 30 Apr 2002
Posts: 942
Location: Israel, Jerusalem
|
| Posted: Mon Jul 29, 2002 6:49 am Post subject: |
|
|
| delta407 wrote: | | ...how did it perform without the rate limiting? |
Have no idea but if you do not specify it, the app. doesn't use all your band...not like prozilla 
_________________
All for one and one for All
--
MACPRO machine... |
|
| Back to top |
|
|
bazik
Retired Dev
Joined: 22 Jul 2002
Posts: 277
Location: Behind you.
|
| Posted: Mon Jul 29, 2002 7:03 am Post subject: |
|
|
| DArtagnan wrote: |
=> `/usr/portage/distfiles/gnome-vfs-1.0.5.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 206 Partial Content
Length: 1,048,954 (12,031 to go) [application/x-tar]
100%[======================================================>] 1,048,954 11.47M/s ETA 00:00
|
I get over 20M/s when I cache all the files in my Proxy first
_________________
Gentoo Linux/Sparc Developer
http://dev.gentoo.org/~bazik/ |
|
| Back to top |
|
|
DArtagnan
l33t
Joined: 30 Apr 2002
Posts: 942
Location: Israel, Jerusalem
|
| Posted: Mon Jul 29, 2002 7:29 am Post subject: |
|
|
| bAZiK wrote: | | DArtagnan wrote: |
=> `/usr/portage/distfiles/gnome-vfs-1.0.5.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 206 Partial Content
Length: 1,048,954 (12,031 to go) [application/x-tar]
100%[======================================================>] 1,048,954 11.47M/s ETA 00:00
|
I get over 20M/s when I cache all the files in my Proxy first |
How do you do this?
_________________
All for one and one for All
--
MACPRO machine... |
|
| Back to top |
|
|
debian
n00b
Joined: 24 Jun 2002
Posts: 22
|
| Posted: Mon Jul 29, 2002 1:31 pm Post subject: |
|
|
| DArtagnan wrote: | | delta407 wrote: | | ...how did it perform without the rate limiting? |
Have no idea but if you do not specify it, the app. doesn't use all your band...not like prozilla |
While you mentioned prozilla...I could get much better downloading speed with prozilla by uncomment the following line in /etc/make.conf:
| Code: | | FETCHCOMMAND='/usr/bin/proz --no-getch -s ${URI} -P ${DISTDIR}' |
I compare it against your method and prozilla is about twice as fast. However, I'm only using ADSL, which may be different in your case.
Note: you must `emerge prozilla` first. |
|
| Back to top |
|
|
DArtagnan
l33t
Joined: 30 Apr 2002
Posts: 942
Location: Israel, Jerusalem
|
| Posted: Mon Jul 29, 2002 4:01 pm Post subject: |
|
|
| debian wrote: | | DArtagnan wrote: | | delta407 wrote: | | ...how did it perform without the rate limiting? |
Have no idea but if you do not specify it, the app. doesn't use all your band...not like prozilla |
While you mentioned prozilla...I could get much better downloading speed with prozilla by uncomment the following line in /etc/make.conf:
| Code: | | FETCHCOMMAND='/usr/bin/proz --no-getch -s ${URI} -P ${DISTDIR}' |
I compare it against your method and prozilla is about twice as fast. However, I'm only using ADSL, which may be different in your case.
Note: you must `emerge prozilla` first. |
I KNOW!!! Prozilla rocks but it doesn't work for me, I think my prozy blocks my prozilla , is it possible to configure prozilla for proxy pass?
_________________
All for one and one for All
--
MACPRO machine... |
|
| Back to top |
|
|
delta407
Bodhisattva
Joined: 23 Apr 2002
Posts: 2876
Location: Chicago, IL
|
| Posted: Mon Jul 29, 2002 8:33 pm Post subject: |
|
|
| bAZiK wrote: | | DArtagnan wrote: |
=> `/usr/portage/distfiles/gnome-vfs-1.0.5.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 206 Partial Content
Length: 1,048,954 (12,031 to go) [application/x-tar]
100%[======================================================>] 1,048,954 11.47M/s ETA 00:00
|
I get over 20M/s when I cache all the files in my Proxy first |
No, see, if you'll notice it had 12,031 bytes to transfer, so that number is really far off.
_________________
I don't believe in witty sigs. |
|
| Back to top |
|
|
debian
n00b
Joined: 24 Jun 2002
Posts: 22
|
| Posted: Tue Jul 30, 2002 6:58 am Post subject: |
|
|
| DArtagnan wrote: | | I KNOW!!! Prozilla rocks but it doesn't work for me, I think my prozy blocks my prozilla , is it possible to configure prozilla for proxy pass? |
Strange, prozilla should work where wget work. Any special restriction imposed by your proxy? |
|
| Back to top |
|
|
DArtagnan
l33t
Joined: 30 Apr 2002
Posts: 942
Location: Israel, Jerusalem
|
| Posted: Tue Jul 30, 2002 7:01 am Post subject: |
|
|
| debian wrote: | | DArtagnan wrote: | | I KNOW!!! Prozilla rocks but it doesn't work for me, I think my prozy blocks my prozilla , is it possible to configure prozilla for proxy pass? |
Strange, prozilla should work where wget work. Any special restriction imposed by your proxy? |
I really don't know...what should I ask my admin about it?
_________________
All for one and one for All
--
MACPRO machine... |
|
| Back to top |
|
|
debian
n00b
Joined: 24 Jun 2002
Posts: 22
|
| Posted: Tue Jul 30, 2002 7:17 am Post subject: |
|
|
| DArtagnan wrote: | | debian wrote: | | DArtagnan wrote: | | I KNOW!!! Prozilla rocks but it doesn't work for me, I think my prozy blocks my prozilla , is it possible to configure prozilla for proxy pass? |
Strange, prozilla should work where wget work. Any special restriction imposed by your proxy? |
I really don't know...what should I ask my admin about it? |
You don't need to ask, what information would you expect from those arrogant bastards(well, I'm an admin but I'm an exception ).
First of all we need to know what kind of problem you got when using prozilla, then we might get some hint why wget works where prozilla doesn't. |
|
| Back to top |
|
|
DArtagnan
l33t
Joined: 30 Apr 2002
Posts: 942
Location: Israel, Jerusalem
|
| Posted: Tue Jul 30, 2002 7:20 am Post subject: |
|
|
| debian wrote: | | DArtagnan wrote: | | debian wrote: | | DArtagnan wrote: | | I KNOW!!! Prozilla rocks but it doesn't work for me, I think my prozy blocks my prozilla , is it possible to configure prozilla for proxy pass? |
Strange, prozilla should work where wget work. Any special restriction imposed by your proxy? |
I really don't know...what should I ask my admin about it? |
You don't need to ask, what information would you expect from those arrogant bastards(well, I'm an admin but I'm an exception ).
First of all we need to know what kind of problem you got when using prozilla, then we might get some hint why wget works where prozilla doesn't. |
Well, don't forget that "wget" allows you to configure it for proxy at: /etc/wget/wget
There you can specify if to use proxy and then address:port
Prozilla does not support proxy configuration...
_________________
All for one and one for All
--
MACPRO machine... |
|
| Back to top |
|
|
debian
n00b
Joined: 24 Jun 2002
Posts: 22
|
| Posted: Tue Jul 30, 2002 8:30 am Post subject: |
|
|
| DArtagnan wrote: | Well, don't forget that "wget" allows you to configure it for proxy at: /etc/wget/wget
There you can specify if to use proxy and then address:port
Prozilla does not support proxy configuration... |
You can set it up in the shell before executing prozilla:
| Code: | | export http_proxy="http://yourproxy.com:8080/" |
Also, for ftp:
| Code: | | export ftp_proxy="http://yourproxy.com:8080/" |
Btw, you can catch me in #gentoo of OPN if you think forum is not interactive enough. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Documentation, Tips & Tricks
