| View previous topic :: View next topic |
| Author |
Message |
BernieKe
Tux's lil' helper
Joined: 02 Jul 2002
Posts: 130
Location: California/Bangalore/Belgium
|
 Posted: Mon Jul 16, 2007 8:50 am Post subject: |
 |
|
| Quick question: is it ok for me to set the http-replicator cache to /usr/portage/distfiles? |
|
| Back to top |
|
 |
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Wed Jul 18, 2007 6:49 am Post subject: |
|
|
| no. |
|
| Back to top |
|
|
BernieKe
Tux's lil' helper
Joined: 02 Jul 2002
Posts: 130
Location: California/Bangalore/Belgium
|
| Posted: Wed Jul 18, 2007 7:06 am Post subject: |
|
|
That's what I thought, but I had this working for the past few days like this, and everything seemed ok. (It happened by accident by the way, I discovered what I'd done after already having emerged a number of packages.)
But I now got why it was working, apparently I had commented out the http_proxy setting on the machine that's running http-replicator.
Considering that the replicator host doesn't use the replicator, and only serves the distfiles directory out, would there still be an issue with a setup like this?
It seems to make life easier, removing the need for double writes, and repcacheman (or manual alternatives.)
If you could explain why the above would be a bad thing to do (if it actually is), I'd appreciate it.
Thanks,
Bernie |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Wed Jul 18, 2007 9:28 am Post subject: |
|
|
Scroll up a couple of posts and see the link for the new, improved version of repcacheman that is much faster and much less of a memory hog and actually works with the latest portage.
| BernieKe wrote: | Considering that the replicator host doesn't use the replicator, and only serves the distfiles directory out, would there still be an issue with a setup like this?
|
All boxes should have http_proxy set to point to replicators cache, even the box hosting replicator. Anything less means the cache isn't being used and maximized.
| BernieKe wrote: |
If you could explain why the above would be a bad thing to do (if it actually is), I'd appreciate it.
|
The short answer is portage is a bad neighbor. It leaves half downloaded and corrupt files laying around the distfile dir and as a bonus leaves those junk files owned by root. Http-replicator will serve those corrupt, incomplete files to other clients because it doesn't checksum the files. replicator doesn't do the checksums because it isn't gentoo specific plus it streams files to clients as they are received. It couldn't even try to do checksums till the whole file was received which means it couldn't simultaneously stream files as they are downloaded. repcacheman is gentoo specific and does the checksums but only does this when requested, not continuously.
repcacheman deletes dups, and imports files to the cache that pass the checksum test, if any. Test your system out and see how often it actually has to do checksums in actual use. I haven't had to do checksums in many months. This isn't true for all parts of the world, some areas have better ftp mirrors closer to them. Just make sure you have a full, complete set of http mirrors defined and no ftp mirrors if you can. Portage will still download by ftp even if no ftp mirrors are listed in GENTOO_MIRRORS.
checksums are kinda expensive to do, but rarely happen for most users. If you didn't mind losing the mostly rare ftp downloads, you could just rm -rf /usr/portage/distfiles on the server. But some users are on dialup and take days to download openoffice etc. rm -rf will lose the partial download, repcacheman won't. |
|
| Back to top |
|
|
BernieKe
Tux's lil' helper
Joined: 02 Jul 2002
Posts: 130
Location: California/Bangalore/Belgium
|
| Posted: Thu Jul 19, 2007 2:50 am Post subject: |
|
|
Thanks a lot for the clear reply!
I've installed the updated repcacheman, and everything works fine once again.
I do however have one small patch for you, in order to also ignore git and mercurial sources.
| Code: |
93c93
< dc=filecmp.dircmp (distdir,dir,['cvs-src','git-src','hg-src','.locks'])
---
> dc=filecmp.dircmp (distdir,dir,['cvs-src','.locks'])
|
|
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Fri Aug 03, 2007 7:27 am Post subject: |
|
|
| BernieKe wrote: |
I do however have one small patch for you, in order to also ignore git and mercurial sources.
|
Ignoring git sounds good, I'm not familiar with mercurial but since you asked I added it also. Thanks! |
|
| Back to top |
|
|
neosimago
n00b
Joined: 07 Mar 2006
Posts: 9
|
| Posted: Thu Nov 22, 2007 9:10 am Post subject: repcacheman fails |
|
|
I'm having troubles with the repcacheman python script. Re-installing python doesn't fix the problem, and other python script seems to work fine. Here's the output:
| Code: | Checking authenticity and integrity of new files...
Searching for ebuilds...
Done!
Found 25230 ebuilds.
Extracting the checksums....
Done!
Verifying checksum's....
/usr/portage/distfiles/GDM-FlyAway.tar.gz
Traceback (most recent call last):
File "/usr/bin/repcacheman.py", line 203, in ?
if t["MD5"]:
KeyError: 'MD5' |
more of the output can be found at : http://rafb.net/p/FmewYB48.html
=> I have tried removing the problem files to no avail, and supposedly the script it designed to handle them anyways. I'll check in on the forums to see if this gets fixed later. --kudos!
Last edited by neosimago on Thu Nov 22, 2007 6:46 pm; edited 1 time in total |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
|
| Back to top |
|
|
neosimago
n00b
Joined: 07 Mar 2006
Posts: 9
|
| Posted: Thu Nov 22, 2007 7:39 pm Post subject: Re: repcacheman fails |
|
|
Thanks a bunch! I'm using your 4.3 beta of repcacheman and it's working out fine.
|
|
| Back to top |
|
|
neosimago
n00b
Joined: 07 Mar 2006
Posts: 9
|
| Posted: Thu Nov 22, 2007 7:47 pm Post subject: |
|
|
flybynite:
I'm using a gentoo local server with squid running. Will this be conflicting or redundant with http-replicator also installed? I'm not running into any problems now, and i hope not in the future, so I'll keep you posted on this issue if anything shows up.
I would also like to add Apt-get ubuntu package and source mirrors to my gentoo box, because I run a mixed environment. How would I do that? I haven't been able to find a good source on line to do that. Apparently the changes in /etc/make.conf for http_proxy doesn't work the same way as it does in ubuntu distros. More insight as to how http-replicator works would help definately in this situation. I don't suppose repcacheman would be of any help in an Apt-get http source mirror, because it's gentoo specific right? Any thoughts about writing a py script for Apt-get mirrors?
And, yes the rep4.py script does run faster with less resouces. Keep up the good work! |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Fri Nov 23, 2007 5:20 am Post subject: |
|
|
| neosimago wrote: |
I'm using a gentoo local server with squid running. Will this be conflicting or redundant with http-replicator also installed?
|
Squid doesn't know anything about portage so it is actually very inefficient with portage. When I was looking for a cache I first tried squid and was totally unsatisfied so that helped spur me on to seeing replicator developed. Don't get me wrong, squid is good at some things, just not with portage.
| neosimago wrote: |
I don't suppose repcacheman would be of any help in an Apt-get http source mirror, because it's gentoo specific right? |
Actually it's the opposite. replicator was developed first as a debian apt-get style cache with some general purpose http caching as well. I worked with the developer gertjan to add gentoo specific features later when I was looking to develop a better cache and not wanting to start from scratch.
I've not run debian style in a long time so forgive me if I miss something, but removing the -s and -f options from /etc/init.d/http-replicator should return replicator to a debian style cache. There may be other options that are helpful, check the man page, but I know these two specific options are only usable with gentoo.
The problem is, without those two options, replicator doesn't work well with portage and suffers some of the same problems as squid, namely the miss rate skyrockets.
So I'd bet you would have to run two instances of http-replicator on different ports to serve both gentoo and debian style clients. That setup should would work well and be very efficient  |
|
| Back to top |
|
|
neosimago
n00b
Joined: 07 Mar 2006
Posts: 9
|
| Posted: Fri Nov 23, 2007 7:31 am Post subject: |
|
|
flybynite
thanks for the reply. I'm looking through the man pages for http-replicator, and tried the home page: http://gertjan.freezope.org/replicator <=apparently i'm getting a message of bad gateway trying to access only that particular page. I would very much like to find more information about the best ways to run two instances of replicator, one for gentoo, which i have working well now, and one for ubuntu's apt-get. I suppose i could re-create another /etc/init.d/http-replicator script to start under another name, but that would be like having a rogue package loose on my system. What would be the best way to port replicator available on the ubuntu platform so that it provides the same functions on a gentoo system for apt-caching? |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Fri Nov 23, 2007 8:12 am Post subject: |
|
|
| neosimago wrote: | | I suppose i could re-create another /etc/init.d/http-replicator script to start under another name, but that would be like having a rogue package loose on my system. What would be the best way to port replicator available on the ubuntu platform so that it provides the same functions on a gentoo system for apt-caching? |
What kind of box is it ubuntu or gentoo? If gentoo use /usr/local/portage to create a duplicate package changing the ebuild to not conflict. You could make the new version 99 and use slotting for example.
This way the package isn't rogue....
There is some info about ebuilds here http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?part=2&chap=1#doc_chap2
If ubuntu, sorry, I can't help you. |
|
| Back to top |
|
|
neosimago
n00b
Joined: 07 Mar 2006
Posts: 9
|
| Posted: Fri Nov 23, 2007 4:54 pm Post subject: |
|
|
| flybynite wrote: | | neosimago wrote: | | I suppose i could re-create another /etc/init.d/http-replicator script to start under another name, but that would be like having a rogue package loose on my system. What would be the best way to port replicator available on the ubuntu platform so that it provides the same functions on a gentoo system for apt-caching? |
| Quote: |
What kind of box is it ubuntu or gentoo? If gentoo use /usr/local/portage to create a duplicate package changing the ebuild to not conflict. You could make the new version 99 and use slotting for example. |
--flybynite
it's a gentoo box. I can see to create a duplicate ebuild to not conflict. Where can the new version 99 found? The idea would be to slot another install to handle caching on a local gentoo machine that would be able to serve to ubuntu Apt clients on the local network. How would replicator work on an ubuntu machine? Has replicator been packaged for Apt? If so, I could use that model and adapt it to the gentoo machine to serve ubuntu machines.
--thanks
This way the package isn't rogue....
There is some info about ebuilds here http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?part=2&chap=1#doc_chap2
If ubuntu, sorry, I can't help you. |
|
|
| Back to top |
|
|
myceliv
Apprentice
Joined: 29 Nov 2007
Posts: 178
|
| Posted: Sat Dec 15, 2007 6:53 pm Post subject: |
|
|
Hopefully you can view that page. Looks like it shouldn't be too hard to do what you want, and there are specific tips on that page about 'apt-cacher' (which is in synaptic, so you could look at how it's configured) and gertjan site also talks about using 'http-replicator' with debs. Quick search of Ubuntu Forums yields nothing. I'm surprised. Bet if you start a thread there you'll get sorted in no time. |
|
| Back to top |
|
|
uprooter
n00b
Joined: 06 Mar 2005
Posts: 70
|
| Posted: Thu Jan 10, 2008 10:42 pm Post subject: |
|
|
Is there a way to tell the replicator only to download and save locally???
I need this because I share my portage tree using NFS and I want a single point of download for all of my servers.
it's a big waste if I download using the replicator, the data are sent to the client and then the client send back the file to the server using the NFS. |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Fri Jan 11, 2008 3:55 pm Post subject: |
|
|
| uprooter wrote: | Is there a way to tell the replicator only to download and save locally???
|
Yes, stop using NFS
replicator and nfs do mostly the same thing and you only need one or the other in most cases. Are you trying to do something special? |
|
| Back to top |
|
|
uprooter
n00b
Joined: 06 Mar 2005
Posts: 70
|
| Posted: Sat Jan 12, 2008 5:42 pm Post subject: |
|
|
The difference is that with NFS i'm holding only one copy of the portage tree on my LAN.
with http-replicator I need to have a local portage tree on each machine.
Fix me if I'm wrong. |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Sat Jan 12, 2008 6:24 pm Post subject: |
|
|
| uprooter wrote: | The difference is that with NFS i'm holding only one copy of the portage tree on my LAN.
with http-replicator I need to have a local portage tree on each machine.
Fix me if I'm wrong. |
Your not really wrong, but there is a way.
Just move the packages and the distfile dir out from under the "portage" dir by adding this to /etc/make.conf
| Code: |
PKGDIR=/var/tmp/packages
DISTDIR=/var/tmp/distfiles
|
Now you can NFS share the ebuilds etc. and only sync once but still have replicator manage and share binary packages and distfiles. This will prevent the double network copy you were asking about.
This is similar to what I use. I didn't like duplicating portage on all my boxes and laptops, it was a waste of space. I also didn't like not being able to upgrade or add packages while the server was down or my laptop was on the road so I needed each box to have a copy of portage.
A long time ago I found https://forums.gentoo.org/viewtopic-t-401647.html. I now use a variation of this system. I can compress portage down to ~47MB and copy this to my laptops and other boxes using replicator. I have a custom script that does this for me.
For some more Ideas the compressed portage idea see https://forums.gentoo.org/viewtopic-t-465367.html or if you want to know more about my simple system just ask. |
|
| Back to top |
|
|
uprooter
n00b
Joined: 06 Mar 2005
Posts: 70
|
| Posted: Sun Jan 13, 2008 7:32 am Post subject: |
|
|
| Code: | PKGDIR=/var/tmp/packages
DISTDIR=/var/tmp/distfiles |
Not really helpful since the distfiles are now stored on each machine.
I want only one copy of the tree and the distfiles (which is heavier than the tree in my case) on the LAN.
I`m talking about 20 old desktops/servers that some of them have 6GB IDE/SCSI HDD.
well,,
I can implement this on my own with a little perl daemon that listens for incoming connections ans spawn wget for each URL request it gets from the clients, then configure the clients to have a special FETCHCOMMAND in make.conf that submits the URL to that perl daemon.
Just thought that the http-replicator is able to do it since it's already part of it's job. |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Sun Jan 13, 2008 8:13 pm Post subject: |
|
|
| uprooter wrote: |
Not really helpful since the distfiles are now stored on each machine.
I want only one copy of the tree and the distfiles (which is heavier than the tree in my case) on the LAN.
|
all your distfiles don't need to be stored at each machine. Only the files needed for the package that your are emerging are downloaded and they only need to be there long enough for the emerge. After the update, delete them with rm -rf /var/tmp/distifles.
emerging is much faster when the files are local and not over the network anyway.
I still don't understand what you want. Why use replicator at all in your case? Is there some advantage to you that I don't see? |
|
| Back to top |
|
|
dbishop
Tux's lil' helper
Joined: 08 Dec 2007
Posts: 107
|
| Posted: Sun May 04, 2008 5:15 am Post subject: ERROR 400: Bad Request |
|
|
From the http-replicator server: no packages will download via http, they always fail and then fetch using ftp. Same thing happens from clients, http fails but ftp succeeds. If I comment out the http_proxy line in the client /etc/make.conf file, the same files will always download via http. Same is true for the http-replicator server.
| Code: | >>> Emerging (1 of 2) x11-libs/goffice-0.6.2 to /
>>> Downloading 'http://gentoo.osuosl.org/distfiles/goffice-0.6.2.tar.bz2'
--22:53:36-- http://gentoo.osuosl.org/distfiles/goffice-0.6.2.tar.bz2
=> `/usr/portage/distfiles/goffice-0.6.2.tar.bz2'
Resolving local.mydomain.com... 192.168.0.243
Connecting to local.mydomain.com|192.168.0.243|:8090... connected.
Proxy request sent, awaiting response... 400 Bad Request
22:53:36 ERROR 400: Bad Request.
>>> Downloading 'http://distfiles.gentoo.org/distfiles/goffice-0.6.2.tar.bz2'
--22:53:36-- http://distfiles.gentoo.org/distfiles/goffice-0.6.2.tar.bz2
=> `/usr/portage/distfiles/goffice-0.6.2.tar.bz2'
Resolving local.mydomain.com... 192.168.0.243
Connecting to local.mydomain.com|192.168.0.243|:8090... connected.
Proxy request sent, awaiting response... 400 Bad Request
22:53:36 ERROR 400: Bad Request.
>>> Downloading 'http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/goffice-0.6.2.tar.bz2'
--22:53:36-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/goffice-0.6.2.tar.bz2
=> `/usr/portage/distfiles/goffice-0.6.2.tar.bz2'
Resolving local.mydomain.com... 192.168.0.243
Connecting to local.mydomain.com|192.168.0.243|:8090... connected.
Proxy request sent, awaiting response... 400 Bad Request
22:53:36 ERROR 400: Bad Request. |
After this, the ftp fetching begins and succeeds, but of course the cache is not properly updated. If the client ftp's the files, they do not end up on the server at all; if the server ftp's the files they end up in /usr/portage/distfiles/ as expected and can be moved with repcacheman, and then the client will get them from the http-replicator cache a desired. tcpdump shows that the requests are going to the http-replicator server properly.
I'm sure it's something obvious, but I'm just missing it. Any help would be appreciated...
I should mention that everything works perfectly if the files are already in /var/cache/http-replicator/ and I don't have any issues with repcacheman.
Edited to add:
It seems that http-replicator is addressing the proxied http request to the default gateway rather than the true destination server. The local client x.x.x.11 uses the http_proxy x.x.x.243 which will return the file properly if it is in it's cache. If it is not there, the http_proxy server (x.x.x.243) asks x.x.x.1 (the default gateway) for the file. Any requests made by the http_proxy machine itself (as opposed to requests made through it from other clients) are properly fetched and placed in it's local distfiles cache, and repcacheman works properly on them, and then the x.x.x.11 client will fetch properly from the http_proxy machine (x.x.x.243).
I suspect it's something I've misconfigured, but I'm just not seeing it. Any help is greatly appreciated. |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Sat May 17, 2008 8:12 am Post subject: Re: ERROR 400: Bad Request |
|
|
| How about showing your config's on the server and the clients? |
|
| Back to top |
|
|
melinux
n00b
Joined: 25 May 2006
Posts: 59
Location: Malta
|
| Posted: Sat Feb 14, 2009 4:24 pm Post subject: |
|
|
I've been running http-replicator for a while. Now I started getting:
| Code: |
/usr/lib/portage/pym/portage_manifest.py:39: DeprecationWarning: DEPRECATION NOTICE: The portage_manifest module was replaced by portage.manifest
/usr/lib/portage/pym/portage_checksum.py:39: DeprecationWarning: DEPRECATION NOTICE: The portage_checksum module was replaced by portage.checksum
/usr/lib/portage/pym/portage_exception.py:39: DeprecationWarning: DEPRECATION NOTICE: The portage_exception module was replaced by portage.exception
|
Is there going to be any related update? |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Sat Feb 14, 2009 4:45 pm Post subject: |
|
|
| I seem to remember fixing this a while ago, I guess it might be changing again? What version of replicator and portage are you running? |
|
| Back to top |
|
|
|
Documentation, Tips & Tricks
