HOWTO:Download Cache for your LAN-Http-Replicator (ver 3.0)

flybynite · l33t Joined: 06 Dec 2002 Posts: 620

This is most likely caused by your overlays not conforming to the gentoo standard.

This:

hothead · Posted: Mon Dec 27, 2004 4:12 pm Post subject:

I found out that the gentoo-de tree is causing this problem.
Now I've another question:
Is it good to use the same directory for distfiles and replicator cache? When setting it to the same directory I noticed that repcacheman marked all my distfiles as dubs and deleted them.

hothead
_________________
Got a question? - http://justfuckinggoogleit.com/

flybynite · l33t Joined: 06 Dec 2002 Posts: 620

hothead · Posted: Wed Dec 29, 2004 2:17 pm Post subject:

The gentoo-de tree has some more ebuilds that are not in the main portage tree.
For example the whole vdr (plugins) ximian-openoffice etc..

You can view the cvs here:http://www.gentoo.de/viewcvs/gentoo-x86/

hothead
_________________
Got a question? - http://justfuckinggoogleit.com/

soulwarrior · Guru Joined: 21 Oct 2002 Posts: 331

Thanks for this great program

I do have a question, does http-replicator delete obsolete files, which are no longer needed by any ebuild in the portage tree?

flybynite · l33t Joined: 06 Dec 2002 Posts: 620

Sorry for the delay, I've been out for a while..

I've made no attempt to try to purge outdated distfiles. I've given the reasons why in an earilier post. It's actually a harder problem that you think at first because nobody can agree what needs to be purged.

What I can offer is that http-replicator/repcacheman create no special worries and any of the current scripts you can find on this board can work with replicator.

Search on this board and you can find many scripts that purge/delete/trash distfiles based on no longer in portage/newer version available/not installed/etc....

Trespasser · n00b Joined: 12 Aug 2003 Posts: 14

I have just installed the http-replicator on one virutal server that is running on af VMware ESX Server (one of 4 esx-servers).
The good thing here is that the hardware is the same all way round, so I have just updated the http-replicator-server and build the binary packages for everything new. After that I change the make.conf's of the other gentoo-server running on our ESX-servers and updated them in a snap.
I have about 15-20 (some testing going on here) Gentoo-servers running on ESX-server, so this can really make a complete Gentoo-install quick ( I know I can just copy the image file, but sometimes the disk-space in the file isn't enough).

This works like a charm I must say.

Again, Thnx.

flybynite · l33t Joined: 06 Dec 2002 Posts: 620

Thanks for the report Trespasser!!

Many people only know Gentoo as a compile from source disto. They don't realize how easy it is to compile a custom version of a package and then use http-replicator to distribute the binary to all your other machines.

This is same way many people administrate a large number of boxes using other disto's. They may use the binaries for the base system from distro "X", but they compile a custom version of Apache from source. With gentoo, they can still use the binaries from the latest release, and use portage to compile apache exactly as they want it. Then they can easily distribute the binary to other gentoo machines. This is a similiar workflow to other disto's, only easier because you have a few thousand of other admins helping through portage.

Http-Replicator is the best way I know of to manage multiple gentoo boxes. It multiplies the power of portage! I hope many others will see the advantages and use http-replicator to administer multiple machines like your doing.

jopalm · Posted: Sun Jan 23, 2005 7:13 am Post subject:

I've been sucessfully using HTTP-Replicator on my six box lan for several months. I'm very pleased with it's functionality and performance. However now I'm attempting to move my cache server from one box to another and have run into a problem.

Everything appears to be correctly configured on both server and clients. HTTP-Replicator is sucessfully started. The trouble begins when a client wants to emerge a package and tries connecting to the server for the source file. At this point the client states the connection is refused. A look at the server log indicates that, upon connection, the server has stopped.

flybynite · l33t Joined: 06 Dec 2002 Posts: 620

For some reason, the file requested by the client is missing...

I would bet you have a typo in the make config of the client. Post the FETCHCOMMAND your using. You might have left a space or unbalanced quote when you changed it to the new server.

If you don't spot it, post some more info. Your configs and the output of the emerge could also be helpful....

You can confirm some of your settings like this:

jopalm · Posted: Tue Jan 25, 2005 4:37 pm Post subject:

Thanks for the response!

I'm off-site at the moment and won't be able to get the specifics you inquired about until this evening (PST).

jopalm · Posted: Wed Jan 26, 2005 6:42 am Post subject:

OK, I believe I have collected the relevant data...

First, with the server set-up and running on magnolia (new install) we get this fail mode during an emerge from ballard (actually, it's the same from any of the six clients)

flybynite · l33t Joined: 06 Dec 2002 Posts: 620

flybynite · l33t Joined: 06 Dec 2002 Posts: 620

jopalm · Posted: Wed Jan 26, 2005 3:00 pm Post subject:

Ok, here's a look at magnolia - the non-functional server:

flybynite · l33t Joined: 06 Dec 2002 Posts: 620

Ok, latest replicator, and no unstable packages in use.

There is an update to portage, r15, it came out the same day as r14 which probably means it was buggy, might want to upgrade...

You missed the env on the CLIENT, not the failed server magnolia. My original theory is the env on the CLIENT is in error.

Also the /etc/conf.d/http-replicator on the working server and the failed server could also be helpful.

We'll get this, one step at a time...

fyreflyer · n00b Joined: 02 Jan 2005 Posts: 12

I just want to say thanks flybynite!

I found setup to be a breeze; http-replicator makes me feel like a better netizen.

jopalm · Posted: Thu Jan 27, 2005 8:29 am Post subject:

Oops, my mistake... YOu asked about the client and I listed the server.

Here we go with the client variables before an emerge, a failed emerge and then an echo of hte variable right after.

jopalm · Posted: Thu Jan 27, 2005 8:41 am Post subject:

An here are the conf.d files from the working server fremont and the failed server magnolia:

flybynite · l33t Joined: 06 Dec 2002 Posts: 620

Everything still looks normal. The request from the client should be normal and http-replicator still shouldn't die if it wasn't.....

I looked again at the traceback. It shows that the error occurs in Python proper, in asyncore. The back and forth is just Python checking for data in the socket, finding data is ready, telling http-replicator which says ok give it to me, then Python errors when reading the data from the socket. Just shouldn't happen....

jopalm · Posted: Fri Jan 28, 2005 10:27 pm Post subject:

Well, no joy....

When trying to browse to magnolia and view the cache, we trigger the same error as when a client emerges an ebuild. Same messages in log and killed the http-replicator process.

Next, I re-emerged python (from source) on the broken magnolia host with no change in behavior during client emerges.

Next, I stopped distccd on magnolia and tried emerging again. Same results.

At this point I have a proposal. Unless you still have some ideas you want to follow up on (in which case I'll go at it for as long as you want...) I suggest we set this aside for the time being.

As I'm reorganizing the distribution of services and applications on my network, it's looking more and more like it will be desirable to completely rebuild magnolia. Since we're both leaning to a lower-level cause of the emerge failures (that is, a systemic problem) I think it might make sense to see if this problem persists after the system rebuild. With any luck, I should be able to swap in a new disk and expedite this for this weekend.

I really do appreciate all of you help and suggestions. If you want to chase this a little further, let me know, else I'll follow the above plan and let you know of the results...

Regards,
-John

flybynite · l33t Joined: 06 Dec 2002 Posts: 620

jopalm · Posted: Sun Jan 30, 2005 5:40 am Post subject:

Well I have good news for a change...

I was able to swap in a new hard drive and get the basic system reinstalled on magnolia (the recalcitrant server) this evening. One of the first services I activated was http-replicator so that I could build up the new system from my existing cache.

It's working like a charm. I've synced my other boxes and they're all drawing against the cache on magnolia with no problem. I'm also able to fire up a browser and view the cache now.

Only hitch *might* be that I'm currently updating world on magnolia. We'll see if it survives that and still functions after gcc, python, etc are all updated!

Thanks again for you help!

Regards,
-John

flybynite · l33t Joined: 06 Dec 2002 Posts: 620

zsoltika · Posted: Thu Feb 10, 2005 7:09 am Post subject:

Hi 4 everyone,

Just found this topic, and I'm really interested in it, but got one question.
The story is: some of my workmates using Gentoo (I one of them currently, from the Rad Hat thing, and I'm pretty proud of it). Our sysadm and firm uses internet quota which means every single user have a limited amount of download capabilities.
So if we set this replicator thing the question is:
if the package.(tar.(gz|bz2)|tgz|rpm|.*){1} wouldn't be found on any of the mate's cache neither mine which machine would download it from outer repositories? How to configure to download from the outernet only from the local machine?
Sorry if this was a dumb one...