HOWTO:Download Cache for your LAN-Http-Replicator (ver 3.0)

[melinux]

[flybynite]

[jongeek]

[melinux]

[Onkl]

I have set up http-replicator to also serve as PORTAGE_BINHOST. Works as advertised.

However, when I use

[flybynite]

[Onkl]

Thanks a lot.

And since an "emerge -b" builds the package with 644 it is indeed strange that quickpkg is so restrictive.

[sam_i_am]

Hi,

Thanks for this great program. I'm using at home and at work without any issues except:

On one host (x86), the http-replicator doesn't start. It says "failed to start service". Log says "HttpReplicator started", but it seems to die. I enabled debug option, but no other error messages. However, on a different host (x86_64) on the same network, it works with the exact same config parameters.

Any tips on how to go about finding out why?

Other than the being x86, the failing host also runs apache on port 80 and 443, but port 8080 is clear.

Sam
_________________
Sam

[knight77]

Hi there.

The problem is still there (http-replicator-3.0-r2 stable in portage x86).

I filed a bug on it: https://bugs.gentoo.org/show_bug.cgi?id=339079

There is also another thread here about this problem, workaround included: https://forums.gentoo.org/viewtopic-t-787761.html

[Havin_it]

Thought this might be worth a mention, even though it's a bit non-standard...

I've used http-replicator for years, and I've just realised I wasn't doing so in the prescribed way. I've never even been aware of repcacheman until today, in fact. I never liked the idea of the distfiles ending up duplicated on the server, so I do things this way:

1. Set http-replicator's cache dir to the same as my $DISTDIR
   
2. Local clients use the proxy, server machine does not
   

The upshot is that when clients fetch a distfile, it goes in (or is already in, and gets served from) the cache via http-replicator; when the server fetches, the files are saved in the same dir and http-replicator will serve these for local clients as well, even though it didn't fetch them personally.

This seems to have worked fine for me over the years, though I'd welcome any drawbacks to this approach that y'all veterans might be able to identify. Hope it helps someone.

[depontius]

I had http-replicator fail to come up a week or two ago, also. The problem was that the lockfile is at /var/log/http-replicator/http-replicator.pid, and the portage user didn't have write access.

A while back Linux adopted a tmpfs-based /run directory, and /var/run became a symlink to /run. When it was in /var/run it was persistent, and the emerge process could chown /var/run/http-replicator to portage, and it would stick. Now that /run is tmpfs, it's recreated on every boot, and such things don't stick. The fix would be to do a chown against /run/http-replicator in the initscript, prior to changing to the portage userid.

The problem I have is that this doesn't universally fail, so I don't understand what's going on. A few days ago that system lost power, and when I powered it back up, I made a mental note about http-replicator, since I hadn't actually changed the initscript. Then I forgot. Upon seeing this thread, I checked that system and http-replicator is running, even though it doesn't seem that it should be. Moreover, now the lockfile is /run/http-replicator.pid, owned by root, instead of /run/http-replicator/http-replicator.pid, owned by portage.

On another system where I'm running http-replicator it's still on the deeper path. Looking into the initscripts, the locations I see for the lockfile are correct, yet they're all 3.0-r3 - I don't quite get it.
_________________
.sigs waste space and bandwidth

[khayyam]

[depontius]

Thanks for the info. I'd seen stuff on /etc/tmpfiles.d as a systemd thing, but didn't realize that OpenRC had it, as well.

What bothers me more is that right now is that I have 2 systems running http-replicator-3.0-r3, both x86. One has /run/http-replicator.pid and the other has /run/http-replicator/http-replicator.pid - in the initscript.
_________________
.sigs waste space and bandwidth

[Havin_it]

Just wondering, is it possible/supported to run multiple instances of http-replicator? I've got another caching-proxying task it might be useful for, but I want to use a different cache dir for it.

Possible?

[mbar]

Has anybody the same problem as me (~amd64)? http-replicator-4.0_alpha2

[ToeiRei]

same problem here. Trying to track it down.

Update 1:
Looks like a Namespace / Path problem to me as removing 'Params' did make it complain about the next missing module

Solved by downgrading to net-proxy/http-replicator-3.0-r3
_________________
Please stand by - The mailer daemon is busy burning your messages in hell...

[TomWij]

Please report bugs at Bugzilla, we don't keep track of bugs on the forum. Thanks.

[jpc22]

ive read the whole thread and a lot of other documentation for hours and i cant get the damn thing to work

CLIENT make.conf

http_proxy="http://192.168.1.138:8080"
PORTAGE_BINHOST="192.168.1.138:8080/usr/portage/packages" ####(tried all possible variations, ip whitout port , / /usr /all... )


CLIENT ~ # emerge -g jfsutils
--2014-12-08 18:07:57-- http://192.168.1.138:8080/usr/portage/packages/Packages
Connecting to 192.168.1.138:8080... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

--2014-12-08 18:07:58-- (try: 2) http://192.168.1.138:8080/usr/portage/packages/Packages
Connecting to 192.168.1.138:8080... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

--2014-12-08 18:08:00-- (try: 3) http://192.168.1.138:8080/usr/portage/packages/Packages
Connecting to 192.168.1.138:8080... connected.
HTTP request sent, awaiting response... No data received.
Giving up.

Fetcher exited with a failure condition.


!!! Error fetching binhost package info from '192.168.1.138:8080/usr/portage/packages'
!!! FETCHCOMMAND_192.168.1.138 failed
_________________
Revised Murphy's law: source distros can be immune to the previous in runtime, provided you endure maximum annoyance in setup.

[Havin_it]

It appears you're trying to use http-replicator as a regular HTTP server to serve up your binpkgs, which won't work: it is only a proxy, and does nothing unless you have a proper server running to serve the files from. Http-replicator simply caches a copy of the file as you download it from the proper server (or serves its cached copy if it's already present). Either way, you need to be able to fetch the file without http-replicator, or you will not be able to do so with it.

So you need to install apache, lighttpd or another http server and configure it to serve your binpkgs. And if that server is on your LAN, then there's nothing to gain by using http-replicator, and you'll just end up with two copies of every file on your binhost: one wherever your http server is serving it from, and one in the cache.

[flybynite]

I know this is an old post, but the previous post has been possibly misleading new users for too long already.

[Thistled]

This is really beginning to annoy me now.
I'm beginning to think the new layout for portage, i.e /var/db/repos/gentoo and /var/cache/distfiles
is breaking the replicator.

[Gatak]

I did an alternative setup. I exported my distfiles over samba with a password so that each client can fetch and update the central repository. This also works with the portage tree itself, if you like. This way only one client needs to run six-sync, and all others just do eix-update.

[Thistled]

[unheatedgarage]