Migrating away from http-replicator

[Ant P.]

This is one of the last packages on my systems that only work with python2. It's extra-painful because it also requires portage and all its deps to be built with PYTHON_TARGETS=python2_7.

I've been looking for alternatives, but would it be better to just get rid of a fetch proxy entirely and share $DISTDIR over NFS? What happens when two emerge processes try to download the same file in that setup?

[Etal]

Some time ago, had a VPS with spare disk space, and I had nginx set up to cache distfiles. Unfortunately it's long gone and I don't think I have a backup of the config.

The basic idea was that nginx would be set up as a caching proxy with gentoo mirrors set as upstream. So when I make a request to nginx, it goes out to the mirror and forwards the response while at the same time downloading it locally, so that the next time it gets a request for the same package, it just returns the local copy. I think that's basically what http-replicator does, and should be very simple to set up since nginx is essentially a proxy that can also serve web pages

(A small complication I ran into was that the proxy host would end up having 2 copies of each of its packages if it uses itself as the mirror (one in the nginx directory and one in /usr/portage/distfiles). So what I did was set DESTDIR to both directories (it actually takes a list), had nginx check /usr/portage/distfiles if it doesn't have it in its own directory before going out to the mirror, and the proxy host to connect to the mirrors (not itself) so that it downloads directly into distfiles. In retrospect, a cron job to just regularly wipe distfiles would have sufficed even if a little less efficient.)

[Tony0945]

[Havin_it]

Hope the necrobump is forgiveable, as http-replicator has now been given the last rites I feel this topic has currency again. (I'm probably late to that party, but my affected server had to go on update-freeze for a couple of months while I took some other old packages out behind the barn.)

For my part I'm not keen on the samba/nfs mount approach as it leaves the client box reliant on the mount staying up throughout the merge process (which in my case is usually a day plus). I prefer to get all the distfiles onto the client before commencing a big merge, to reduce the things that can clobber the process the moment I leave for work and waste 8 hours' build-time

My preferred approach would be similar to what @Etal did with nginx, although use apache2 in my case since the server already runs it. I imagine it's possible: I've set up various proxies though my apache before, and I know it has extensive options for such. The stuff I've done was always guided though, and had a defined endpoint: in this case it would be proxying to any host requested, which I'm not at all familiar with.

If I can get something working I'll report it here. meanwhile if anyone has input or other suggestions, I'd welcome it.

[Tony0945]

[cboldt]

I'm in the same boat. Been working with nginx because that is what I have. I'm able to get it to pass the download through. After quite a few attempts to get nginx to save what it is passing through, no joy. At this point don't have a working config to share. I do have plenty of frustration if anybody is running short.

[Ant P.]

I just gave up on http-replicator after I saw it masked and made everything share a rw NFS mount. I already have it serving ebuilds over NFS, so if a problem were to happen it'd show up through those first.

[Tony0945]

Ant P, maybe this isn't a bad way to go. My server that gets portage from the internet is up 24/7 anyway, barring reboots or maintenance problems.
Generally, I like backups in situations like this, but replacing portage is only an "emerge --sync" away, right?
Do you share /usr/portage? or one or more sub-directories?
How about using rsync with the server as source? pro's and con's? I'd be interested in your thoughts.
And VERY interested in the thoughts of ultra-prudent NeddySeagoon.

What DO others who have multiple machines do who are not using http-replicator doing? Syncing one machine on Monday. another on Tuesday, ... ?

EDIT:
BTW, I moved the ebuild directory to local and tried to re-emerge version 3. The emerge failed because of missing patches. I'm reluctant to try v 4.0 because why switch to an unsupported version?

[Anon-E-moose]

Never used http-replicator, but have set my system up this way.

mini server (24x7) nas as well as mini web server and portage (ebuilds and distfiles) and the rsync server

I have shareable portage on a tmpfs, /tmp/portage (sync daily)
I also keep a compressed tarball of the whole portage directory minus the distfiles on a mirrored raid array as well as the distfiles in their own directory on the array.

[Tony0945]

[Anon-E-moose]

I'll lay it out in more detail in the morning.
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland

[Tony0945]

Thank You!

[Ant P.]

[Anon-E-moose]

Let me see if I can more more sense of what I posted earlier

Box A - mini server - nas, web, backup including /usr/portage, distfiles and packages.
- run rsyncd
- from /etc/rsyncd.conf

[Havin_it]

For those interested, here's an apache2 vhost config for a basic caching forward http proxy.

[Ant P.]

[Tony0945]

Ran across this Gentoo wiki Local Mirror
My /etc/rsyncd.conf is already laid out as shown. Except I have no max connections line.

Is this all I have to do? Then unmerge http-repllicator?

[Anon-E-moose]

Local mirror handles the sharing of /usr/portage/<files> but not caching distfiles
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland

[Tony0945]

How about changing

[Tony0945]

OK, this seems to work for distfiles. it probably needs refinement:

On the server, named Trantor, I added this to /etc/samba/smb.conf

[Tony0945]

There must be a permissions problem. The clent could not download the taraball for gentoo-sources-5.4.X until I unmounted /usr/portage/distfiles. The server is still on gentoo-sources-4.x
All i see in /var/log/messages is:

[mmogilvi]

I recently updated a post I wrote several months ago describing several aspects of the current status of http-replicator in a fair amount of detail. See https://forums.gentoo.org/viewtopic-t-1101822-highlight-httpreplicator.html

New changes: I suspect that repcacheman will probably stop working before http-replicator itself, for reasons outlined in the above post.

----
@Anon-E-moose: Generally it is best to configure clients to use http-replicator via the "http_proxy" variable in make.conf, rather than pointing gentoo.conf's sync-uri to an incomplete proxy. This allows it to work transparently even for files that aren't downloaded at all yet.

Also see the original http-replcator forum thread: https://forums.gentoo.org/viewtopic-t-173226.html , and the instructions that version 3's ebuild logs after install (although the FETCHCOMMAND is no longer correct; see the status post linked above):

[mmogilvi]

I intend to keep running a local copy of the http-replicator 3 ebuild I saved off my personal overlay until it actually breaks more than I can live with. After that, if a better alternative doesn't pop up, I'll probably just stop trying to use any proxy server at all, even though I don't really want to increase load on other people's servers.

For version 3, @Tony0945 mentioned missing patches: The trick is to include the files from the net-proxy/http-replicator/files directory in your local overlay, not just the ebuild file. Also, technically neither version 3 nor version 4 has been supported for many years (see above linked forum post).

I don't really like any of the alternatives mentioned on this thread so far:

• Other proxy servers like nginx seem much more general and bloated than the special purpose http-replicator. They definitely don't have anything like the repcacheman script to remove duplicates from distfiles that are easily re-downloadable from the cache. Nor are they likely to have the ability to use "(export DISTFILES=/var/cache/http-replicator/ ; eclean -i distfiles)" (from app-portage/gentoolkit) to clean up the cache directory. (On the other hand, figuring out how to set up some other proxy server is the best replacement strategy I've heard of so far.)
  
• Periodically rsync'ing distfiles requires extra care in sequencing actions to really get the benefit. It may copy around files that are only needed on one machine, and/or won't use already-downloaded copies files that don't happen to be needed by any of the "parent" rsync servers (especially since the "rootmost" server is likely headless and light).
  
• Remote file systems (NFS or Samba) require always making sure it is mounted before using emerge. Or else auto-mount it at boot time, but that can be problematic if you aren't always on the same LAN with the server, or if the server isn't always up and running. Plus these tend to be really big/wide exposure points, increasing security risks.
  

[cboldt]

I have not been working on the nginx possibility, but think it may not even work in the long run. It saves the cached files named as md5sum of a concatentation that includes URI. This precludes cleaning with distclean. It also creates likelihood of duplicate entries in the nginx cache, if the same source happens to be asked from different repositories.

There make be a way to command nginx to save the cached files as their actual filename, but I wasn't going to look for that until I got the caching to work at all .

Maybe easier to learn the http-replicator code and update that to python3

[Anon-E-moose]

It would take a rewrite to move http-replicator to python3 and I suspect it's not exactly straightforward as the code hasn't been touched/updated since 2013.
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland