| View previous topic :: View next topic |
| Author |
Message |
nagmat84
Apprentice
Joined: 27 Mar 2007
Posts: 262
|
 Posted: Wed Apr 03, 2024 4:46 pm Post subject: How to create a custom repo for distribution to clients? |
 |
|
I have a custom repository with some ebuilds (mostly "meta" ebuilds which pull in upstream ebuilds from the Gentoo repo). As I already have a build server which creates binary packages for some clients, I want to host that repository on the build server, too, and then let the clients sync it.
I followed https://wiki.gentoo.org/wiki/Creating_an_ebuild_repository and https://wiki.gentoo.org/wiki/Handbook:AMD64/Portage/CustomTree#Creating_a_custom_ebuild_repository. On the build server everything is clear. In particular, I have
/var/db/repos/my-repo/metadata/layout.conf | Code: | masters = gentoo
auto-sync = false
thin-manifests = true
sign-manifests = false |
and
/etc/portage/repos.conf/my-repo.conf | Code: | [my-repo]
location = /var/db/repos/my-repo |
But how do I make that repo available to the clients and how do I set up the clients? Preferably, I would like to use Rsync (or GIT) over SSH as the build server already runs an SSH daemon. I would prefer not to setup a separate Rsync daemon in addition to SSH as my build server is connected to the public Internet.
Is it sufficient to simply add a corresponding /etc/portage/repos.conf/my-repo.conf on each client?
How do the settings in the file /etc/portage/repos.conf/my-repo.conf and in /var/db/repos/my-repo/metadata/layout.conf relate with each other? In /var/db/repos/my-repo/metadata/layout.conf the setting auto-sync is set to false, but on the clients /etc/portage/repos.conf/my-repo.conf mudt contain settings for synchronization. |
|
| Back to top |
|
 |
grknight
Retired Dev
Joined: 20 Feb 2015
Posts: 1919
|
| Posted: Wed Apr 03, 2024 4:59 pm Post subject: |
|
|
Basically, you have it ready.
Don't put auto-sync in layout.conf though. That's up to the clients.
On the clients, just put a sync-uri = ssh:// like in man git-clone and a sync-type=git and they will pull anything from git over ssh. If you want to control which user the repo is sync'd as, use the sync-user key in the repos.conf entry (for example easier key management).
rsync is also possible in a similar manner using a correct uri and type. |
|
| Back to top |
|
|
nagmat84
Apprentice
Joined: 27 Mar 2007
Posts: 262
|
| Posted: Wed Apr 03, 2024 5:07 pm Post subject: |
|
|
| Quote: | | Don't put auto-sync in layout.conf though. That's up to the clients. |
Meanwhile, I have figured out that part, too. This seems to be an error in https://wiki.gentoo.org/wiki/Handbook:AMD64/Portage/CustomTree#Creating_a_custom_ebuild_repository which shows the key auto-sync as part of layout.conf.
Is there also a way to setup rsync over SSH without using a separate rsync daemon on the build server?
The last question also specifically has the main Gentoo repo in mind. I would like use my build server as a "private" Gentoo mirror for all my clients, i.e. the build sever shall sync its copy of the Gentoo repo from the official mirrors and my clients should only sync with my build server. This way I want to avoid that my clients have a newer version of the Gentoo repo than the build server and then accidentally build a newer version of some package themselves only because the build server has not yet created a binary package, because the build server is falling behind. |
|
| Back to top |
|
|
grknight
Retired Dev
Joined: 20 Feb 2015
Posts: 1919
|
| Posted: Wed Apr 03, 2024 5:16 pm Post subject: |
|
|
| nagmat84 wrote: | | Is there also a way to setup rsync over SSH without using a separate rsync daemon on the build server? |
The default of modern rsync is to use ssh. One can do sync-uri=ssh:// (followed by relevant details) to force it with a sync-type=rsync. |
|
| Back to top |
|
|
nagmat84
Apprentice
Joined: 27 Mar 2007
Posts: 262
|
| Posted: Wed Apr 03, 2024 5:24 pm Post subject: |
|
|
Hopefully the last question: How do I configure /etc/portage/repos.conf/something.conf, if a) the SSH daemon is not listening on port 22 but on another port and b) requires RSA authentication?
If I manually call rsync, I use rsync -e 'ssh -p 222'. But how do I do that with repos.conf?
Sorry, for all that questions, but there seems to be no complete documentation on repos.conf. Evenonly seem to contain examples, not a full documentation and man repos.conf didn't bring up anything. |
|
| Back to top |
|
|
grknight
Retired Dev
Joined: 20 Feb 2015
Posts: 1919
|
| Posted: Wed Apr 03, 2024 5:28 pm Post subject: |
|
|
| sync-uri=ssh://[USER@]HOST[:PORT]/PATH should work. If not, use the user's ~/.ssh/config or global /etc/ssh/ssh_config file to specify an option for a host. |
|
| Back to top |
|
|
nagmat84
Apprentice
Joined: 27 Mar 2007
Posts: 262
|
| Posted: Wed Apr 03, 2024 6:11 pm Post subject: |
|
|
| Quote: | | sync-uri=ssh://[USER@]HOST[:PORT]/PATH should work. |
Unfortunately, the syntax with an explicit port does not work. So I created a global /etc/ssh/ssh_config.d/01_my-build-server.my-domain.tld with the necessary port setting. After that emerge --sync on my client was able to connect to the build server, but failed with | Code: | >>> Starting rsync with ssh://my-build-server.my-domain.tld:/var/db/repos/gentoo/...
>>> Checking server timestamp ...
ERROR: The remote path must start with a module name not a /
rsync error: error starting client-server protocol (code 5) at main.c(1859) [Receiver=3.2.7] |
The message part on "module name" suggests that Portage does not support Rsync over SSH, but requires a proper Rsync daemon on the build server side. |
|
| Back to top |
|
|
grknight
Retired Dev
Joined: 20 Feb 2015
Posts: 1919
|
| Posted: Wed Apr 03, 2024 6:30 pm Post subject: |
|
|
| nagmat84 wrote: | | Quote: | | sync-uri=ssh://[USER@]HOST[:PORT]/PATH should work. |
Unfortunately, the syntax with an explicit port does not work. So I created a global /etc/ssh/ssh_config.d/01_my-build-server.my-domain.tld with the necessary port setting. After that emerge --sync on my client was able to connect to the build server, but failed with | Code: | >>> Starting rsync with ssh://my-build-server.my-domain.tld:/var/db/repos/gentoo/...
>>> Checking server timestamp ...
ERROR: The remote path must start with a module name not a /
rsync error: error starting client-server protocol (code 5) at main.c(1859) [Receiver=3.2.7] |
The message part on "module name" suggests that Portage does not support Rsync over SSH, but requires a proper Rsync daemon on the build server side. |
I came to these options based the file /usr/lib/python3.11/site-packages/portage/sync/modules/rsync/rsync.py which specifically recognizes ssh and extra options.
I suggest removing the : before the path.
To see what is sent, use the --debug option with --sync.
Perhaps review that file to parse how the command is constructed. I am in no way an expert in the rsync nuances. |
|
| Back to top |
|
|
nagmat84
Apprentice
Joined: 27 Mar 2007
Posts: 262
|
| Posted: Wed Apr 03, 2024 6:33 pm Post subject: |
|
|
Silly me, there were still to errors in my configuration files. /etc/portage/repos.conf/gentoo.conf must be | Code: | | sync-uri = ssh://portage@server.mhnnet.de/var/db/repos/gentoo/ |
. I forgot to remove the trailing ':' after the host name which has previously separated the port number.
/etc/ssh/ssh_config.d/01_my-build-server.my-domain.tld must also include the IP addresses such that the non-standard port is also used if portage --sync connects to the SSH server by IP address (for some reason Portage seems to cache the IP address). | Code: | Host my-build-server.my-domain.tld <IPv4> [<IPv6>]
Hostname my-build-server.my-domain.tld
Port <my-non-standard-port> |
|
|
| Back to top |
|
|
Plus!
n00b
Joined: 04 Mar 2008
Posts: 15
|
|
| Back to top |
|
|
|
Portage & Programming
