Share /etc/portage and world file

[Massimo B.]

Hello,

I have to maintain several Gentoo installations and like to keep them in sync as possible. That means after extending overlays, /etc/portage/package* settings or installing new packages (world file changes) I usually merge /etc/portage/ and /var/lib/portage/ between the machines. Currently I still use dirdiff to merge, later I'm going to use git and etckeeper or something like this.

This here is the discussion thread for the Howto.
The Howto is now released at 8757879.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770

[fedeliallalinea]

You could use /etc/portage/sets/ for all shared package and the world file for the specific packages for that system.
_________________
Questions are guaranteed in life; Answers aren't.

[Massimo B.]

Nice. So I could define 2 sets /etc/portage/sets/machine_shared and machine_foo, having an empty /var/lib/portage/world file and 2 sets in world_sets. That would mean for installing new packages I would need to add a file into one of the 2 files, then build @world.
Or like you said, having a shared set @machine_shared and the rest in the world file. Then later I could still move lines from the world file into my @machine_shared.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770

[NeddySeagoon]

Massimo B.

This would be a good page for the wiki, once the design has settled.

I would put global USE flags into make.conf but thats an old habit from when there was no package.use anyway.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[pingtoo]

Massimo B.

This is very clever design. I wish you have success design and share your result for everyone.

Good luck

[Goverp]

Massimo,

you might find some useful ideas (or maybe not...) in this wiki article. It's quite old, and I ought to revisit it.
_________________
Greybeard

[Genone]

You could also check into creating a custom profile for your shared settings (if all your systems currently are on the same profile). As for the distcc issue, consider https://forums.gentoo.org/viewtopic-t-1158289-start-0-postdays-0-postorder-asc-highlight-.html

[Massimo B.]

[Massimo B.]

[Massimo B.]

Finally my world file is now only getting machine specific installations. This usually are firmwares for local hardware only.
I could put also intel-related stuff like sys-firmware/intel-microcode sys-power/intel-undervolt there. But because all my current machines are Intel based I'll put that to the shared as well. It always depends on how common or different your machines are.

Almost all packages go into the shared list as all of my machines are used for daily work and development.
/etc/portage/sets/machine_shared get's all shared packages that all machines have installed.
/var/lib/portage/world_sets always contains @machine_shared.

Currently I'm running the same architecture, profile and same global amd64 (stable) keyword on all machines.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770

[Massimo B.]

Howto is now released at 8757879.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770

[Goverp]

[Massimo B.]

[Massimo B.]

Someone pointed to use ansible instead of doing that extensive file separation. Before starting to learn ansible, would that also replace the approach to have cleanly separated shared and local files??
For sure the manual merging should be far more comfortable in ansbile...
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770

[szatox]

With ansible you can either drop files from several sources into a single directory or create a single file containing multiple sections populated according to defined variables.
It depends entirely on what you consider more convenient.

I find one file more convenient than sourcing multiple files doing one task together, splitting it into parts is a hack that makes automatic processing (by package manager) easier, but also makes manual updates painful, since opening THE file and searching through it with / doesn't work anymore, and must be replaced by a long sequence of greps and vims and saves and so on.
Things that are independent of each other should go into different files though.

However, when using ansile, you don't define sets on the machine. You add them to a list of desired packages in the inventory, and have ansible install them for you.
Inventory is a persistent resource you store on the machine you use for running ansible. Ansible then uses ssh to connect to the target machine and run commands it deems necessary to bring the target host to the state declared in your inventory.

You can declare group "mta" with postfix, and group "mda" with dovecot, add your host to both, and (if you merge variables properly) ansible will ensure that postfix and dovecot are both installed your mail server. So, ansible replaces sets rather than supplements them. Unless you have a very specific need that requires a different approach.

[mv]

I do a similar thing since years with /etc/portage/sets and /etc/portage/package.* and /etc/portage/make.conf/: In my setup, these directories contain only symlinks. The symlinks point to files in directories which are common for all of my systems, but the systems differ in the symlinks: Some files contain data common to all of my systems (and for these, I have symlinks in all of my machines), some others contain data common only to some of my machines (and only the corresponding machines have symlinks), and some have also unique data; in the latter case, only one machine has the corresponding symlink (or in some rare cases contains the file directly).
(In the description, you can replace "file" by "directory" in some cases.)

[Massimo B.]

[mv]

[Massimo B.]

I see, so by symlinking all shared content into a separate directory tree, you are sure that you just can merge that whole tree.
I'm going to merge /etc/portage while just ignoring all the *.local files, while sometimes I also need to cherry-pick there.

You also seem to have some customized process to merge that shared tree via tar, git etc. I'm going to see by learning Ansible if that will make things easier. Actually the merging and tracking can all be done with git and as for /etc with permissions and stuff etc-keeper can do that as well. I always need to sync against a central point. As you have said, not all machines are always accessible. I could cross-sync via rsync/sshfs/dirdiff etc. but as number of machines are increasing it's always better to have a star topology merging against a central repository.

What I'm still not sure, if creating my own profile or creating my own overlay could improve my current approach.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770

[szatox]

Wow, the level of complexity in those directories is insane.
Looks like even multiple git branches with rerere enabled would be easier (+ history as a bonus point). Ansible would definitely be easier.
Either way, sharing configs in a network-dependent way is a bad idea, each machine absolutely needs its own copy.

Ansible works very well with git, you can have both, playbooks and inventory in a repo. With ansible making your own profile is unnecessary, you put all the custom things into the ansible inventory anyway. Overlay is a different thing, if you have custom ebuilds, by all means go for it.

[Massimo B.]

Generally speaking, if I have CXXFLAGS="${CFLAGS}" in make.conf and I temporarily call the emerge with a CFLAGS like 'CFLAGS="foo" emerge foo', then "foo" does not arrive in CXXFLAGS, why?

Question about the distcc section of 8757879:

Currently it does not work like this as CFLAGS is set but CXXFLAGS isn't:

[Massimo B.]

[mv]

[mv]