gentoo on usb:rescue, quick-install , use-able, update-able

[nomono]

subject: gentoo on usb:rescue, quick-install , use-able, update-able

Akin to Larry the Cow getting a bit frustrated I wanted to find a solution for my user-case. Gentoo naturally.

user-case(s)

I am teaching chess to a group of adults and I wanted them to use scid, a chess database program. As it is also available for windows there should not be any problems for them using it on their laptops etc, mostly preinstalled with windows. But I ran into problems when one of my students wanted to compact a database. I tried to find out for some time what the problem was; but it looked like I needed to dive into the way windows likes to do things like permissions and so on. I stopped there and besides who knows what kind of configurations are out there.

So Linux to the rescue. The most obvious to me seemed using a live-usb of some kind on which scid is installed and my students could save their work. After the usual search-and-try; MX-linux seemed up to the task at hand. Decide in a virtualbox-install of MX-linux things about the live-usb and there is persistency; I couldn't ask for more, things looked bright.

As always the proof of the pudding is in the eating. Things worked pretty reasonable up to last update of MX-linux. It seemed Debian has made a major upgrade and MX-linux followed in their footsteps. This reminded me of my dislike for LTS-distributions; a reinstall of the live-usb's is pretty soon necessary. Another problem I encountered was that a fervent student of mine updated MX-linux on his stick frequently. As the stick can only use persistency for this task his system is becoming very sloooooow.

Gentoo-afecionados will be called upon to find all kind of solutions in all kind of linux-distributions out there; I know as this happens to me. And I don't know about you, but I need to find out everytime the way the distribution in question is doing things. If only I could convince them to use gentoo...., especially when the computer is aged; in gentoo I know how to make things light but how to do it in (pick your name for the distribution)... I cannot convince them to install gentoo as the process is for initiated and uninitiated a tedious one.

A cheap mini intel based pc of mine with mmc as its hd needed a clean install. I was testing something and made the all-too-human error to delete files as root; don't do that I know! Well the system became unbootable etc...; and of course I want gentoo put back on it. How to escape the already mentioned tedious process but keep all the benefits.

Possible solution: works for me.

Toying with all kind of things -linux and gentoo- I've read and to whom (the authors) I should do justice by naming them; I developed some ideas, some itches... Lets build a gentoo-system in virtualbox with some reasonable choices and use something akin to mkstage4 to install it on usb. With usb-3 it should work fast enough (tests that I've done has shown me that even usb-2 is workable). The mkstage4-way can be used on that cheap mini intel; and can perhaps be used to persuade others to install it in place of the to me troublesome distribution they are using. This gentoo on usb can be seen as just a disk with a different OS by my students and things are more simple; and things work according to the reasonable choices I've made. And perhaps I could make it update-able in a for them easy-ish way (not the emerge --update ..etc...@world -way).

Tried first some gentoo-derivatives (like calculate.linux, cloverOS) perhaps this could speed up the process; but yep too many choices are not the way I would make them; got some ideas though. So, build my own. Decided on systemd, xfce4, lightdm and some programs I think should be in there: tor, networkmanager, scid (of course), evince, gimp, claws mail, gftp, firefox, ssvnc, transmission, linphone, audacious, kodi, xcalendar, abiword, fbreader, wine, boabab, gparted; and some more. A predefined user-account with the name user. You need to remember when building the kernel you need to use genkernel to get the modules and initramfs; the usb wants to be usable on all kind of x86_64 machines. The cpu-architecture and cpu-flags are not set in the make.conf in /etc/portage; we need evrything as generic as possible; except for the x86_64 choice I've made.

Gentoo-usb

The system build with gentoo is running in virtualbox.

Made a directory reading stage4 in /root. Created three bash-ish scripts sh.basis, sh.boot, sh.user; in reality just a way to keep track of the tar parameters.
sh.basis
->
cd /root
tar -vcpP --use-compress-prog=pbzip2 \
--ignore-failed-read --xattrs-include='*.*' --numeric-owner \
--exclude=/home/* \
--exclude=/root/.ssh/* --exclude=/root/*/.bash_history \
--exclude=/dev/* --exclude=/media/* --exclude=/mnt/*/* \
--exclude=/proc/* --exclude=/run/* --exclude=/sys/* \
--exclude=/tmp/* --exclude=/usr/src/* --exclude=/var/lock/* \
--exclude=/var/log/* --exclude=/var/run/* --exclude=/root/basis.tar.bz2 \
--exclude=/var/tmp/* --exclude=/var/cache/* \
--exclude=/boot/* --exclude=/root/boot.tar.bz2 \
--exclude=/root/user.tar.bz2 \
-f /root/basis.tar.bz2 /*
->
sh.boot
->
cd /boot
tar -vcpP --use-compress-prog=pbzip2 \
--ignore-failed-read --xattrs-include='*.*' --numeric-owner \
-f /root/boot.tar.bz2 *
->
sh.user
->
cd /home
tar -vcpP --use-compress-prog=pbzip2 \
--ignore-failed-read --xattrs-include='*.*' --numeric-owner \
--exclude=/home/user/.bash_history --exclude=/home/user/.ssh/* \
-f /root/user.tar.bz2 user
->

You probably notice that these are based upon the mkstage4 from the repository. Run these as root in the root-directory: sh sh.user ...etc... (I like to start bash-scripts with sh <script>). This gives you three tarballs in /root. Next up add an usb-stick. I use usb-3 and select it using Devices-USB-"the stick you want to use". It is not necessary to do this within virtualbox; you could use the tarballs from your main-computer with the same receipt; you must be root though (probably).

Made a directory install in /root in which I created a file named chroot (I know sh.chroot would be better)
chroot
->
#!/bin/bash

# checks if run as root:
if ! [ "`whoami`" == "root" ]
then
echo "`basename $0`: must be root."
exit 1
fi

cp --dereference /etc/resolv.conf /mnt/gentoo/etc/
mount --types proc /proc /mnt/gentoo/proc
mount --rbind /sys /mnt/gentoo/sys
mount --make-rslave /mnt/gentoo/sys
mount --rbind /dev /mnt/gentoo/dev
mount --make-rslave /mnt/gentoo/dev

chroot /mnt/gentoo /bin/bash
->
Must be a partial copy of somebody's else's script; thanks.


.Ok, as root in /root. We wish to partition the usb-stick
lsblk
.this should tell you what needs to be partitioned; lets say sdX
parted -a optimal /dev/sdX
(parted) mklabel gpt
.I know there are mbr-only bioses out there; planning to use hybrid later on in this receipt
.after mklabel usual there is no need to delete partitions
(parted) unit mib
(parted) mkpart primary 1 3
(parted) name 1 grub
(parted) set 1 bios_grub on
.use (parted) p as often as you feel comfortable with
(parted) mkpart primary 3 131
(parted) name 2 boot
(parted) mkpart primary 131 2631
.makes a swap partition of 2500 mib; change to what you like
(parted) name 3 swap
(parted) mkpart primary 2631 -1
(parted) name 4 rootfs
(parted) set 2 boot on
(parted) p
.inspect if things are as they should be
(parted) quit
.see if evrything worked out
lsblk
.For the time being thrust hybrid mbr/gpt to satisfy mbr-only bioses
gdisk /dev/sdX
r
.recovery and transformation options
h
.make hybrid
1
.added partition
Y
.for grub
<enter>
.Hex code
Y
.presume that msdos mbr needs to be bootable
.when asked for more protection of partions choose N for the time being
w
.write to disk and exit
Y
.if you want to check this use gpdisk again

.We are now set to make the filesystems
mkfs.vfat -F 32 /dev/sdX2
.uefi needs a fat32
mkswap /dev/sdX3
mkfs.ext4 /dev/sdX4
.The filesystems should be ready now
ls /mnt
.when /mnt/gentoo doesn't exsist create it now
mkdir /mnt/gentoo
mount /dev/sdX4 /mnt/gentoo
mkdir /mnt/gentoo/boot
mount /dev/sdX2 /mnt/gentoo/boot
.checking evrything with
df -h
.current directory (/root) needs to have p.e. basis.tar.bz2
tar xvpf basis.tar.bz2 -I pbzip2 -C /mnt/gentoo
tar xvpf boot.tar.bz2 -I pbzip2 -C /mnt/gentoo/boot
tar xvpf user.tar.bz2 -I pbzip2 -C /mnt/gentoo/home
.remember the awkwardly name chroot in /root/install
sh /root/install/chroot
.to check if all is well
df -h
.should only show df on the stick
source /etc/profile
export PS1="(chroot) ${PS1}"
.I usually do not do this export; but we are citing things from our beloved handbook!
.At this moment we need to edit /etc/fstab
cd /etc
.I am a sucker for using mc, so
mc
.open fstab in mc with F4 and use CTRL-O to switch with a shell
blkid
.use the info from blkid and CTRL-O combined with mouse-copy-paste to get the UUIDs correct
.and save the corrected fstab
.we are ready to make the stick bootable
grub-install --target=i386-pc /dev/sdX
.this is the mbr-version
grub-install --target=x86_64-efi --efi-directory=/boot
.this is the uefi-version
.using them both seems to secure being bootable with and without uefi
.UUIDs are changed so
grub-mkconfig -o /boot/grub/grub.cfg
.Our usb-stick should be use-able and bootable now
exit
.probably need to use exit twice; after having used mc
.we need to have exited the chroot
umount -l /mnt/gentoo/dev{/shm,/pts,}
umount -lR /mnt/gentoo/sys
umount -R /mnt/gentoo
.probably enough unmounting; just trying not to be to sloppy
.when using virtualbox I am tempted to just shut down the virtual-machine

.When all this is done find some way to test the usb-stick. For me, i try to boot from them on a laptop.
.If all is well there are some adjustments that can be undertaken.
emerge --unmerge virtualbox-guest-additions
locale-ctl >choose locale<
hostname-ctl >choose hostname<
.when nvidia-drivers are never needed
emerge --unmerge nvidia-drivers
.When using xfce4 and you want to change the user's name
systemctl stop lightdm
alt-F4
.seems to work most of the time, at first I just tried alt-F1, -F2 etc.. till I got the terminal
.log in as root
.new-user-name is; you guessed it
.old-user-name is; in my case user
usermod -l new-user-name -m -d /home/new-user-name old-user-name
.you need to kill the processes running under the old-user
.usualy I use htop for this
.also a groupmod is needed
groupmod --new-name new-username old-username
.we need to change password(s); do as root
passwd user-name
.to change root's password
passwd
.when all is done, do a reboot and hopefully evrything works
systemctl reboot

Quick-install, rescue

I could use the gentoo-usb to boot from usb on my cheap mini pc. With the tar.bz2's copied on the stick and using the same method as described for usb, mutatis mutandis;
it seemed to work. Trying to boot without the usb didn't work, the system didn't find or recognize the mmc. It did boot from usb however.
The culprit was found. The settings of Firmware in genkernel; default it doesn't include all firmware (it would grow to big on initramfs). I made an error setting FIRMWARE variables in genkernel.conf, thinking I could just give it the name of the missing firmware file that needed to be loaded from initramfs. It seems that in FIRMWARE_FILES you need to set the directory names only, and/or filenames that are directly under /lib/firmware). Testing, failing, resolving done; it was a quick-install even with this encountered problem.
Hopefully the stick is equipped with programs to make rescue possible; before making an new install on the cheap mini pc I could mount the mmc and see if anything was worth saving.

Useable and Update-able

The fervent student I spoke of earlier seems to be able to use the stick; he is asking about update-ability though.

The question seems to be update-ability without emerge --update ..etc.. @world; this would work; but he would want something faster. We're getting dangerously close to a binary distribution here; but something can be said for updating in our usual gentoo-manner in our virtualbox-version of the system and exporting the result to the usb-sticks. A way to do this could be to make a kind op PKG-server from where updates are possible; and this is feasible. In my use-case; I tried some scenarios; this is not exactly what I am looking for. Mostly desireable would be to have a kind of update to states. That is to provide an update for the stick to a state of my virtualbox-version when I deem it possible/needed/whatever; like I do for my main gentoo system; perform an update when the sync has been some time ago, or when it seems like a good thing to do. Tried something like this along the lines with quickpkg en eix selecting after some date; could be made workeable but there are problems, for which it would be a good thing if I could remember what these problems where. Suffice to say I went another route.

Still in awe with the mkstage4-way of installing it is only natural to experiment with incremental tar. An added benefit would be that this experience could lead me to a to me more satisfactory way of making backups of my regular linux-machines. That is to say I like the way of doing things like we all experience when following the gentoo-handbook; using the commandline gives you (at least me) some kind of feeling of control and flexibility. So I would like incrematal tar to give the possibility of update to the state of an updated virtualbox-version. I came up with the following; and it is in a works-for-me stadium, but it is only tested with a one-state-update.

.Create incr.upd.snap-directory in /root
.Create incr.tar snapshots and update directory in /root/incr.upd.snap
.Create sh.basis sh.boot and sh.user in /root/incr.upd.snap/incr.tar
.sh.basis is the following
->
tar -vcpP --listed-incremental=/root/incr.upd.snap/snapshots/snapshot.basis \
--no-check-device \
--use-compress-prog=pbzip2 \
--ignore-failed-read --xattrs-include='*.*' --numeric-owner \
--exclude=/etc/fstab \
--exclude=/usr/src/* --exclude=/var/lock/* \
--exclude=/var/log/* --exclude=/var/run/* \
--exclude=/var/tmp/* --exclude=/var/cache/* \
-f /root/incr.upd.snap/snapshots/basis.tar.bz2 /bin /etc /lib /lib64 /opt /sbin
/usr /var
->
.mkstage4 is still recognisable; --no-check-device was added after the usual failure/trying/testing/resolving, it is needed because when we update we are always using different devices
.and naming the directories to be included directly, this is because incremental tar when deflating tries to restore the exact filesystem-state.
.sh.boot is the following
->
tar -vcpP --listed-incremental=/root/incr.upd.snap/snapshots/snapshot.boot \
--no-check-device \
--use-compress-prog=pbzip2 \
--ignore-failed-read --xattrs-include='*.*' --numeric-owner \
-f /root/incr.upd.snap/snapshots/boot.tar.bz2 /boot
->
.sh.user following
->
tar -vcpP --listed-incremental=/root/incr.upd.snap/snapshots/snapshot.user \
--no-check-device \
--use-compress-prog=pbzip2 \
--ignore-failed-read --xattrs-include='*.*' --numeric-owner \
--exclude=/home/user/.bash_history --exclude=/home/user/.ssh/* \
-f /root/incr.upd.snap/snapshots/user.tar.bz2 /home/user
->
.copy sh.basis sh.boot sh.user to sh.basis.parm sh.boot.parm sh.user.parm respectively
.giving here only the example how a change has to be added in the *.parm version
.sh.boot.parm following
->
tar -vcpP --listed-incremental=/root/incr.upd.snap/snapshots/snapshot.boot \
--no-check-device \
--use-compress-prog=pbzip2 \
--ignore-failed-read --xattrs-include='*.*' --numeric-owner \
-f /root/incr.upd.snap/snapshots/boot.$1.tar.bz2 /boot
->
.the only change is the $1 in -f /root/incr.upd.snap/snapshots/boot.$1.tar.bz

The way I used this was to have this directory in the original virtualbox-version.
.as root do
cd /root/incr.upd.snap/incr.tar
sh sh.basis
sh sh.boot
sh sh.user
.you should have basis,boot and user.tar.bz2 in /root/incr.upd.snap/snapshots after this
.accompanied by snapshot.basis snapshot.boot and snapshot.user
.I cloned the virtualbox-machine and updated with the usual emerge --sync and emerge --update ..etc
.In the updated machine as root do
sh sh.basis.parm 1
sh sh.boot.parm 1
sh sh.user.parm 1
.you should have basis.1.tar.bz2 boot.1.tar.bz2 user.1.tar.bz2 in /root/incr.upd.snap/snapshots

Testing the update, change to the original virtualbox-machine in a way that in the original virtualbox-machine basis.1.tar.bz2, boot.1.tar.bz2 user.1.tar.bz2 are in /root/incr.upd.snap/snapshots; if something goes wrong it's probably a good idea to clone the original virtualbox-machine another time so we can test when a problem is corrected.
.open a terminal, I've forgotten this in previous receipts; hopefully it was self-evident
.do as root
systemctl stop lightdm
.I do not know that this is necessary but we are working in a live filesystem, we should be carefull
ALT-F4
.hopefully get a terminal and log in as root
.we are testing the 1-update but I will be using $x for the $x-update following
cd /
tar xvp --listed-incremental=/dev/null -f /root/incr.upd.snap/snapshots/user.$x.tar.bz2 -I pbzip2
tar xvp --listed-incremental=/dev/null -f /root/incr.upd.snap/snapshots/boot.$x.tar.bz2 -I pbzip2
tar xvp --listed-incremental=/dev/null -f /root/incr.upd.snap/snapshots/basis.$x.tar.bz2 -I pbzip2
.as said use 1 in this example instead of $x
.on the usb-stick one of these untars stopped early; I just reissued te command and it worked well
.probable tar tells you that it made mistakes; they were not serious in my case
ldconfig
.I do not know if this is necessary; from an earlier failure it seemed like a good idea to me
grub-install --target=i386-pc /dev/sdX
grub-install --target=x86_64-efi --efi-directory=/boot
.this was necessary on the hybrid gpt/mbr usb-stick; hybrid is probably not very stable
grub-mkconfig -o /boot/grub/grub.cfg
.necessary because of the boot-update
systemctl reboot
.pray that evrything works

This seems to work for me. It is probably safer to perform the operations in a /mnt/gentoo-manner, but then we need a second linux-system; at home for me this is no problem, but it could be for those students of mine who are using windows-machines.

PS
I've written this in the hope it is usefull for someone. It would be nice if a method like this could be made safe and clean; but this is certainly not a task for me.

>>greetings<<
Harry

[lefsha]

1. I bet no one will read AND understand what is written above without a bottle of Whisky or Cognac
2. Try to be brief and leave your emotions aside.
3. For such a long text use Abstract to tell the audience what its all about and why it is here.

4. Using Gentoo for a single program is a huge overkill. People quite often misunderstand
the purpose of Gentoo.

5. Using Gentoo AND systemd is the contradiction. Or more correctly the use area is much narrower
then simply Gentoo. If you need/like systemd - use Arch.

6. Gentoo by itself has no advantage to run on USB stick.

7. Gentoo is anything but not a quick install, neither a quick update. It's not a quick run in many cases too.

8. The presented solution looks like a nail if we got a hummer.

9. Gentoo is the one of those - you don't need it if you ask whether you do.

10. Use Ubuntu.
_________________
Lefsha

[389292]

lefsha you've read the entire thing?

[lefsha]

I'm not crazy yet. I have so called walked through and was shocked.

Briefly from my understanding it's a not well formatted (politely expressed) how-to
for non existent use case.

I buy a beer to the one who explain me why "Ubuntu" could not be used instead.
_________________
Lefsha