| View previous topic :: View next topic |
| Author |
Message |
Khumarahn
Apprentice
Joined: 17 Apr 2009
Posts: 199
|
 Posted: Mon Sep 24, 2018 10:57 pm Post subject: |
 |
|
| I am interested in squashfs primarily for a system with little memory, namely Teres 1 arm laptop. It has 2G ram and 16G MMC. With these little resources, optimizing portage from 1G to 50M is important. Then, the sync should ideally be also efficient... May not have enough ram free to keep whole portage tree. |
|
| Back to top |
|
 |
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20496
|
| Posted: Tue Sep 25, 2018 3:54 am Post subject: |
|
|
tmpfs is optional. I apparently edited that out of my previous post. My squashfs images are ~59M.
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Tue Sep 25, 2018 4:45 am Post subject: |
|
|
| Khumarahn wrote: | | What is the easiest way to keep portage in squashfs, and recompress in on emerge --sync? |
You could use squashmount and call | Code: | | squashmount remount [repos mount point] |
in an /etc/portage/postsync.d hook. |
|
| Back to top |
|
|
Khumarahn
Apprentice
Joined: 17 Apr 2009
Posts: 199
|
| Posted: Tue Sep 25, 2018 4:04 pm Post subject: |
|
|
| mv wrote: | You could use squashmount and call | Code: | | squashmount remount [repos mount point] |
in an /etc/portage/postsync.d hook. |
Thanks! This worked beautifully. I get a strange error on squashmount remount:
| Code: | | mount: /usr/portage: wrong fs type, bad option, bad superblock on overlayfs, missing codepage or helper program, or other error. |
But despite the error, it works correctly. Is there an easy way to see where the error comes from?
My config:
| /etc/squashmount.pl: | #!/usr/bin/perl (this is only for editors)
@order = qw(overlay overlayfs aufs! unionfs-fuse! unionfs??# funionfs??#);
$rm_changes = $rm_workdir = $rm_readonly = 0;
my $defaults = {
COMPRESSION => 'xz',
COMPOPT_XZ => ['-Xbcj', 'arm'],
};
@mounts = (
standard_mount('portage', '/usr/portage', $defaults),
);
1; # The last executed command in this file should be a true expression |
| /etc/portage/repo.postsync.d/20-teres-squash-gentoo: | #!/bin/sh
# This file remounts the squashmount mount-point "gentoo" after each syncing
# of the "gentoo" repository.
set -u
repository_name=$1
sync_uri=$2
repository_path=$3
# Run only for repository gentoo:
[ x"$repository_name" = x'gentoo' ] || exit 0
# Run only if mount-point portage is configured
[ x"$(squashmount --quiet print-tag portage 2>/dev/null)" = x'portage' ] || exit 0
squashmount remount portage |
|
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Wed Sep 26, 2018 6:35 am Post subject: |
|
|
| Khumarahn wrote: | | I get a strange error on squashmount remount |
squashmount has its own thread and a github bugtracker, so we should perhaps not clutter the discussion here.
My guess is that either your kernel is too old (older than 3.18 requires $obsolete_overlayfs being set, see squasmount --man) or some options are not compiled in your kernel (cf. MOUNT_OVERLAY in squashmount --man) and therefore mounting overlay(fs) fails and something else (probably unionfs-fuse) is used as a fallback. With you can see what was effectively used for mounting. If you pass squashmount remount the option(s) -vvv you can see which commands/options are actually executed to understand which options overlayfs does not like. If you need further help or suspect a bug, I suggest we move discussion to either the above mentioned thread or (preferrably) to the github bug tracker. |
|
| Back to top |
|
|
Khumarahn
Apprentice
Joined: 17 Apr 2009
Posts: 199
|
| Posted: Wed Sep 26, 2018 6:44 pm Post subject: |
|
|
@mv, thank you! Running squashmount -vvv, I found that on the first attempt to mount overlayfs, the mount command does not accept the --workdir option. Probably because I am on a very old kernel with whatever custom patches by Allwinner and sunxi. By the next mount, without --workdir, it mounts and works.
I found that it is important to disable hardlinks for portage sync (`sync-allow-hardlinks = no`). Otherwise the sync is ugly, slow and takes lots of space.
I will ask more questions in the other thread or github, if I have any. So far, I am rather satisfied! |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Wed Sep 26, 2018 7:54 pm Post subject: |
|
|
| Khumarahn wrote: | | --workdir option |
This option (and the working directory) are used for the variant in newer kernels; squashmount calls this variant "overlay".
The variant in older kernels (without a working directory) is by squashmount called "overlayfs".
Since the default value of @order is qw(overlay!? overlayfs!? aufs! unionfs-fuse! unionfs! funionfs!), squashmount will first attempt 'overlay' and in case of failure 'overlayfs'. Thus, to suppress the error message, you might exchange the first two entries in your configuration file:
| /etc/squashmount.pl wrote: | | @order=qw(overlayfs!? overlay!? aufs! unionfs-fuse! unionfs! funionfs!); |
This way, first the variant without a working directory is attempted. However, you will have to remove that line once you upgrade to >=linux-3.18. |
|
| Back to top |
|
|
Khumarahn
Apprentice
Joined: 17 Apr 2009
Posts: 199
|
| Posted: Wed Sep 26, 2018 9:10 pm Post subject: |
|
|
I already put overlayfs first... I get this that error message with
| Code: | | @order = qw(overlayfs? overlay) |
|
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Wed Sep 26, 2018 9:20 pm Post subject: |
|
|
| Khumarahn wrote: | | I already put overlayfs first.. |
You are right; I misremembered. The working directory was introduced in linux-3.15. The correct recommendation would probably have been instead to set
| /etc/squashmount.pl wrote: | | $obsolete_overlayfs = 1; |
|
|
| Back to top |
|
|
Khumarahn
Apprentice
Joined: 17 Apr 2009
Posts: 199
|
| Posted: Wed Sep 26, 2018 9:22 pm Post subject: |
|
|
| /etc/squashmount.pl: | | $obsolete_overlayfs = 1; |
That worked  ))) Thanks! |
|
| Back to top |
|
|
Princess Nell
l33t
Joined: 15 Apr 2005
Posts: 926
|
| Posted: Tue Feb 23, 2021 9:21 pm Post subject: |
|
|
I went off the script posted here and aufs years ago and switched to https://www.brunsware.de/blog/portage-tree-squashfs-overlayfs. And finally, I got around to this (local portage dir, app-portage/squashfs-portage) as I got sick of installing it manually on a new machine every time.
| Code: |
# Copyright 2021 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
EAPI=7
DESCRIPTION="Use squashfs and overlays for portage tree"
HOMEPAGE="https://www.brunsware.de/blog/portage-tree-squashfs-overlayfs"
LICENSE="GPL-2"
SLOT="0"
KEYWORDS="~amd64 ~x86"
DEPEND="sys-fs/squashfs-tools"
RDEPEND="${DEPEND}"
BDEPEND=""
pkg_setup() {
mkdir "${S}"
}
src_install() {
newinitd "${FILESDIR}"/squashfs_portage-init.d squash_portage
newconfd "${FILESDIR}"/squashfs_portage-conf.d squash_portage
}
|
Take the init script from HOMEPAGE and copy it to files/squashfs_portage-init.d. Remove the 4 lines starting with SQFS_CUR, OVERLAY_DIR, RO_DIR and PORT_DIR, and instead put them into files/squashfs_portage-conf.d. ebuild/repoman etc. to taste. |
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3463
|
| Posted: Tue Aug 09, 2022 7:08 pm Post subject: |
|
|
I've just had a reason to revisit my old, tiny VPS and take care of its ever growing portage tree.
Well, I guess I might just as well add bootstrap and reset buttons to the emerge sync script I wrote a few years ago and share it.
Things to watch out for during setup:
* adjust paths:
- currently uses old /usr/portage instead of more modern /var/db/repos/gentoo, make sure TARGET matches your system settings
- I put the actual data under /media because reasons. Feel free to send it anywhere else if it bothers you (or you want to use it on a machine with removable storage devices, just to avoid conflicts)
* name the script just like the target directory, or set environmental variables before running it. Multiple repositories can be supported with symlinks.
e.g. portage for /usr/portage, gentoo for /var/db/repos/gentoo, guru for /var/dv/repos/guru etc.
* Run with command bootstrap once (reset if the repo gets corrupted for whatever reason and you need to fully resync), mount during boot, update when you want to sync portage, and squash to compress the tree (will be needed for overlays if you have them).
Fall-through switch kicks off subsequent actions automatically and allows for some retries (or squashing overlays without running emerge --sync for everything again... Perhaps doing it in a loop instead of relying on script's name would be better now )
I handle this point with cron's @reboot and @daily tags.
| Code: | # cat /opt/scripts/squash_it/portage
#! /bin/bash
#set -x
set_vars () {
OVFS="${OVFS:-/media/overlay}"
INSTANCE="${INSTANCE:-$(basename $0)}"
TARGET="${TARGET:-/usr/${INSTANCE}}"
SQDIR="${OVFS}/squash"
LOWER="${OVFS}/lower/${INSTANCE}"
UPPER="${OVFS}/upper/${INSTANCE}"
WORK="${OVFS}/work/${INSTANCE}"
TMP="${OVFS}/tmp"
SQNAME="${INSTANCE}"
SQFS="${SQDIR}/${SQNAME}"
SQCMD="/usr/bin/mksquashfs"
# excluding files does not work as reliably as I'd like, but git it already compressed, so squashing it mostly wastes CPU time and clogs RAM and disk during update. We're better off leaving it in the overlayfs.
SQPARAM="-quiet -no-progress -comp xz -Xdict-size 100% -wildcards -e '.git/'"
# This was for testing, not needed anymore, but I'm too lazy to remove it right now
MODE="eval"
}
delay (){
echo "### running command: ==> $@ <=="
sleep 5
eval "$@"
}
main (){
set_vars
case $1 in
"sudo: reset" )
"${MODE}" /bin/umount -l "${TARGET}"
"${MODE}" /bin/umount -l "${LOWER}"
"${MODE}" rm -rf "${LOWER}" "${UPPER}" "${WORK}" "${TMP}" "${SQFS}"
;&
bootstrap )
"${MODE}" mkdir -p "${LOWER}" "${UPPER}" "${WORK}" "${TMP}" "${SQDIR}"
"${MODE}" mkdir "/tmp/empty.$$"
"${MODE}" "${SQCMD}" "/tmp/empty.$$" "${SQFS}" ${SQPARAM}
"${MODE}" rmdir "/tmp/empty.$$"
echo "disk usage (squash + upper) and uncompressed data size (lower) after bootstrap"
du -hs "${SQFS}" "${LOWER}" "${UPPER}"
main mount
main update
;;
update )
"${MODE}" emerge --sync -q || exit 1
echo "disk usage (squash + upper) and uncompressed data size (lower) after sync"
du -hs "${SQFS}" "${LOWER}" "${UPPER}"
;&
squash )
"${MODE}" "${SQCMD}" "${TARGET}" "${TMP}/${SQNAME}" ${SQPARAM} || exit 2
;&
clean )
"${MODE}" /bin/umount -l "${TARGET}" || exit 3
"${MODE}" /bin/umount -l "${LOWER}" || exit 4
"${MODE}" mv "${TMP}/${SQNAME}" "${SQFS}" || exit 5
"${MODE}" find "${UPPER}/ -mindepth 1 -maxdepth 1 ! -name .git -exec rm -rf {} \+" || exit 6
;&
mount )
"${MODE}" /bin/mount -o loop "${SQFS}" "${LOWER}" || exit 7
"${MODE}" /bin/mount -t overlay overlay -o "lowerdir=${LOWER},upperdir=${UPPER},workdir=${WORK}" "${TARGET}" || exit 8
echo "disk usage (squash + upper) and uncompressed data size (lower) after remount"
du -hs "${SQFS}" "${LOWER}" "${UPPER}"
;;
*)
echo "usage: $0 < sudo: reset | bootstrap | update | squash | clean | mount >"
esac
}
main "$@"
|
The workdir of portage apparently contains 250MB of text files, uses 700-900MB of disk space (depending on FS) and compresses own to 35MB.
I know, I know, disk space is cheap, but my VPS only have 20GB of it. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Documentation, Tips & Tricks
