Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
The really simple way to use the portage tree on squashfs ?
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Goverp
Advocate
Advocate


Joined: 07 Mar 2007
Posts: 2181

PostPosted: Wed Mar 03, 2021 12:39 pm    Post subject: Reply with quote

Updated 2021/03/14

Here's mine to grab the latest snapshot, using the "current" symlink, fetching both the current snapshot and the sha512sum.txt file from the same server, to avoid problems with pooled servers out of sync. It no longer maintains its own timestamp.

It keeps the synchronized snapshot and the current sha512sum.txt file in "/var/db/snapshot/".
It checks the validity of the sha512sum.txt file, and that the sha512sum for the snapshot agrees with that expected.
(The validity check demands what I think is an obsolete key "Gentoo ebuild repository signing key (Automated Signing Key) <infrastructure@gentoo.org>". It's no longer listed amongst the signing keys, but Googling shows it was used in 2018. It's also the key used for the taballs in gentoo/snapshots.)

It's tailorable for file names, directories, and whether you want lzo or xz compressed snapshots.
It no longer writes messages to syslog.
If the snapshot was mounted before you invoke the script, it gets remounted after it runs, and it prints the timestamp from its Metadata file.
It works with dash, and so it should work with any shell worthy of note.

Enjoy.
Code:

#!/bin/sh

### Update a squashfs portage snapshot from an rsync mirror
# Version 0.2 Paul Gover 2021/3/14

set -u -e       # Catch typos and unchecked commands

# Portage definitions - basically from /etc/portage/make.conf and/or /etc/portage/repos.conf/gentoo
PORTDIR="/var/db/repos/gentoo"
SYNCURI="rsync.uk.gentoo.org"
SYNCDIR="gentoo/snapshots/squashfs"

# Defintions for what and where to store the snapshot.  Note that rsync mirrors offer a choice of .lzo and .xz compressed SHOTs.
DIR="/var/db/snapshot"
SUMS="sha512sum.txt"
TYPE="lzo.sqfs"
NAME="gentoo-current"

NDIR="$DIR.new"
ODIR="$DIR.old"

### Utility functions

# Attempt a command, but don't exit just because it failed
tryto() {
        $@ || echo "Command $* failed - continuing."
}

# Issue an error message to stderr, tidy up, then exit
die() {
        echo "Error: %s\n" "$*" >& 2
        tryto rm -r "$NDIR"
        exit 1
}

# Run a command.  If if fails, exit with an apprpriate error message
command() {
        $@ || die "Command $* failed - exiting."
}

# Swap new for current directories safely
update() {
        [ -e "$ODIR" ] && command rm -r "$ODIR"
        command mv "$DIR" "$ODIR"
        command mv "$NDIR" "$DIR"
        command rm -r "$ODIR"
}

### Mainline code

tryto mkdir "$NDIR"                     # Assume fails means NDIR is left from before.
tryto cp  -a "$DIR/*" "$NDIR"           # Assume fail means DIR is not yet populated - i.e. this is first run

command rsync --copy-links --verbose --update "$SYNCURI::$SYNCDIR/$SUMS" "::$SYNCDIR/$NAME.$TYPE" "$NDIR"
command gpg --verify "$NDIR/$SUMS"

cd "$NDIR"                              # sha512sum --check has to be run from the same relative directory used to create the checksums
command sha512sum --check --ignore-missing --status "$SUMS"

if mountpoint -dq "$PORTDIR"
then
        command umount "$PORTDIR"
        update
        command mount "$PORTDIR"
        grep -F "TIMESTAMP" "$PORTDIR/Manifest"
else
        update
fi

chgrp -R portage /var/db/snapshot

_________________
Greybeard


Last edited by Goverp on Sun Mar 14, 2021 11:36 am; edited 2 times in total
Back to top
View user's profile Send private message
Goverp
Advocate
Advocate


Joined: 07 Mar 2007
Posts: 2181

PostPosted: Thu Mar 04, 2021 1:01 pm    Post subject: Problems with gentoo rsync mirrors Reply with quote

OK, now I can see what's going wrong. It's not the scripts, it's not the files themselves, it's the rsync server pools being inconsistent.

My script stopped this morning with the message:
Code:
No newer snapshot on rsync://rsync.uk.gentoo.org/gentoo/snapshots/squashf than 20210302.  Sync aborted

which was odd, as it created that yesterday.
Thinking it was a problem with rsync, I tried wget instead. That got gentoo-20210303.lto.sqfs, but its checksum was not in sha512sum.txt, which only held entries up to 20210302. Digging shows that the UK Bytemark mirror has the files up to 3rd March, whereas the others in the UK pool only go up to 2nd March. rsyncing twice, once to get the checksums to see if there was any point in doing an update, and then separately for the gentoo-current file, depends on which server it gets each time.

Possible fixes:
(a) rsync both the sha512sum.txt and gentoo-current.lto.sqfs files (use --existing on the rsync, or use a list of filenames)
(b) use "rsync --debug=connect2", which gives the IP address used on the first sync, then use that instead of the pool name on the second sync. Hopefully the files are consistent on the same server.
(c) somehow scan the pool to find which mirror in the pool has the most up-to-date data, and again use it's IP address. Except I don't know how to do this! Can anyone help?
(d) use an rsync server nearer the gentoo infrastructure master, but that's more than a little against the spirit of mirroring servers.

Note that all this means that any sync method may get inconsistent results if the mirrors are out of date. That's obvious, but it might mean that downloading the tarball and then syncing might not do what you hoped.

An aside: my script ought to include a "gpg --verify sha512sum.txt". Interestingly, that demands what I think is an obsolete key "Gentoo ebuild repository signing key (Automated Signing Key) <infrastructure@gentoo.org>". It's no longer listed amongst the signing keys, but Googling shows it was used in 2018. It's also the key used for the taballs in gentoo/snapshots.
_________________
Greybeard
Back to top
View user's profile Send private message
user
Apprentice
Apprentice


Joined: 08 Feb 2004
Posts: 211

PostPosted: Thu Mar 04, 2021 1:17 pm    Post subject: Reply with quote

cron job like
- download if changed
- verify gpg checksum file
- verify checksum
- umount
- replace squashfs file
- mount
Code:
bash -c 'umask 022 && \
cd /var/cache/portage/squashfs/ && \
wget -q --secure-protocol=PFS --timestamping https://<<your mirror>>/gentoo/snapshots/squashfs/{gentoo-current.xz.sqfs,sha512sum.txt} && \
gpg --verify sha512sum.txt && \
sha512sum --check --ignore-missing sha512sum.txt && \
sudo umount /var/db/repos/gentoo && \
cp -p gentoo-current.xz.sqfs gentoo-portage.xz.sqfs && \
sudo mount /var/db/repos/gentoo'

fstab entry
Code:

/var/cache/portage/squashfs/gentoo-portage.xz.sqfs              /var/db/repos/gentoo squashfs auto,ro,loop,nodev,noexec,nosuid 0 0
Back to top
View user's profile Send private message
Anon-E-moose
Watchman
Watchman


Joined: 23 May 2008
Posts: 6160
Location: Dallas area

PostPosted: Thu Mar 04, 2021 1:23 pm    Post subject: Reply with quote

I noticed the inconsistent results from the squashfs directory, then I swapped a dir up (snapshots) and it was still inconsistent from one day to the next.
So now, I'm back to rsync portage and then create my own squashfs (~4 seconds using 16 cores).
Note: I'm not using emerge --sync, but directly rsync'ng against a mirror and then run gemato separately (better control over the process)
_________________
UM780, 6.1 zen kernel, gcc 13, profile 17.0 (custom bare multilib), openrc, wayland
Back to top
View user's profile Send private message
Leonardo.b
Guru
Guru


Joined: 10 Oct 2020
Posts: 308

PostPosted: Mon Mar 08, 2021 1:12 am    Post subject: Reply with quote

A note about the use of cron jobs, I just realized.

I had the sync job running nightime, and the system upgrades too.
I guess it may be a problem if the cron job remounts /usr/portage while Portage is doing an upgrade.
Maybe nothing bad would happen, but I changed the sync time because I don't want to know it's not.
Back to top
View user's profile Send private message
elover
Apprentice
Apprentice


Joined: 20 Nov 2019
Posts: 170
Location: Spain

PostPosted: Sat Mar 13, 2021 2:32 pm    Post subject: Reply with quote

Goverp wrote:
And here's mine to grab the latest snapshot, ignoring the "current" symlink.
It keeps the synchronized snapshot, the current sha512sum.txt file, and the timestamp in a file, in "/var/db/snapshot/
if the latest snapshot on the rsync sha512sum.txt is no later than the timestamp, it skips further processing.
It checks the sha512sum for the snapshot agrees with that expected.
It's tailorable for file names, directories, and whether you want lzo or xz compressed snapshots.
It's set to write a few messages to syslog, but you can turn that off.
It works with dash, and so it should work with any shell worthy of note.

Enjoy.
Code:
#!/bin/sh

### Update a squashfs portage snapshot from an rsync mirror

set -u -e       # Catch typos and unchecked commands

# Portage definitions - basically from /etc/portage/make.conf and/or /etc/portage/repos.conf/gentoo
PORTDIR="/var/db/repos/gentoo"
SYNCURI="rsync://rsync.uk.gentoo.org/gentoo/snapshots/squashfs"

# Defintions for what and where to store the snapshot.  Note that rsync mirrors offer a choice of .lzo and .xz compressed SHOTs.
DIR="/var/db/snapshot"
SUMS="sha512sum.txt"
PREFIX="gentoo-"
TYPE="lzo.sqfs"
NAME="gentoo-snapshot"
STAMP="timestamp"

# If Gentoo change the filename convention from "gentoo-YYYYMMDD.foo", we're SCREWED
PATTERN="${PREFIX}[0-9]{8}[.]$TYPE"     # Strictly we should escape any .'s in TYPE, but why bother?

# Set non-null to write messages to syslog
LOG="log"

NDIR="$DIR.new"
ODIR="$DIR.old"

### utility functions

# Issue and log a message; first parameter is log facility.level
say () {
        [ "$LOG" ] && logger --priority "user.info" "squashsync: $*"
        printf "%s\n" "$*"
}

warn() {
        [ "$LOG" ] && logger --priority "user.warn" "squashsync: $*"
        printf "Warning: %s\n" "$*"

}

# Attempt a command, but don't exit just because it failed
tryto() {
        $@ || warn "Command $* failed - continuing."
}

# Issue an error message to stderr, tidy up, then exit
die() {
        [ "$LOG" ] && logger --priority "user.err" "$*"
        printf "Error: %s\n" "$*" >& 2
        tryto rm -r "$NDIR"
        exit 1
}

# Run a command.  If if fails, exit with an apprpriate error message
command() {
        $@ || die "Command $* failed - exiting."
}

# Application functions

# Print the checksum and date for the most recently dated record from stdin (an sha512sum.txt file)
# The parameter is the date of the previous snapshot already downloaded
# If no records are later than that, prints a null string
latestsum() {
        local line date lline ldate
        lline=""
        ldate="$1"

        while read -r line
        do
                # Keep only the latest date and line
                date="${line#*$PREFIX}"
                date="${date%%.*}"
                if [ "$date" -gt "$ldate" ]
                then
                        ldate="$date"
                        lline="$line"
                fi
        done

        printf "%s" "$lline"
}

# Swap new for current directories safely
update() {
        [ -e "$ODIR" ] && command rm -r "$ODIR"
        command mv "$DIR" "$ODIR"
        command mv "$NDIR" "$DIR"
        command rm -r "$ODIR"
}

### Mainline code

say "Starting portage tree snapshot update with $SYNCURI"

# Get the date of the current snapshot from our timestamp file (if any, otherwise use the epoch start).
cdate=$(cat "$DIR/$STAMP")
[ "$cdate" ] || cdate="19700101"

# Sync a new SUMS file into the NDIR.  Start with a copy, so the original remains if thing go wrong
tryto mkdir "$NDIR"     # Assume fails means NDIR is left from before.
tryto cp  "$DIR/$SUMS" "$NDIR/$SUMS"    # Assume fail means DIR is not yet populated - i.e. this is first run
command rsync --verbose "$SYNCURI/$SUMS" "$NDIR/$SUMS"

# Ignore the line for the new "current" snapshot; it tends to be out of date and hence with the wrong checksum
# Instead, grep SUMS for the right type and select the line with the latest date after the current snapshot

line=$(grep -E -o "[[:xdigit:]]+[ *]+$PATTERN" "$NDIR/$SUMS" | latestsum $cdate)

[ "$line" ] || die "No newer snapshot on $SYNCURI than $cdate.  Sync aborted"

nsum="${line%%[ *]*}"
nfile="${line##*[ *]}"
ndate="${line#*$PREFIX}"
ndate="${ndate%%.*}"

# OK, according to SUMS there's a later snapshot.  Sync a new copy into NDIR.
target="$NAME.$TYPE"
tryto cp "$DIR/$target" "$NDIR/$target"
command rsync --verbose "$SYNCURI/$nfile" "$NDIR/$target"

# Check the sum is correct
shasum=$(sha512sum "$NDIR/$target")
shasum="${shasum%%[ *]*}"

if [ -z "$shasum" ]
then die "Unable to calculate sha512sum for current snapshot $NDIR/$target"
elif [ "$shasum" != "$nsum" ]
then die "Checksum $shasum for $NDIR/$target differs from $N$SYNCURI/$nfile."
else
        printf "%s" "$ndate" > "$NDIR/$STAMP"   # Create the timestamp file

        # Replace the current snapshot with the new.  If portage tree is mounted, unmount and remount it
        if mountpoint -dq "$PORTDIR"
        then
                command umount "$PORTDIR"
                update
                command mount "$PORTDIR"
        else
                update
        fi

        say "Successful sync to portage tree snapshot dated $ndate."
fi


Hi, great job

I am a normal user who does not understand scripts. (Sorry)

I get this error when I run it:

Code:
sudo sh /home/elover/Documentos/actualizar.sh                                                                                                                              ─╯
Starting portage tree snapshot update with rsync://rsync.uk.gentoo.org/gentoo/snapshots/squashfs
cp: no se puede efectuar `stat' sobre '/var/db/snapshot/sha512sum.txt': No existe el fichero o el directorio
Warning: Command cp /var/db/snapshot/sha512sum.txt /var/db/snapshot.new/sha512sum.txt failed - continuing.

 -========== B Y T E M A R K   H O S T I N G   M I R R O R  ==========-

sha512sum.txt

sent 43 bytes  received 29,716 bytes  19,839.33 bytes/sec
total size is 29,627  speedup is 1.00
cp: no se puede efectuar `stat' sobre '/var/db/snapshot/gentoo-snapshot.lzo.sqfs': No existe el fichero o el directorio
Warning: Command cp /var/db/snapshot/gentoo-snapshot.lzo.sqfs /var/db/snapshot.new/gentoo-snapshot.lzo.sqfs failed - continuing.
rsync: link_stat "/snapshots/squashfs/gentoo-20210312.lzo.sqfs" (in gentoo) failed: No such file or directory (2)

sent 8 bytes  received 131 bytes  278.00 bytes/sec
total size is 0  speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1816) [Receiver=3.2.3]
Error: Command rsync --verbose rsync://rsync.uk.gentoo.org/gentoo/snapshots/squashfs/gentoo-20210312.lzo.sqfs /var/db/snapshot.new/gentoo-snapshot.lzo.sqfs failed - exiting.


I guess then you put it in cron, how do you configure it.

Sorry for asking so much, but my SSD has already died.
Back to top
View user's profile Send private message
Goverp
Advocate
Advocate


Joined: 07 Mar 2007
Posts: 2181

PostPosted: Sun Mar 14, 2021 9:30 am    Post subject: Reply with quote

elover,

Ah, sorry, you're being caught by the same problem described in previous posts on this thread. That script can go wrong if you are using an rsync URI for a server pool (and most of those listed on the Gentoo download pages are pooled). The script gets the sha512sum.txt file (whiich lists the available snapshots) and chooses the latest, but when it goes back to get the snapshot, it gets a different server which is out of date from the pool. (Why some of the pool's servers are a day out of date is probably down to the various mirror service providers, and may be beyond the power of the internet to get it fixed everywhere.)

I have a new version of the script that's a lot simpler, and gets the sha512sum.txt file and the current snapshot in one operation, so it will use the same server and get a consistent result. I'm on the wrong PC at the moment, but I'll post the new version this morning (UK time).
_________________
Greybeard
Back to top
View user's profile Send private message
Goverp
Advocate
Advocate


Joined: 07 Mar 2007
Posts: 2181

PostPosted: Sun Mar 14, 2021 11:35 am    Post subject: Reply with quote

OK, I've updated my script. I edited the post containing the script, rather than leave the broken one in the thread.
_________________
Greybeard
Back to top
View user's profile Send private message
geki
Advocate
Advocate


Joined: 13 May 2004
Posts: 2387
Location: Germania

PostPosted: Sun Mar 14, 2021 12:02 pm    Post subject: Reply with quote

JFYI, I recommend you to check your local university FTP site for a gentoo rsync mirror. In germany, we got plenty of them. No more out-of-date pools. Like so:

https://<your favorite FTP site>/gentoo/ or https://<your favorite FTP site>/pub/gentoo/

I use, for example, https://ftp.fau.de/gentoo/. But there are a dozen others in germany, too.

Goverp
Maybe better to add a config variable to be set by the user of the script.
_________________
hear hear
Back to top
View user's profile Send private message
Goverp
Advocate
Advocate


Joined: 07 Mar 2007
Posts: 2181

PostPosted: Sun Mar 14, 2021 2:03 pm    Post subject: Reply with quote

geki wrote:
...
Goverp
Maybe better to add a config variable to be set by the user of the script.

That's SYNCURI, unless I'm misunderstanding you.
_________________
Greybeard
Back to top
View user's profile Send private message
NiXZe
Tux's lil' helper
Tux's lil' helper


Joined: 22 May 2003
Posts: 77
Location: Uppsala Sweden

PostPosted: Wed Aug 14, 2024 7:34 pm    Post subject: Reply with quote

Goverp wrote:
Yup, trying my new script, the gentoo-current link seems to be one day behind - I wonder if the process that builds the snapshots directory sets the link BEFORE it creates the new snapshot, maybe for "safety", and then somehow fails to update it for the new one. Whatever. To add to the fun, the TIMESTAMP in the Manifest file is yet another day back. So today I have

from the sha512sum.txt file entries for .lzo files:
Code:
d94925dce716d81f025031387e63debdd5b4dbdd00de63ee033434b4a20cdc4b2ae4b8106e697aa686e29a12280b574c7c3b3bd96839e90967ec0859eb478b96  gentoo-20210227.lzo.sqfs
f54d7cfe625dadb8bca084e1b724947d06dd4666f59b13a6e838f8a7195e85326b49c3d81a3197764a1af5272418ad2131e09377ab260e1bca01012843c7fcb6  gentoo-20210228.lzo.sqfs
a30995ba0a9011fce03eb80331033549582e5669ae3c9c8f29439fb7fcc25266bd7e1ed161214c6207d918ad9038133b89d058c846390dc30c3e010a77c97e0e  gentoo-20210301.lzo.sqfs
a30995ba0a9011fce03eb80331033549582e5669ae3c9c8f29439fb7fcc25266bd7e1ed161214c6207d918ad9038133b89d058c846390dc30c3e010a77c97e0e  gentoo-current.lzo.sqfs

so the current claims to be 2021/03/01
but with my latest rsynced gentoo-current.lzo.sqfs, sha512sum returns
Code:
f54d7cfe625dadb8bca084e1b724947d06dd4666f59b13a6e838f8a7195e85326b49c3d81a3197764a1af5272418ad2131e09377ab260e1bca01012843c7fcb6  gentoo-current.lzo.sqfs

which you can see is the one for gentoo-20210228.lzo.sqfs, which the web page for ftp://rsync.uk.gentoo.org/gentoo/snapshots/squashfs/ claims was created: 2021-03-01 01:45 (which is reasonable if the snapshot creation started at midnight on 02/28), but having mounted the snapshot, the Manifest file within it says
Code:
grep TIMESTAMP /var/db/repos/gentoo/Manifest
TIMESTAMP 2021-02-27T01:38:33Z

so it seems to be a day older again!

OK, I'm not too bothered about being a day (or even 2) out, but clearly summat is going weird. I'll see if I can work out who to contact in the infrastructure world.


Trying to get this working reliably I ended up opening a Bug due to the inconsistencies between checksum and actual files.
The reason (at least currently) is CDN and caching, which gives a possible out of sync view of these files.
Details explained in the response on the bug: https://bugs.gentoo.org/937906#c1

rsync on main mirror would probably work most of the time.
_________________
What? Where? Oh! this one, it's Gentoo.
If you don't have anything constructive to say, you might want to consider staying quiet instead.
To many destructive comments, which in no way will help making the Gentoo community better.
Back to top
View user's profile Send private message
Goverp
Advocate
Advocate


Joined: 07 Mar 2007
Posts: 2181

PostPosted: Thu Aug 15, 2024 9:04 am    Post subject: Reply with quote

I stopped using the rsync method of updating my squashfs snapshot about a year ago. For some reason I couldn't figure out, rsync was no longer saving any IO compared with simply downloading the snapshot. So now I use a simple wget. It also means I changed the mirrors from which I download the snapshot - now I get it from the usual Gentoo source mirrors (which also have snapshot directories) instead of the Gentoo rsync mirrors.

That's not directly linked to the issue of the snapshots and checksum files being out of sync for the "current" snapshot, except that I've not had that problem since I started using wget! I presume that's because of using the source mirrors, rather than the use of wget. (The snapshots on the rsync mirrors appeared to be second-class citizens - about once a month I'd find them days out of date, whereas the source mirrors' snapshots have so far always been up to date. That might just be luck.) Whatever.

When I'm back on the right machine, I'll upload my current script in case anyone's interested.
_________________
Greybeard
Back to top
View user's profile Send private message
Goverp
Advocate
Advocate


Joined: 07 Mar 2007
Posts: 2181

PostPosted: Thu Aug 15, 2024 11:48 am    Post subject: Reply with quote

As promised:
Code:
#!/bin/sh

### Update a squashfs portage snapshot from an rsync mirror
# Version 0.4 Paul Gover 2024/2/1
# Changes:
# Use wget instead of rsync, allowing the use of source mirrors instead of rsync mirrors.
# In theory, rsync gave I/O reductions, but since about the start of 2023 this was no longer true.
# Trim the mirror name of protocol and stuff, to allow copy and paste from mirror lists.
# Ensure the snapshot directory and files are in portage group
# Leave the snapshot.old directory as a fallback.  Not something portage can do for you!

set -u -e       # Catch typos and unchecked commands

# Portage definitions - basically from /etc/portage/make.conf and/or /etc/portage/repos.conf/gentoo
portage="/var/db/repos/gentoo"

mirror="http://www.mirrorservice.org/sites/distfiles.gentoo.org/snapshots/squashfs/"

# Calculate the mirror server and directory
mirror="${mirror#*://}"         # strip protocol prefix, if any
mirror="${mirror%/}"            # strip trailing /, if any
mirrorDirectory="${mirror#*/}"
mirrorURI="${mirror%/"$mirrorDirectory"}"

# Defintions for what and where to store the snapshot.
currentDir="/var/db/snapshot"
checksums="sha512sum.txt"
shapshotType="lzo.sqfs"
shapshotName="gentoo-current"

newDir="$currentDir.new"
oldDir="$currentDir.old"
snapshot="$shapshotName.$shapshotType"

# Swap new for current directories safely
update() {
        [ -e "$oldDir" ] && rm -r "$oldDir"
        mv "$currentDir" "$oldDir"
        mv "$newDir" "$currentDir"
}

### Mainline code

printf 'Starting portage tree synchronization\n'

# Ensure there's a nice clean and tidy new directory
[ -d "$newDir" ] && rm -r "$newDir"
mkdir "$newDir"         # Ensure there's a new directory

cd "$newDir"    # In the new directory

wget "$mirrorURI/$mirrorDirectory/$checksums"
wget "$mirrorURI/$mirrorDirectory/$snapshot"

gpg --verify  "$checksums"
sha512sum --check --ignore-missing "$checksums"

cd

[ "$newDir/$snapshot" -nt "$currentDir/$snapshot" ] || { printf 'Newly-synchronized snapshot from %s is no newer than current.\n' "$mirrorURI" ; exit 1 ; }

if mountpoint -q "$portage"
then
        umount "$portage"
        update
        mount "$portage"
        printf 'Portage tree synchronized %s.\n' "$(grep -F "TIMESTAMP" "$portage/Manifest")"
else
        update
fi

chgrp -R portage "$currentDir"

_________________
Greybeard
Back to top
View user's profile Send private message
NiXZe
Tux's lil' helper
Tux's lil' helper


Joined: 22 May 2003
Posts: 77
Location: Uppsala Sweden

PostPosted: Thu Aug 15, 2024 7:13 pm    Post subject: Reply with quote

Goverp wrote:
I stopped using the rsync method of updating my squashfs snapshot about a year ago. For some reason I couldn't figure out, rsync was no longer saving any IO compared with simply downloading the snapshot. So now I use a simple wget. It also means I changed the mirrors from which I download the snapshot - now I get it from the usual Gentoo source mirrors (which also have snapshot directories) instead of the Gentoo rsync mirrors.

That's not directly linked to the issue of the snapshots and checksum files being out of sync for the "current" snapshot, except that I've not had that problem since I started using wget! I presume that's because of using the source mirrors, rather than the use of wget. (The snapshots on the rsync mirrors appeared to be second-class citizens - about once a month I'd find them days out of date, whereas the source mirrors' snapshots have so far always been up to date. That might just be luck.) Whatever.

When I'm back on the right machine, I'll upload my current script in case anyone's interested.


If you are using one and the same mirror that would probably be more reliable than the distfiles CDN.
But the same should also hold true most of the time for any one and same rsync mirror.

Not recommending, but after getting the explanation in the bug report the below thisbeen reliable
Code:
rsync --progress -v --copy-links rsync://gentoo.osuosl.org/gentoo/snapshots/squashfs/sha512sum.txt rsync://gentoo.osuosl.org/gentoo/snapshots/squashfs/gentoo-current.xz.sqfs  .
grep gentoo-current.xz.sqfs sha512sum.txt | sha512sum -c


I think the most important part is to always download both files at the same time, in the same connection, and not using the CDN.
_________________
What? Where? Oh! this one, it's Gentoo.
If you don't have anything constructive to say, you might want to consider staying quiet instead.
To many destructive comments, which in no way will help making the Gentoo community better.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2
Page 2 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum