Gentoo Forums :: View topic - mirrorselect not outputing servers

mirrorselect not outputing servers

View unanswered posts
View posts from last 24 hours

Gentoo Forums Forum Index

Portage & Programming

View previous topic :: View next topic

Author

Message

jagdpanther
l33t
l33t

Joined: 22 Nov 2003
Posts: 764

Posted: Sun Nov 24, 2024 5:52 pm Post subject: mirrorselect not outputing servers

I wanted to update the "GENTOO_MIRRORS=" line in my /etc/portage/make.conf because of slow down-load speeds.
Here is today's execution of the same command I ran in June 2023 to get my current GENTOO_MIRRORS servers.

Code:

$ mirrorselect -R "North America" -4 -D -s 5 -b 10 -S -o
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=North America" hosts.
* Limiting test to https hosts.
* Downloading a list of mirrors...
Got 238 mirrors.
* Downloading 100k files from each mirror... [7 of 7]
No search results found. Check your filter settings and re-run mirrorselect

So I tried some less restrictive options:

Code:

$ mirrorselect -R "North America" -4 -s 5 -b 5 -S -o
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=North America" hosts.
* Limiting test to https hosts.
* Downloading a list of mirrors...
Got 238 mirrors.
Using netselect to choose the top5 hosts, in blocks of 5. 2 of 2 blocks complete.te.

ERROR: Netselect failed to return any mirrors. Try again using block mode.

$ mirrorselect -4 -s 5 -b 5 -S -o
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to https hosts.
* Downloading a list of mirrors...
Got 238 mirrors.
Using netselect to choose the top5 hosts, in blocks of 5. 15 of 15 blocks complete.te.

ERROR: Netselect failed to return any mirrors. Try again using block mode.

I was using '-b' which is that block mode right?

Ideas?

Banana
Moderator

Joined: 21 May 2004
Posts: 1770
Location: Germany

Posted: Mon Nov 25, 2024 7:31 am Post subject:

There is a stats page about the mirrors: https://mirrorstats.gentoo.org/rsync/
Maybe this can help to narrow it down.

I do not see a -S option here: https://man.linuxreviews.org/man8/mirrorselect.8.html (or maybe I'm blind)

A suggested command from the wiki page would be something like this:

Code:

mirrorselect -s3 -b10 -D -R "North America"

or maybe try -c USA instead of the -R option.
_________________
Forum Guidelines

PFL - Portage file list - find which package a file or command belongs to.
My delta-labs.org snippets do expire

Genone
Retired Dev

Joined: 14 Mar 2003
Posts: 9614
Location: beyond the rim

Posted: Mon Nov 25, 2024 11:14 am Post subject:

Increase the verbosity level (-d option) to see what is going on, in particular anything netselect might print to stderr.

freke
Veteran
Veteran

Joined: 23 Jan 2003
Posts: 1032
Location: Somewhere in Denmark

Posted: Mon Nov 25, 2024 6:06 pm Post subject:

Banana wrote:

Code:

mirrorselect -s3 -b10 -D -R "North America"

or maybe try -c USA instead of the -R option.

-S is https-mirrors only

Code:

ns ~ # mirrorselect
Usage: mirrorselect [options]

examples:

automatic:
# mirrorselect -s5
# mirrorselect -s3 -b10 -o >> /mnt/gentoo/etc/portage/make.conf
# mirrorselect -D -s4

interactive:
# mirrorselect -i -r

Options:
--version show program's version number and exit
-h, --help show this help message and exit

Main modes:
-a, --all_mirrors This will present a list of all filtered search
results to make it possible to select mirrors you wish
to use. For the -r, --rsync option, it will select
the rotation server only. As multiple rsync URL's are
not supported.
-D, --deep Deep mode. This is used to give a more accurate speed
test. It will download a 100k file from each server.
Because of this you should only use this option if you
have a good connection.
-i, --interactive Interactive Mode, this will present a list to make it
possible to select mirrors you wish to use.

Server type selection (choose at most one):
-c COUNTRY, -country COUNTRY
only use mirrors from the specified country NOTE:
Names with a space must be quoted eg.: -c 'South
Korea'
-F, --ftp ftp only mode. Will not consider hosts of other types.
-H, --http http only mode. Will not consider hosts of other types
-S, --https https only mode. Will not consider hosts of other
types
-r, --rsync rsync mode. Allows you to interactively select your
rsync mirror. Requires -i or -a to be used.
-R REGION, -region REGION
only use mirrors from the specified region NOTE: Names
with a space must be quoted eg.: -R 'North America'
-4, --ipv4 only use IPv4
-6, --ipv6 only use IPv6

Other options:
-b BLOCKSIZE, -blocksize BLOCKSIZE
This is to be used in automatic mode and will split
the hosts into blocks of BLOCKSIZE for use with
netselect. This is required for certain routers which
block 40+ requests at any given time. Recommended
parameters to pass are: -s 3 -b 10
-d VERBOSITY, -debug VERBOSITY
debug mode, pass in the debug level [1-9]
-f FILE, -file FILE
An alternate file to download for deep testing. Please
choose the file carefully as to not abuse the system
by selecting an overly large size file. You must also
use the -m, --md5 option.
-m MD5, --md5=MD5 An alternate file md5sum value used to compare the
downloaded file against for deep testing.
-o, --output Output Only Mode, this is especially useful when being
used during installation, to redirect output to a file
other than /etc/portage/make.conf
-P PROXY, -proxy PROXY
Proxy server to use if not the default proxy in the
environment
-q, --quiet Quiet mode
-s SERVERS, -servers SERVERS
Specify Number of servers for Automatic Mode to
select. this is only valid for download mirrors. If
this is not specified, a default of 1 is used.
-t TIMEOUT, -timeout TIMEOUT
Timeout for deep mode. Defaults to 10 seconds.
-e EXCLUDE, -exclude EXCLUDE
Exclude host from mirrors list.

For me adding -D seems to result in

Code:

No search results found. Check your filter settings and re-run mirrorselect

Haven't done any real testing nor debugging (-d) to follow up on it though.

Code:

ns ~ # mirrorselect -R 'Europe' -D -s 5 -b 5 -S -o
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=Europe" hosts.
* Limiting test to https hosts.
* Downloading a list of mirrors...
Got 238 mirrors.
* Downloading 100k files from each mirror... [41 of 41]
No search results found. Check your filter settings and re-run mirrorselect
ns ~ # mirrorselect -R 'Europe' -s 5 -b 5 -S -o
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=Europe" hosts.
* Limiting test to https hosts.
* Downloading a list of mirrors...
Got 238 mirrors.
Using netselect to choose the top5 hosts, in blocks of 5. 9 of 9 blocks complete.te.

GENTOO_MIRRORS="https://135.181.111.198/gentoo/gentoo-distfiles/ \
https://130.236.254.253/gentoo/ \
https://mirrors.ircam.fr/pub/gentoo-distfiles/ \
https://129.102.1.37/pub/gentoo-distfiles/ \
https://130.236.254.251/gentoo/"

Also it ends quite a bit quicker with the -D option, so I wonder if it really works...

Code:

ns ~ # time mirrorselect -D -6 -R 'Europe' -s5 -b5 -S -o
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=Europe" hosts.
* Limiting test to https hosts.
* Downloading a list of mirrors...
Got 238 mirrors.
* Downloading 100k files from each mirror... [41 of 41]
No search results found. Check your filter settings and re-run mirrorselect

real 0m19.697s
user 0m1.122s
sys 0m0.175s
ns ~ # time mirrorselect -6 -R 'Europe' -s5 -b5 -S -o
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=Europe" hosts.
* Limiting test to https hosts.
* Downloading a list of mirrors...
Got 238 mirrors.
Using netselect to choose the top5 hosts, in blocks of 5. 9 of 9 blocks complete.te.

GENTOO_MIRRORS="https://mirrors.ircam.fr/pub/gentoo-distfiles/ \
https://mirror.netcologne.de/gentoo/ \
https://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/ \
https://[2a0d:f302:136:1d62::1]/gentoo/ \
https://mirror.kumi.systems/gentoo/"

real 2m40.018s
user 0m2.427s
sys 0m3.377s

Actually adding -d2 shows that the test-file it's supposed to fetch seems to be non-existant on the mirrors?

Code:

ns ~ # time mirrorselect -d2 -D -R 'Europe' -s5 -b5 -S -o
main(); config_path = /etc/portage/make.conf
get_filesystem_mirrors(): config_path = /etc/portage/make.conf
get_filesystem_mirrors(): mirrorlist = ['https://mirrors.ircam.fr/pub/gentoo-distfiles/', '\\', 'https://ftp.fau.de/gentoo', '\\', 'https://mirror.netcologne.de/gentoo/', '\\', 'https://[2a0d:f302:111:83ae::1]/gentoo/', '\\', 'https://[2a0d:f302:136:1d62::1]/gentoo/']
get_filesystem_mirrors(): ignoring non-accessible mirror = \
get_filesystem_mirrors(): ignoring non-accessible mirror = \
get_filesystem_mirrors(): ignoring non-accessible mirror = \
get_filesystem_mirrors(): ignoring non-accessible mirror = \
get_filesystem_mirrors(): fsmirrors = []
using url: https://api.gentoo.org/mirrors/distfiles.xml
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=Europe" hosts.
* Limiting test to https hosts.
getlist(): fetching https://api.gentoo.org/mirrors/distfiles.xml
* Downloading a list of mirrors...
Got 238 mirrors.
Extractor(): fetched mirrors, 41 hosts after filtering
* Downloading 100k files from each mirror... [1 of 41]
_deeptime(): maxtime is 10
deeptime(): ip's for host gentoo.mirrors.ovh.net: ['213.32.5.7']
deeptime(): testing url: https://213.32.5.7/gentoo-distfiles/distfiles/mirrorselect-test
deeptime(): connection to host gentoo.mirrors.ovh.net
returned HTTPError: HTTP Error 404: Not Found for ip 213.32.5.7
Switching back to original url
deeptime(): connection to host gentoo.mirrors.ovh.net
returned HTTPError: HTTP Error 404: Not Found for ip 213.32.5.7
Switching back to original url
deeptime(): unable to connect to host gentoo.mirrors.ovh.net
* Downloading 100k files from each mirror... [2 of 41]
_deeptime(): maxtime is 10
deeptime(): ip's for host mirror.leaseweb.com: ['37.58.58.140']
deeptime(): testing url: https://37.58.58.140/gentoo/distfiles/mirrorselect-test
deeptime(): connection to host mirror.leaseweb.com
returned HTTPError: HTTP Error 404: Not Found for ip 37.58.58.140
Switching back to original url
deeptime(): connection to host mirror.leaseweb.com
returned HTTPError: HTTP Error 404: Not Found for ip 37.58.58.140
Switching back to original url
deeptime(): unable to connect to host mirror.leaseweb.com
* Downloading 100k files from each mirror... [3 of 41]

Which is why it doesn't return any mirrors when using -D I assume.

EDIT:
Found this when I was about to bug.... https://bugs.gentoo.org/921759

jagdpanther
l33t
l33t

Joined: 22 Nov 2003
Posts: 764

Posted: Tue Nov 26, 2024 12:23 am Post subject:

Thanks for the replies.

Banana:
I may try to use https://mirrorstats.gentoo.org/rsync/ to pick sites by hand.

Here are my runs from your two suggestions. I added a '-o' because I didn't want to update /etc/portage/make.conf without seeing the update first.

Code:

$ mirrorselect -s3 -b10 -D -R "North America" -o
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=North America" hosts.
* Downloading a list of mirrors...
Got 238 mirrors.
* Downloading 100k files from each mirror... [26 of 26]
No search results found. Check your filter settings and re-run mirrorselect

Code:

$ mirrorselect -s3 -b10 -D -c USA -o
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "country=USA" hosts.
* Downloading a list of mirrors...
Got 238 mirrors.
* Downloading 100k files from each mirror... [17 of 17]
No search results found. Check your filter settings and re-run mirrorselect

Genone:

I tried my original mirrorselect command adding '-d 2'. It appears that something called deeptime() is having issues:

Code:

$ mirrorselect -R "North America" -4 -D -s 5 -b 10 -S -d 2 -o
main(); config_path = /etc/portage/make.conf
get_filesystem_mirrors(): config_path = /etc/portage/make.conf
get_filesystem_mirrors(): mirrorlist = ['https://mirrors.rit.edu/gentoo/', '\\', 'https://mirror.csclub.uwaterloo.ca/gentoo-distfiles/', '\\', 'https://mirror.servaxnet.com/gentoo/', '\\', 'https://mirror.clarkson.edu/gentoo/', '\\', 'https://gentoo.osuosl.org/']
get_filesystem_mirrors(): ignoring non-accessible mirror = \
get_filesystem_mirrors(): ignoring non-accessible mirror = \
get_filesystem_mirrors(): ignoring non-accessible mirror = \
get_filesystem_mirrors(): ignoring non-accessible mirror = \
get_filesystem_mirrors(): fsmirrors = []
using url: https://api.gentoo.org/mirrors/distfiles.xml
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=North America" hosts.
* Limiting test to https hosts.
getlist(): fetching https://api.gentoo.org/mirrors/distfiles.xml
* Downloading a list of mirrors...
Got 238 mirrors.
Extractor(): fetched mirrors, 7 hosts after filtering
* Downloading 100k files from each mirror... [1 of 7]
_deeptime(): maxtime is 10
deeptime(): ip's for host mirror.csclub.uwaterloo.ca: ['129.97.134.71']
deeptime(): testing url: https://129.97.134.71/gentoo-distfiles/distfiles/mirrorselect-test
deeptime(): connection to host mirror.csclub.uwaterloo.ca
returned HTTPError: HTTP Error 404: Not Found for ip 129.97.134.71
Switching back to original url
deeptime(): connection to host mirror.csclub.uwaterloo.ca
returned HTTPError: HTTP Error 404: Not Found for ip 129.97.134.71
Switching back to original url
deeptime(): unable to connect to host mirror.csclub.uwaterloo.ca
* Downloading 100k files from each mirror... [2 of 7]
_deeptime(): maxtime is 10
deeptime(): ip's for host mirror.reenigne.net: ['172.83.105.10']
deeptime(): testing url: https://172.83.105.10/gentoo/distfiles/mirrorselect-test
deeptime(): connection to host mirror.reenigne.net
returned HTTPError: HTTP Error 404: Not Found for ip 172.83.105.10
Switching back to original url
deeptime(): connection to host mirror.reenigne.net
returned HTTPError: HTTP Error 404: Not Found for ip 172.83.105.10
Switching back to original url
deeptime(): unable to connect to host mirror.reenigne.net
* Downloading 100k files from each mirror... [3 of 7]
_deeptime(): maxtime is 10
deeptime(): ip's for host gentoo.osuosl.org: ['64.50.233.100', '64.50.236.52', '140.211.166.134']
deeptime(): testing url: https://64.50.233.100/distfiles/mirrorselect-test
deeptime(): connection to host gentoo.osuosl.org
returned HTTPError: HTTP Error 404: Not Found for ip 64.50.233.100
Switching back to original url
deeptime(): testing url: https://64.50.236.52/distfiles/mirrorselect-test
deeptime(): connection to host gentoo.osuosl.org
returned HTTPError: HTTP Error 404: Not Found for ip 64.50.236.52
Switching back to original url
deeptime(): testing url: https://140.211.166.134/distfiles/mirrorselect-test
deeptime(): connection to host gentoo.osuosl.org
returned HTTPError: HTTP Error 404: Not Found for ip 140.211.166.134
Switching back to original url
deeptime(): connection to host gentoo.osuosl.org
returned HTTPError: HTTP Error 404: Not Found for ip 140.211.166.134
Switching back to original url
deeptime(): unable to connect to host gentoo.osuosl.org
* Downloading 100k files from each mirror... [4 of 7]
_deeptime(): maxtime is 10
deeptime(): ip's for host mirrors.mit.edu: ['18.7.29.125']
deeptime(): testing url: https://18.7.29.125/gentoo-distfiles/distfiles/mirrorselect-test
deeptime(): connection to host mirrors.mit.edu
returned HTTPError: HTTP Error 404: Not Found for ip 18.7.29.125
Switching back to original url
deeptime(): connection to host mirrors.mit.edu
returned HTTPError: HTTP Error 404: Not Found for ip 18.7.29.125
Switching back to original url
deeptime(): unable to connect to host mirrors.mit.edu
* Downloading 100k files from each mirror... [5 of 7]
_deeptime(): maxtime is 10
deeptime(): ip's for host mirrors.rit.edu: ['129.21.171.72']
deeptime(): testing url: https://129.21.171.72/gentoo/distfiles/mirrorselect-test
deeptime(): connection to host mirrors.rit.edu
returned HTTPError: HTTP Error 404: Not Found for ip 129.21.171.72
Switching back to original url
deeptime(): connection to host mirrors.rit.edu
returned HTTPError: HTTP Error 404: Not Found for ip 129.21.171.72
Switching back to original url
deeptime(): unable to connect to host mirrors.rit.edu
* Downloading 100k files from each mirror... [6 of 7]
_deeptime(): maxtime is 10
deeptime(): ip's for host mirror.clarkson.edu: ['128.153.145.19']
deeptime(): testing url: https://128.153.145.19/gentoo/distfiles/mirrorselect-test
deeptime(): connection to host mirror.clarkson.edu
returned HTTPError: HTTP Error 404: Not Found for ip 128.153.145.19
Switching back to original url
deeptime(): connection to host mirror.clarkson.edu
returned HTTPError: HTTP Error 404: Not Found for ip 128.153.145.19
Switching back to original url
deeptime(): unable to connect to host mirror.clarkson.edu
* Downloading 100k files from each mirror... [7 of 7]
_deeptime(): maxtime is 10
deeptime(): ip's for host mirror.servaxnet.com: ['162.255.226.75']
deeptime(): testing url: https://162.255.226.75/gentoo/distfiles/mirrorselect-test
deeptime(): connection to host mirror.servaxnet.com
returned HTTPError: HTTP Error 404: Not Found for ip 162.255.226.75
Switching back to original url
deeptime(): connection to host mirror.servaxnet.com
returned HTTPError: HTTP Error 404: Not Found for ip 162.255.226.75
Switching back to original url
deeptime(): unable to connect to host mirror.servaxnet.com
deeptest(): got 7 hosts, and returned []

deeptest(): final rethost []
deeptest(): final md5 failures 0 of 7
No search results found. Check your filter settings and re-run mirrorselect

Using the same -R 'Europe' as you I still get no results after a min. or two.

Code:

$ mirrorselect -R 'Europe' -D -s 5 -b 5 -S -o
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=Europe" hosts.
* Limiting test to https hosts.
* Downloading a list of mirrors...
Got 238 mirrors.
* Downloading 100k files from each mirror... [41 of 41]
No search results found. Check your filter settings and re-run mirrorselect

Without the '-D' I receive an instant error

Code:

$ mirrorselect -R 'Europe' -s 5 -b 5 -S -o
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=Europe" hosts.
* Limiting test to https hosts.
* Downloading a list of mirrors...
Got 238 mirrors.
Using netselect to choose the top5 hosts, in blocks of 5. 9 of 9 blocks complete.te.

ERROR: Netselect failed to return any mirrors. Try again using block mode.

Genone
Retired Dev

Joined: 14 Mar 2003
Posts: 9614
Location: beyond the rim

Posted: Tue Nov 26, 2024 10:55 am Post subject:

deeptime() is just the name of the function within mirrorselect that does most of the testing. As you've already figured out, the core issue is apparently that mirrorselect hasn't been updated to match the current mirror directory structure. Might be worth to ask on the bug report if there are any status updates.

Display posts from previous:

	Gentoo Forums Forum Index Portage & Programming	All times are GMT
Page 1 of 1

Jump to:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Copyright 2001-2024 Gentoo Foundation, Inc. Designed by Kyle Manna © 2003; Style derived from original subSilver theme. | Hosting by Gossamer Threads Inc. © | Powered by phpBB 2.0.23-gentoo-p11 © 2001, 2002 phpBB Group
Privacy Policy