Slow package fetching

diizzy said:
… https://www.freshports.org/ports-mgmt/fastest_pkg/
Click to expand...

Brighton and Hove, UK.

Today, with a wired connection (TP-Link HomePlug AV to a router in the same room):

Code: 
% fastest_pkg -v
pkg0.tuk.freebsd.org: 53%
(28, 'Operation timed out after 5007 milliseconds with 3578793 out of 6686116 bytes received')

pkg0.tuk.freebsd.org: 714.7 kB/s
pkg0.nyi.freebsd.org: 77%
(28, 'Operation timed out after 5000 milliseconds with 5159977 out of 6686116 bytes received')

pkg0.nyi.freebsd.org: 1.0 MB/s
pkg0.isc.freebsd.org: 0%
(28, 'Failed to connect to pkg0.isc.freebsd.org port 80 after 2722 ms: Operation timed out')
pkg0.isc.freebsd.org: 0.0 B/s
pkg0.bme.freebsd.org: 100%
pkg0.bme.freebsd.org: 6.9 MB/s
pkg0.pkt.freebsd.org: 100%
pkg0.pkt.freebsd.org: 6.7 MB/s

Fastest:
pkg0.bme.freebsd.org: 6.9 MB/s

…

– and wireless (iwn(4)) to the same router:

Code: 
% date
Sun 13 Feb 2022 15:41:19 GMT
% fastest_pkg -v
pkg0.tuk.freebsd.org: 64%
(28, 'Operation timed out after 5000 milliseconds with 4280857 out of 6686116 bytes received')

pkg0.tuk.freebsd.org: 856.3 kB/s
pkg0.nyi.freebsd.org: 100%
pkg0.nyi.freebsd.org: 3.5 MB/s
pkg0.isc.freebsd.org: 0%
(28, 'Failed to connect to pkg0.isc.freebsd.org port 80 after 2705 ms: Operation timed out')
pkg0.isc.freebsd.org: 0.0 B/s
pkg0.bme.freebsd.org: 100%
pkg0.bme.freebsd.org: 8.1 MB/s
pkg0.pkt.freebsd.org: 100%
pkg0.pkt.freebsd.org: 7.5 MB/s

Fastest:
pkg0.bme.freebsd.org: 8.1 MB/s

…

% fastest_pkg -v
pkg0.tuk.freebsd.org: 18%
(28, 'Operation timed out after 5001 milliseconds with 1249857 out of 6686116 bytes received')

pkg0.tuk.freebsd.org: 249.9 kB/s
pkg0.nyi.freebsd.org: 100%
pkg0.nyi.freebsd.org: 2.9 MB/s
pkg0.isc.freebsd.org: 0%
(28, 'Failed to connect to pkg0.isc.freebsd.org port 80 after 2713 ms: Operation timed out')
pkg0.isc.freebsd.org: 0.0 B/s
pkg0.bme.freebsd.org: 100%
pkg0.bme.freebsd.org: 8.1 MB/s
pkg0.pkt.freebsd.org: 100%
pkg0.pkt.freebsd.org: 7.8 MB/s

Fastest:
pkg0.bme.freebsd.org: 8.1 MB/s

…

After reverting to wired and failing to regain TCP/IPv6 routing:

Code: 
% ping -6 freshports.org
ping: UDP connect: No route to host
% date ; fastest_pkg -v
Sun 13 Feb 2022 16:19:15 GMT
pkg0.tuk.freebsd.org: 18%
(28, 'Operation timed out after 5000 milliseconds with 1220457 out of 6686116 bytes received')

pkg0.tuk.freebsd.org: 244.1 kB/s
pkg0.nyi.freebsd.org: 32%
(28, 'Operation timed out after 5000 milliseconds with 2179457 out of 6686116 bytes received')

pkg0.nyi.freebsd.org: 435.9 kB/s
pkg0.isc.freebsd.org: 0%
(28, 'Failed to connect to pkg0.isc.freebsd.org port 80 after 2739 ms: Operation timed out')
pkg0.isc.freebsd.org: 0.0 B/s
pkg0.bme.freebsd.org: 100%
pkg0.bme.freebsd.org: 6.2 MB/s
pkg0.pkt.freebsd.org: 100%
pkg0.pkt.freebsd.org: 6.5 MB/s

Fastest:
pkg0.pkt.freebsd.org: 6.5 MB/s

…

% date ; fastest_pkg -v
Sun 13 Feb 2022 16:21:16 GMT
pkg0.tuk.freebsd.org: 18%
(28, 'Operation timed out after 5000 milliseconds with 1240057 out of 6686116 bytes received')

pkg0.tuk.freebsd.org: 248.0 kB/s
pkg0.nyi.freebsd.org: 86%
(28, 'Operation timed out after 5000 milliseconds with 5770457 out of 6686116 bytes received')

pkg0.nyi.freebsd.org: 1.2 MB/s
pkg0.isc.freebsd.org: 0%
(28, 'Failed to connect to pkg0.isc.freebsd.org port 80 after 2706 ms: Operation timed out')
pkg0.isc.freebsd.org: 0.0 B/s
pkg0.bme.freebsd.org: 100%
pkg0.bme.freebsd.org: 6.9 MB/s
pkg0.pkt.freebsd.org: 100%
pkg0.pkt.freebsd.org: 6.4 MB/s

Fastest:
pkg0.bme.freebsd.org: 6.9 MB/s

…

% date ; fastest_pkg -v
Sun 13 Feb 2022 16:21:46 GMT
pkg0.tuk.freebsd.org: 18%
(28, 'Operation timed out after 5000 milliseconds with 1238657 out of 6686116 bytes received')

pkg0.tuk.freebsd.org: 247.8 kB/s
pkg0.nyi.freebsd.org: 100%
pkg0.nyi.freebsd.org: 1.6 MB/s
pkg0.isc.freebsd.org: 0%
(28, 'Failed to connect to pkg0.isc.freebsd.org port 80 after 2715 ms: Operation timed out')
pkg0.isc.freebsd.org: 0.0 B/s
pkg0.bme.freebsd.org: 100%
pkg0.bme.freebsd.org: 7.1 MB/s
pkg0.pkt.freebsd.org: 100%
pkg0.pkt.freebsd.org: 6.8 MB/s

Fastest:
pkg0.bme.freebsd.org: 7.1 MB/s

…

% date ; fastest_pkg -v
Sun 13 Feb 2022 16:22:15 GMT
pkg0.tuk.freebsd.org: 18%
(28, 'Operation timed out after 5000 milliseconds with 1223257 out of 6686116 bytes received')

pkg0.tuk.freebsd.org: 244.7 kB/s
pkg0.nyi.freebsd.org: 83%
(28, 'Operation timed out after 5000 milliseconds with 5603857 out of 6686116 bytes received')

pkg0.nyi.freebsd.org: 1.1 MB/s
pkg0.isc.freebsd.org: 0%
(28, 'Failed to connect to pkg0.isc.freebsd.org port 80 after 2737 ms: Operation timed out')
pkg0.isc.freebsd.org: 0.0 B/s
pkg0.bme.freebsd.org: 100%
pkg0.bme.freebsd.org: 7.0 MB/s
pkg0.pkt.freebsd.org: 100%
pkg0.pkt.freebsd.org: 6.6 MB/s

Fastest:
pkg0.bme.freebsd.org: 7.0 MB/s

…

% date ; fastest_pkg -v
Sun 13 Feb 2022 16:22:38 GMT
pkg0.tuk.freebsd.org: 18%
(28, 'Operation timed out after 5000 milliseconds with 1226057 out of 6686116 bytes received')

pkg0.tuk.freebsd.org: 245.2 kB/s
pkg0.nyi.freebsd.org: 91%
(28, 'Operation timed out after 5000 milliseconds with 6110657 out of 6686116 bytes received')

pkg0.nyi.freebsd.org: 1.2 MB/s
pkg0.isc.freebsd.org: 0%
(28, 'Failed to connect to pkg0.isc.freebsd.org port 80 after 2729 ms: Operation timed out')
pkg0.isc.freebsd.org: 0.0 B/s
pkg0.bme.freebsd.org: 100%
pkg0.bme.freebsd.org: 7.0 MB/s
pkg0.pkt.freebsd.org: 100%
pkg0.pkt.freebsd.org: 6.6 MB/s

Fastest:
pkg0.bme.freebsd.org: 7.0 MB/s

…

No conclusion, I was just curious about speeds. I shouldn't be surprised by the transatlantic variations.

forums.freebsd.org

Solved - gif(4) TCP/IPv6 routing after service netif restart

I use the {link removed} service, most recently configured to work with my wired network at home. gif(4), em(4). Re: I temporarily used Wi-Fi instead of a wired connection, to compare speeds. Then, wishing to revert to wired networking: service netif restart … and other things, including...
forums.freebsd.org forums.freebsd.org
 
Last edited:
Just changed the connection from office router to the network's one, helped. Now the speed is great. One solution for each problem - change your ISP.
 
I reworked the script referred in the comments below, to prefetch the upgrade's files from the distribution files (via the faster download.freebsd.org, instead of the per-file update.freebsd.org).

Bump the RELEASE env, get `freebsd-update -r ... upgrade` until it tries to download 7K files one by one (for some odd reason, maybe ISP-related, capped to 100KB/sec), interrupt it, then run this script (needs sudo for reading the install files)

sh: 
#!/bin/sh

# Fork of https://gist.github.com/pacohope/0b2fca7bcada4f98d7c71ab568aa0a89
# ... which was fork of https://gist.github.com/thefloweringash/8729473
#
# == Changed in this version:
#
# Instead of hashing all extracted files, only hash those that the update
# process needs.  For this, run `freebsd-update -r ...-RELEASE upgrade`, get to
# the point where patches are downloaded, but interrupt once the actual full
# file download start ("Fetching <N> files"), after applying patches.  At this
# point the INDEX-NEW and files.wanted files that we work out from are updated
# by the /usr/sbin/freebsd-update script.
#
# This results in saving download of many thousand files, which is especially
# problematic on bandwidth-constrained links.
#
# Note: after this extraction, there will likely still be missing files. So
# interrupt the upgrade a second time at the file fetch, extract this list of
# urls, and fetch them from a performant link and transfer them to the server
# somehow:
#
#      cat /var/db/freebsd-update/filelist | lam -s 'http://update1.freebsd.org/15.1-RELEASE/amd64/f/' - -s '.gz'
#
# == Previous description:
#
# /usr/sbin/freebsd-update is a clever script that downloads a lot of bsdiff
# patches and whole files when patches are not suitable. The result of
# this process is a collection of files in
# /var/db/freebsd-update/files. If the files already exist, it will
# not fetch them again.
#
# This script provides the entire contents of a given distribution in
# the hashed/gzip'd format that freebsd-update expects, and can
# replace a 4 hour fetch with a 2 minute fetch + 3 minute expand.
#
# Run this before starting the upgrade to the new version.
#
# Paco modified to fetch sources that are missing. I also mount the
# freebsd-update folder on my various VMs so that I only do this whole
# fetch-and-extract once, and then each VM can share this repository.
# I parameterised the source and the release version so that I can
# reuse this.

mkdir -p named

BASEDIR=${1:-${PWD}}
RELEASE="15.1-RELEASE"
SOURCE="http://download.freebsd.org/releases/amd64/amd64/${RELEASE}/"
PARTS="base kernel kernel-dbg lib32 ports"

NAMEDDIR="${BASEDIR}/named"
# Make it possible to override FILESDIR only.
# Useful to write to /var/db/freebsd-update/files (too many files to copy them post-fact easily)
FILESDIR="${FILESDIR:-${BASEDIR}/files}"

### --- probably don't need to change anything below here --- ###

# Make some standard directories we will need
for DIR in "${NAMEDDIR}" "${FILESDIR}"
do
    if [ ! -d "${DIR}" ]
    then
        echo "==> ${DIR} not found. Creating."
        mkdir -p "${DIR}"
    fi
done

echo "==> Extracting all parts"
for p in $PARTS; do
    FULL_PATH="${BASEDIR}/${p}.txz"
    if [ ! -r "${FULL_PATH}" ]; then
        echo -n "Missing file ${FULL_PATH}. Fetching: "
        curl -o "${FULL_PATH}" "${SOURCE}/${p}.txz"
    fi
    echo " => $p"
    #tar -C named -xf "${FULL_PATH}"
done

echo "==> Finding and writing as hashed name the files that freebsd-update expects based on /var/db/freebsd-update/{files.wanted,INDEX-NEW}"

# Note: the script
#
#   - omits INDEX-NEW entries which in 8th col point to an other file (there's
#   a row with that pointed file in first col and same hash).
#
#   - dedups based on hash, since some output files share the same hash (small
#   scripts etc).
#
cat /var/db/freebsd-update/files.wanted |
    while read f; do grep "$f" /var/db/freebsd-update/INDEX-NEW ; done |
    awk -F\| '{if ($8=="") print $1" "$7}' |
    awk '{if (prev!=$2) print $1; prev=$2; }' |
    lam -s "${NAMEDDIR}/" - |
    while read f
do
    if [ -x "/sbin/sha256" ]
    then
        # we're on FreeBSD
        HASH=$(/sbin/sha256 -q "$f")
    elif [ -x "/bin/sha256sum" ]
  then
        # May end up running this on my synology, which has a slightly different sha256sum command.
        SHASUM=$(/bin/sha256sum "$f")
        HASH="${SHASUM%% *}"
    else
        echo "Don't know how to get a sha256sum, sorry"
        exit 1
    fi

  TARGET="${FILESDIR}/${HASH}.gz"
  if [ ! -f "${TARGET}" ]; then
    echo "filelist wants ${f} as hash ${HASH}.gz"
    gzip < "$f" > "${TARGET}"
  else
    echo "filelist wants ${f} as hash ${HASH}.gz, but it is already there (no-op)"
  fi
done
 
You must log in or register to reply here.
Back
Top