Solved How to download packages for offline installation of a *newer* version of FreeBSD?

I

Igor2022

Hello!
I have a box running FreeBSD-13, and I'd like to install FreeBSD-14 on another box, which has no internet connection. I don't have any PCs with FreeBSD-14 yet. I have no problem downloading installation image for the FreeBSD-14 base and making an installation flash stick. What I'm stuck with are the packages for the newer FreeBSD-14. I tried to download them with

pkg \
-C my_config_for_14.conf \
fetch \
-r FreeBSD14 \
-o /path/to/package/dir \
-a

but all I get is a message: "pkg: Warning: Major OS version upgrade detected. Running "pkg bootstrap -f" recommended" and no download ever happens. No matter whether I put "-a" option or a list of packages.
I haven't tried re-bootstrapping pkg to a FreeBSD-14 version as suggested in the warning message, since I cannot rule out possibility that it could break my current installation.

So the question is: How do I download all packages for a newer version of FreeBSD, given that I have no device running that newer version yet?

The contents of the files mentioned above is:

my_config_for14.conf
Code: 
PKG_DBDIR = "/home/system/distrib/FreeBSD/14.0-RELEASE/var/db/pkg";
PKG_CACHEDIR = "/home/system/distrib/FreeBSD/14.0-RELEASE/var/cache/pkg";

REPOS_DIR [
    "/home/system/distrib/FreeBSD/14.0-RELEASE/packages/repos",
]

ABI = "FreeBSD:14:amd64";

REPO_AUTOUPDATE = false;

IP_VERSION = 4

FreeBSD14.conf
Code: 
FreeBSD14:
{
  url: "pkg+https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest",
  mirror_type: "srv",
  signature_type: "fingerprints",
  fingerprints: "/usr/share/keys/pkg",
  enabled: yes
}
 
The DVD image contains quite a few packages but only has a selection of packages, it definitely doesn't contain everything. Are you looking for something specific?
 
(If OP has enough SSD space on the Internet-connected box), one way to pull this off is with a jail - like, install a 14.0-RELEASE jail on a 13.0-RELEASE host, then use the 14.0-RELEASE jail to slurp up the packages for 14.0-RELEASE...

A simpler way to do that would be to connect the second box to Internet, as well - is there a reason why that has not been tried?
 
SirDice said:
The DVD image contains quite a few packages but only has a selection of packages, it definitely doesn't contain everything.
Click to expand...
That's the problem. I've checked the DVD image, and many of desired packages were missing. The list is quite long to quote here, and of course I'm not advocating including my pet packages into the DVD distribution, my needs aren't that common...

astyle said:
A simpler way to do that would be to connect the second box to Internet, as well - is there a reason why that has not been tried?
Click to expand...
The reason is quite stupid and personal. You may read it if you must, I don't mind someone having a raised eyebrow and a laugh.
I've lost the power supply unit to my second box. So I have two equivalent Shuttle DS20U boxes and only one power supply. And my ethernet link is MAC-locked on my primary box, so even if I shutdown the primary box and power up the second box -- it won't pass MAC authorization at my ISP.

BTW, I think I've already solved my problem. I wrote a script to parse packagesite.yaml and invoke wget on each package listed there.
 
astyle said:
(If OP has enough SSD space on the Internet-connected box), one way to pull this off is with a jail - like, install a 14.0-RELEASE jail on a 13.0-RELEASE host, then use the 14.0-RELEASE jail to slurp up the packages for 14.0-RELEASE...
Click to expand...
I would certainly try this. Especially since pkg-fetch is fairly basic as a program.

However a 14-RELEASE jail is not guaranteed to work on a 13-RELEASE host (incl kernel). The other way round, running a 13-RELEASE jail on a 14-RELEASE host is likely to work.
 
Igor2022 said:
I wrote a script to parse packagesite.yaml and invoke wget on each package listed there.
Click to expand...
Works for me to fetch FreeBSD:14:amd64 packages on a 13.1-RELEASE box:
Code: 
# freebsd-version -ru
13.1-RELEASE-p3
13.1-RELEASE-p3

FreeBSD.conf
Code: 
14latest: {
  url: "pkg+http://pkg.freebsd.org/FreeBSD:14:amd64/latest",
  mirror_type: "srv",
  signature_type: "fingerprints",
  fingerprints: "/usr/share/keys/pkg",
  enabled: yes
}

pkg.conf
Code: 
PKG_DBDIR = "/usr/tmp/distrib/FreeBSD/14.0-RELEASE/var/db/pkg";
PKG_CACHEDIR = "/usr/tmp/distrib/FreeBSD/14.0-RELEASE/var/cache/pkg";

ABI = "FreeBSD:14:amd64";

Code: 
# pkg info -E pkg
pkg-1.20.4

Code: 
# pkg -C /usr/tmp/distrib/FreeBSD/14.0-RELEASE/pkg.conf fetch -r 14latest clone
pkg: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
Updating 14latest repository catalogue...
14latest repository is up to date.
All repositories are up to date.
The following packages will be fetched:

New packages to be FETCHED:
    clone: 1.0.8 (22 KiB: 100.00% of the 22 KiB to download)

Number of packages to be fetched: 1

22 KiB to be downloaded.

Proceed with fetching packages? [y/N]: y
Fetching clone-1.0.8.pkg: 100%   22 KiB  22.5kB/s    00:01

# ls -l /usr/tmp/distrib/FreeBSD/14.0-RELEASE/var/cache/pkg/
total 25
lrwxr-xr-x  1 root  wheel     26 Feb 17 01:03 clone-1.0.8.pkg -> clone-1.0.8~4b86acfa47.pkg
-rw-r--r--  1 root  wheel  22537 Feb 17 01:03 clone-1.0.8~4b86acfa47.pkg

# tar xfO clone-1.0.8~4b86acfa47.pkg +COMPACT_MANIFEST
..."FreeBSD_version":"1400097"}

Code: 
# pkg -C /usr/tmp/distrib/FreeBSD/14.0-RELEASE/pkg.conf fetch -r 14latest -a
pkg: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
Updating 14latest repository catalogue...
14latest repository is up to date.
All repositories are up to date.
    [...]
    zziplib: 0.13.72_1 (109 KiB: 0.00% of the 122 GiB to download)
    zzuf: 0.13_1 (129 KiB: 0.00% of the 122 GiB to download)

Number of packages to be fetched: 33859

The process will require 122 GiB more space.
122 GiB to be downloaded.

Proceed with fetching packages? [y/N]:
 
Igor2022 said:
I've lost the power supply unit to my second box. So I have two equivalent Shuttle DS20U boxes and only one power supply. And my ethernet link is MAC-locked on my primary box, so even if I shutdown the primary box and power up the second box -- it won't pass MAC authorization at my ISP.
Click to expand...
I can suggest some solutions...

For starters: Is it internal Power Supply Unit or external DC power adapter that was lost? I think the latter is more likely... especially if your boxes are this stuff: https://www.shuttle.eu/en/products/slim/ds20u ... Pretty recent model, can probably order a replacement...

For MAC-locked stuff: you can use a network switch to work around that. Unmanaged ones are cheaper, and they do come with their own MAC addresses. You may have to do some haggling with your ISP about that, though.

Nothing to feel bad about, that's easily resolvable technical issues, and these Forums are the right place to ask for help on that kind of topic... :)
 
astyle said:
(If OP has enough SSD space on the Internet-connected box), one way to pull this off is with a jail - like, install a 14.0-RELEASE jail on a 13.0-RELEASE host, then use the 14.0-RELEASE jail to slurp up the packages for 14.0-RELEASE...

A simpler way to do that would be to connect the second box to Internet, as well - is there a reason why that has not been tried?
Click to expand...

Is it possible to install a 14.0 jail on a 13.0 host? I've never tried since I didn't think it could be done.
 
astyle said:
they don't, actually. A brand-new, patched kernel is downloaded. See the Handbook for proof:
docs.freebsd.org

Chapter 17. Jails and Containers

Jails improve on the concept of the traditional chroot environment in several ways
docs.freebsd.org docs.freebsd.org

Even for thin jails, you have to download a base.txz, which is the kernel itself 😩 😏
Click to expand...
I'm perplexed. Maybe I'm misunderstanding what you are saying here.

In the handbook I found these statements: "Jails do not have a kernel. They run on the host’s kernel." Also, I can't tell you how many blogs, tutorials and how-tos there are which reiterate this - "jails run on the host's kernel". Many or most of those sources also state that one cannot run a jail whose version of FreeBSD is greater than that of the host.

I understand that the base.txz contains a directory named "/boot/kernel". If you look in that directory it is empty.

I just logged into a test jail I have running on a FreeBSD host which is itself a guest running in Virtualbox on a Windows 10 desktop. When I look at the /boot/kernel/ directory in that jail it is empty. I used the base.txz to create the template from which this thick jail itself was created. I'm inclined to think that the jail is not running a kernel separate from the host and I can see no evidence that it is.

So I repeated the investigation on my primary FreeBSD server box (running v13.2) which has two full-time jails operating. I logged into both of them and the /boot/kernel/ directories are both empty. If these jails are running their own kernels I can see or find them anywhere.

This thread would seem to support the notion that I could use my Windows desktop running Virtualbox to install a FreeBSD v13.2 guest and then create a jail in that guest/host using the base.txz found on the FreeBSD v14.0 .iso disc and that such a jail would fire right up and operate without issues.

Before I take the time to test such a claim I'd like to hear others say that it will work.

NB: I'm a rank amateur with all this. It took me months to figure out how to run a VNET jail inside a Virtualbox guest/host of FreeBSD v14.0. I'm still struggling to get VNET to work on an RPi4B platform.
 
Igor2022 said:
… I think I've already solved my problem. I wrote a script …
Click to expand...

You could, alternatively:
  1. configure things to use pkgbase
  2. create then mount a ZFS boot environment
  3. use pkg to install FreeBSD 14.0-RELEASE, and upgraded packages of ports, to the environment
  4. attend to *.pkgsave files (the few mentioned in the wiki, plus the three mentioned at <https://forums.freebsd.org/posts/642311>)
  5. update the ESP (assuming EFI)
  6. activate then boot the environment.
 
Bucky said:
In the handbook I found these statements: "Jails do not have a kernel. They run on the host’s kernel." Also, I can't tell you how many blogs, tutorials and how-tos there are which reiterate this - "jails run on the host's kernel"
Click to expand...
Yeah, that's a brain fart on my part. 😅

Bucky said:
Many or most of those sources also state that one cannot run a jail whose version of FreeBSD is greater than that of the host.
Click to expand...
How does that square with T-Daemon 's post #6 in this thread?

Bucky said:
This thread would seem to support the notion that I could use my Windows desktop running Virtualbox to install a FreeBSD v13.2 guest and then create a jail in that guest/host using the base.txz found on the FreeBSD v14.0 .iso disc and that such a jail would fire right up and operate without issues.
Click to expand...
I'd suggest giving enough metal (CPU cores and RAM) to VirtualBox to pull that off. And don't forget to enable hyperthreading/multithreading in BIOS of the Windows desktop. Once you have the hardware, it's technically possible to pull off the 14-jail-on-a-13-host-within-a-Vbox-on-windows scenario.

Bucky said:
NB: I'm a rank amateur with all this. It took me months to figure out how to run a VNET jail inside a Virtualbox guest/host of FreeBSD v14.0. I'm still struggling to get VNET to work on an RPi4B platform.
Click to expand...
Try pinging eternal_noob , he might know a thing or two about running FreeBSD on a Pi. One thing I can say, if you have the basic networking stack working properly on such a setup (ifconfig working, you can ping google, and the like), then VNET is just a software puzzle waiting to be solved, and it doesn't matter if you have a Pi or a Threadripper.
 
How does that square with T-Daemon 's post #6 in this thread?
Click to expand...
He is not doing it in a jail.

Once you have the hardware, it's technically possible to pull off the 14-jail-on-a-13-host-within-a-Vbox-on-windows scenario.
Click to expand...
Then you stand alone (by my extensive reading) in making that claim.

Try pinging eternal_noob , he might know a thing or two about running FreeBSD on a Pi. One thing I can say, if you have the basic networking stack working properly on such a setup (ifconfig working, you can ping google, and the like), then VNET is just a software puzzle waiting to be solved, and it doesn't matter if you have a Pi or a Threadripper.
Click to expand...
I have read much of what eternal_noob has written. I'll try rereading it.

There is something about an RPi4B that I haven't figure out yet...
 
T-Daemon said:
Works for me to fetch FreeBSD:14:amd64 packages on a 13.1-RELEASE box
Click to expand...
Thanks! I tried your config and it worked for me too. Then I've tracked down the problem with my config to this line:
Code: 
REPO_AUTOUPDATE = false;
I put it in because I desperately wanted to preserve my current installation, and it looks like I've overdone it a bit. That line prevented downloading of the new package index, and in turn, none packages were downloaded.

astyle said:
I can suggest some solutions...
Click to expand...
Yes, it's an external DC adapter. I've already asked the local distributor for a replacement, but it'll take some time (weeks maybe?) to get done.

grahamperrin said:
You could, alternatively
Click to expand...
Thank you for the suggestion! The problem is already solved in two ways. ))
 
astyle said:
How does that square with T-Daemon 's post #6 in this thread?
Click to expand...
Bucky said:
He is not doing it in a jail.
Click to expand...
That's correct, no jail, just plain system.


Fetching packages for another platform or version than the pkg(8) detected one I've had done before. Specifically fetched 13.1 i386 and 12 i386 packages on 13.1 amd64.

In addition to post #6, for those thinking of creating a local repository from fetched packages and unfamiliar with the procedur in detail, to save them unlinked with original package name (as they are linked in /var/cache/pkg), use the pkg-fetch(8) -o option. See manual for details.
 
balanga said:
Can someone clarify whether you can run a 14.0 base in a 13.2 jail?
Click to expand...
If you mean running a 14.0 base jail in a 13.2 host, then no, you can't.

Chapter 17. Jails and Containers, 17.3. Host Configuration

Chapter 17. Jails and Containers.png
 
T-Daemon said:
Works for me to fetch FreeBSD:14:amd64 packages on a 13.1-RELEASE box
Click to expand...
One minor detail, in case anyone would like to try this approach. I see no reason why downloading the packages should be done as root, so this whole process'd better be run as an unprivileged user. To do so, set this before running pkg:
Code: 
INSTALL_AS_USER=1
export INSTALL_AS_USER
Otherwise pkg will report an error: pkg: . wrong user or group ownership (expected 0/0 versus actual <your-uid>/<your-guid>).
 
Igor2022 said:
One minor detail, in case anyone would like to try this approach. I see no reason why downloading the packages should be done as root, so this whole process'd better be run as an unprivileged user. To do so, set this before running pkg:
Code: 
INSTALL_AS_USER=1
export INSTALL_AS_USER
Otherwise pkg will report an error: pkg: . wrong user or group ownership (expected 0/0 versus actual <your-uid>/<your-guid>).
Click to expand...
hmmm... 🤔 fetch as regular user is an OK idea, but enabling install as a regular user is probably a bad idea security-wise. I think pkg has enough fine-grained controls to pull this off properly - try playing with it and reading the manpages.
 
SirDice said:
I suspect pkg-fetch(8) will work from a user account if -o points to a directory the user can write to.
Click to expand...
Unfortunately, it doesn't work. That is, it does, but only the first time. Any subsequent attempt to run pkg-fetch fails:

Code: 
$ ./test.sh
uid=***(***) gid=***(***) groups=***(***)
==========> First invocation of pkg-fetch
pkg: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
pkg: . wrong user or group ownership (expected 0/0 versus actual ***/***)
Updating Local repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%    7 MiB   7.3MB/s    00:01
Processing entries:   0%
Newer FreeBSD version for package zziplib:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1400097
- running kernel: 1300139
Ignore the mismatch and continue? [y/N]: y
Processing entries: 100%
Local repository update completed. 33764 packages processed.
All repositories are up to date.
The following packages will be fetched:

New packages to be FETCHED:
        bash: 5.2.21 (2 MiB: 100.00% of the 2 MiB to download)

Number of packages to be fetched: 1

The process will require 2 MiB more space.
2 MiB to be downloaded.

Proceed with fetching packages? [y/N]: y
Fetching bash-5.2.21.pkg: 100%    2 MiB   1.8MB/s    00:01
0

==========> Second invocation of pkg-fetch
pkg: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
pkg: . wrong user or group ownership (expected 0/0 versus actual ***/***)
1

In this test pkg-fetch is invoked twice. The first time it prints a warning "wrong user or group ..." but still does the job. The second time it prints the same warning and then fails. Test script is shown below.

test.sh
Bash: 
#! /bin/sh

rm -rf /tmp/test_fetch/packages
rm -rf /tmp/test_fetch/var

mkdir -p /tmp/test_fetch/packages
mkdir -p /tmp/test_fetch/var

id
set | grep INSTALL_AS_USER

echo "==========> First invocation of pkg-fetch"
pkg \
  -C /tmp/test_fetch/test.conf \
  fetch \
  -o /tmp/test_fetch/packages \
  bash
echo $?
echo ""

echo "==========> Second invocation of pkg-fetch"
pkg \
  -C /tmp/test_fetch/test.conf \
  fetch \
  -o /tmp/test_fetch/packages \
  less
echo $?
echo ""

test.conf
Code: 
PKG_DBDIR    = "/tmp/test_fetch/var/db/pkg";
PKG_CACHEDIR = "/tmp/test_fetch/var/cache/pkg";

REPOS_DIR [
    "/tmp/test_fetch/repos";
]

ABI = "FreeBSD:14:amd64";

I suspect that pkg-fetch not only copies the package to the -o dir, but also updates some index files in PKG_DBDIR. And it's those index files that must be owned by root. INSTALL_AS_USER seems to suppress that check, allowing pkg-fetch to proceed on the second and subsequent invocations. Of course no installation is ever performed by fetch operation, it's just a way (workaround?) to suppress the error.
 
Igor2022 said:
Unfortunately, it doesn't work.
Click to expand...
Try this:

Code: 
normal user = nu

nu % mkdir -p /tmp/pkg/{db/pkg,cache/pkg}

nu % ls -ld /tmp/pkg
drwxr-xr-x  5 nu wheel 7 Feb 22 16:35 /tmp/pkg/

nu % ls -l /tmp/pkg
total 1
drwxr-xr-x  2 nu wheel    2 Feb 22 16:35 cache/
drwxr-xr-x  3 nu wheel    3 Feb 22 16:35 db/

/tmp/pkg/pkg.conf
Rich (BB code): 
PKG_DBDIR = "/tmp/pkg/db/pkg";
PKG_CACHEDIR = "/tmp/pkg/cache/pkg";

(PKG_CACHEDIR not necessary if using -o )

ABI = "FreeBSD:14:i386";

# I'm on 14:amd64

PKG_ENV { INSTALL_AS_USER=yes
}

# Also, without setting PKG_ENV in pkg.conf
# nu % env INSTALL_AS_USER=yes pkg -C /tmp/pkg/pkg.conf fetch

/tmp/pkg/dwn.txt
Code: 
argp-standalone
bastille
bzip3
clutter-gtk3
contractor
dht

Rich (BB code): 
nu % pkg -C /tmp/pkg/pkg.conf fetch -r latest -o /tmp/pkg `cat /tmp/pkg/dwn.txt`

(for -r latest see below. That's the repository I use for FreeBSD:14:amd64/latest)

nu % ls -l /tmp/pkg/All
total 272
-rw-r--r--  1 nu wheel 32123 Jan 25 04:18 argp-standalone-1.5.0.pkg
-rw-r--r--  1 nu wheel 43825 Dec 26 20:40 bastille-0.10.20231125.pkg
-rw-r--r--  1 nu wheel 51855 Jan 18 07:06 bzip3-1.4.0_2.pkg
-rw-r--r--  1 nu wheel 22537 Jan  1  1970 clone-1.0.8.pkg
-rw-r--r--  1 nu wheel 38104 Feb 13 16:05 clutter-gtk3-1.8.4_3.pkg
-rw-r--r--  1 nu wheel 35552 Jan 31 07:36 contractor-0.3.5.pkg
-rw-r--r--  1 nu wheel 40631 Dec 24 16:59 dht-0.27.pkg

nu % tar xfO argp-standalone-1.5.0.pkg +COMPACT_MANIFEST
[...]"abi":"FreeBSD:14:i386"[...]

latest.conf
Code: 
latest: {
   url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",
   mirror_type: "srv",  signature_type: "fingerprints",
   fingerprints: "/usr/share/keys/pkg",
   enabled: yes
}

EDIT
Apparently pkg(8) doesn't care about the ABI in the repository configuration when a different ABI is set in pkg.conf.

Checked in 13.2, fetched packages have [...]"abi":"FreeBSD:14:amd64"[...] with ABI = "FreeBSD:14:amd64" set in pkg.conf, repository conf has url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest", ${ABI} here FreeBSD:13:amd64
 
You must log in or register to reply here.
Back
Top