Best method to install packages offline

ikevin8me

Active Member

Reaction score: 12
Messages: 178

I have another machine which is offline. What is the best method to install packages offline?

I mean, due to the dependencies, it is almost impossible to download all the dependent packages to install them.

Is there a way to use the pkg command to download (from a connected FreeBSD machine) but not install them? In this way, I can copy to the offline machine to install them.

I tried to build using the ports, and it seems that it needs to be online too.

Let’s say, I have an offline machine and I need to install OpenJDK… what should I do?
 

mtu

Active Member

Reaction score: 101
Messages: 156

You could compile the required software on another machine and copy the resulting packages over to the offline machine.

One tool that is very well suited for this is ports-mgmt/poudriere. It's rather complicated to set up and confusing to learn, but when you've finally got it right, it will be very comfortable to do what you want.

poudriere(8) takes a list of required packages (with custom options, if you wish), automatically resolves dependencies, builds all required packages and spits out a repository directory that you can just copy over to another machine. On the target machine, you tell pkg.conf(5) to look in that folder, install your software, and that's it.

The only drawback is that when you want some new software, you have to build and copy it first. It's not realistically practical to build every possible package in advance.
 

grahamperrin

Daemon

Reaction score: 345
Messages: 1,166

… use the pkg command to download (from a connected FreeBSD machine) but not install …

Use the --fetch-only option, for example (not using pkg fetch):

Code:
root@mowa219-gjp4-8570p:~ # pkg fetch hw-probe
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating poudriere repository catalogue...
poudriere repository is up to date.
All repositories are up to date.
The following packages will be fetched:

New packages to be FETCHED:
        hw-probe: 1.6.b2 (94 KiB: 100.00% of the 94 KiB to download)

Number of packages to be fetched: 1

94 KiB to be downloaded.

Proceed with fetching packages? [y/N]: n
root@mowa219-gjp4-8570p:~ # pkg install --fetch-only -y hw-probe
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating poudriere repository catalogue...
poudriere repository is up to date.
All repositories are up to date.
The following 4 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        dmidecode: 3.3 [FreeBSD]
        hw-probe: 1.6.b2 [FreeBSD]
        hwstat: 0.5.1 [FreeBSD]
        lscpu: 1.2.0 [FreeBSD]

Number of packages to be installed: 4

175 KiB to be downloaded.
[1/4] Fetching hw-probe-1.6.b2.txz: 100%   94 KiB  96.1kB/s    00:01 
[2/4] Fetching lscpu-1.2.0.txz: 100%    8 KiB   8.4kB/s    00:01 
[3/4] Fetching hwstat-0.5.1.txz: 100%    7 KiB   7.3kB/s    00:01 
[4/4] Fetching dmidecode-3.3.txz: 100%   66 KiB  67.5kB/s    00:01 
Checking integrity... done (0 conflicting)
root@mowa219-gjp4-8570p:~ # pkg install --fetch-only -q -y usbhid-dump usbids
root@mowa219-gjp4-8570p:~ # ls -hl /var/cache/pkg
total 430
lrwxr-xr-x  1 root  wheel    28B Jun 13 02:18 dmidecode-3.3.txz -> dmidecode-3.3~bda94d76af.txz
-rw-r--r--  1 root  wheel    66K Jun  8 11:19 dmidecode-3.3~bda94d76af.txz
lrwxr-xr-x  1 root  wheel    30B Jun 13 02:18 hw-probe-1.6.b2.txz -> hw-probe-1.6.b2~e94bc14851.txz
-rw-r--r--  1 root  wheel    94K Jun  9 14:33 hw-probe-1.6.b2~e94bc14851.txz
lrwxr-xr-x  1 root  wheel    27B Jun 13 02:18 hwstat-0.5.1.txz -> hwstat-0.5.1~b1f8c2ec1b.txz
-rw-r--r--  1 root  wheel   7.1K Jun  8 19:24 hwstat-0.5.1~b1f8c2ec1b.txz
lrwxr-xr-x  1 root  wheel    26B Jun 13 02:18 lscpu-1.2.0.txz -> lscpu-1.2.0~7389189e14.txz
-rw-r--r--  1 root  wheel   8.2K Jun  8 22:22 lscpu-1.2.0~7389189e14.txz
lrwxr-xr-x  1 root  wheel    30B Jun 13 02:20 usbhid-dump-1.4.txz -> usbhid-dump-1.4~96df800cf5.txz
-rw-r--r--  1 root  wheel    30K Jun  8 13:04 usbhid-dump-1.4~96df800cf5.txz
lrwxr-xr-x  1 root  wheel    30B Jun 13 02:20 usbids-20210331.txz -> usbids-20210331~7fced6a9da.txz
-rw-r--r--  1 root  wheel   197K Jun  8 10:10 usbids-20210331~7fced6a9da.txz
root@mowa219-gjp4-8570p:~ # du -hs /var/cache/pkg
2.0M    /var/cache/pkg
root@mowa219-gjp4-8570p:~ #

Copy the cached files to offline storage e.g. a USB flash drive, then …

I have another machine which is offline. …

pkg-add(8) to add from the offline storage. Assuming that both machines share the same architecture, and version of FreeBSD.

Maybe not the best method, but it's fairly easy to grasp.

Certainly not as comprehensive as the poudriere method. Like, if (without using poudriere) what's on the USB flash drive does not include all required ⋯.txz files, you'll need to revisit the online machine and re-run pkg install --fetch-only for the additional requirements.
 

grahamperrin

Daemon

Reaction score: 345
Messages: 1,166


If you feel adventurous, ports-mgmt/poudriere-devel can prefetch packages (to spend less time building).

Code:
root@mowa219-gjp4-8570p:~ # poudriere bulk -b latest -j main -c emulators/virtualbox-ose
[00:00:00] Creating the reference jail... done
[00:00:03] Mounting system devices for main-default
[00:00:03] Using packages from previously failed build: /usr/local/poudriere/data/packages/main-default/.building
[00:00:03] Mounting ccache from: /var/cache/ccache
[00:00:03] Mounting ports from: /usr/local/poudriere/ports/default
[00:00:03] Mounting packages from: /usr/local/poudriere/data/packages/main-default
[00:00:03] Mounting distfiles from: /usr/local/poudriere/ports/default/distfiles
[00:00:03] Copying /var/db/ports from: /usr/local/etc/poudriere.d/options
[00:00:03] Appending to make.conf: /usr/local/etc/poudriere.d/make.conf
/etc/resolv.conf -> /usr/local/poudriere/data/.m/main-default/ref/etc/resolv.conf
[00:00:03] Starting jail main-default
[00:00:04] Logs: /usr/local/poudriere/data/logs/bulk/main-default/2021-06-13_02h43m08s
[00:00:04] Loading MOVED for /usr/local/poudriere/data/.m/main-default/ref/usr/ports
[00:00:04] Ports supports: FLAVORS SELECTED_OPTIONS
[00:00:04] Gathering ports metadata
[00:00:07] Calculating ports order and dependencies
[00:00:08] (-c) Cleaning all packages... done
[00:00:08] Trimming IGNORED and blacklisted ports
[00:00:08] Prefetching missing packages from pkg+http://pkg.FreeBSD.org/${ABI}/latest
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:14:amd64/latest, please wait...
[main-default] Installing pkg-1.16.3...
[main-default] Extracting pkg-1.16.3: 100%
Updating FreeBSD repository catalogue...
[main-default] Fetching meta.conf: 100%    163 B   0.2kB/s    00:01   
[main-default] Fetching packagesite.txz: 100%    6 MiB   6.5MB/s    00:01   
Processing entries: 100%
FreeBSD repository update completed. 30618 packages processed.
All repositories are up to date.
Updating database digests format: 100%
The following packages will be fetched:

New packages to be FETCHED:
        aalib: 1.4.r5_13 (139 KiB: 0.04% of the 332 MiB to download)
        …
        zstd: 1.5.0 (582 KiB: 0.17% of the 332 MiB to download)

Number of packages to be fetched: 245

The process will require 332 MiB more space.
332 MiB to be downloaded.
[main-default] Fetching dbus-1.12.20_4.txz: 100%  368 KiB 377.1kB/s    00:01   
…
[main-default] Fetching py38-pycparser-2.20.txz: 100%  165 KiB 168.9kB/s    00:01   
[00:01:41] Sanity checking the repository
[00:01:41] Checking packages for incremental rebuild needs
[00:01:46] Deleting cups-2.3.3op2.txz: missing dependency: gnutls-3.6.15
[00:01:46] Deleting glib-2.66.8,2.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting gobject-introspection-1.66.1,1.txz: missing dependency: glib-2.66.8,2
[00:01:46] Deleting gtk-doc-1.33.2.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting gtk-update-icon-cache-3.24.26.txz: missing dependency: glib-2.66.8,2
[00:01:46] Deleting harfbuzz-2.8.1.txz: missing dependency: glib-2.66.8,2
[00:01:46] Deleting itstool-2.0.6.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting libIDL-0.8.14_5.txz: missing dependency: glib-2.66.8,2
[00:01:46] Deleting libgudev-234.txz: missing dependency: glib-2.66.8,2
[00:01:46] Deleting libinput-1.16.4.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting libwacom-1.5.txz: missing dependency: glib-2.66.8,2
[00:01:46] Deleting llvm10-10.0.1_5.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting mesa-dri-20.2.3_1.txz: missing dependency: llvm10-10.0.1_5
[00:01:46] Deleting ninja-1.10.2,2.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting pango-1.48.4_1.txz: missing dependency: glib-2.66.8,2
[00:01:46] Deleting py38-Babel-2.9.1.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-CommonMark-0.9.1.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-Jinja2-2.11.2_1.txz: missing dependency: py38-Babel-2.9.1
[00:01:46] Deleting py38-alabaster-0.7.12.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-beaker-1.11.0.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-cffi-1.14.5.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-cryptography-3.3.2.txz: missing dependency: py38-cffi-1.14.5
[00:01:46] Deleting py38-chardet-4.0.0,1.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-cython-0.29.23.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-docutils-0.17.1.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-evdev-1.4.0.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-future-0.18.2.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-gi-docgen-2021.5.txz: missing dependency: py38-Jinja2-2.11.2_1
[00:01:46] Deleting py38-idna-2.10.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-imagesize-1.2.0.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-importlib-metadata-3.3.0_1.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-libxml2-2.9.10_4.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-mako-1.0.14_1.txz: missing dependency: py38-Babel-2.9.1
[00:01:46] Deleting py38-markupsafe-1.1.1_1.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-openssl-20.0.1.txz: missing dependency: py38-cryptography-3.3.2
[00:01:46] Deleting py38-markdown-3.3.4.txz: missing dependency: py38-importlib-metadata-3.3.0_1
[00:01:46] Deleting py38-packaging-20.9.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-pycparser-2.20.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-pygments-2.7.2.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-pyparsing-2.4.7.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-pysocks-1.7.1.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-pystemmer-2.0.1.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-pyudev-0.22.0.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-pytz-2021.1,1.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-recommonmark-0.5.0_2.txz: missing dependency: py38-CommonMark-0.9.1
[00:01:46] Deleting py38-setuptools-57.0.0.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-setuptools_scm-4.1.2_1.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-six-1.16.0.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting py38-snowballstemmer-2.1.0.txz: missing dependency: py38-pystemmer-2.0.1
[00:01:46] Deleting py38-smartypants-2.0.1.txz: missing dependency: py38-setuptools-57.0.0
[00:01:46] Deleting py38-sphinx-3.5.2,1.txz: missing dependency: py38-Babel-2.9.1
[00:01:46] Deleting py38-sphinxcontrib-applehelp-1.0.2.txz: missing dependency: py38-setuptools-57.0.0
[00:01:46] Deleting py38-sphinxcontrib-devhelp-1.0.2.txz: missing dependency: py38-setuptools-57.0.0
[00:01:46] Deleting py38-sphinxcontrib-jsmath-1.0.1.txz: missing dependency: py38-setuptools-57.0.0
[00:01:46] Deleting py38-sphinxcontrib-qthelp-1.0.3.txz: missing dependency: py38-setuptools-57.0.0
[00:01:46] Deleting py38-sphinxcontrib-htmlhelp-1.0.3.txz: missing dependency: py38-setuptools-57.0.0
[00:01:46] Deleting py38-toml-0.10.2.txz: missing dependency: py38-setuptools-57.0.0
[00:01:46] Deleting py38-typogrify-2.0.7.txz: missing dependency: py38-setuptools-57.0.0
[00:01:46] Deleting py38-sphinxcontrib-serializinghtml-1.1.4.txz: missing dependency: py38-setuptools-57.0.0
[00:01:46] Deleting py38-zipp-3.4.0.txz: missing dependency: py38-setuptools-57.0.0
[00:01:46] Deleting qt5-core-5.15.2_5.txz: missing dependency: glib-2.66.8,2
[00:01:46] Deleting qt5-declarative-5.15.2.txz: missing dependency: qt5-core-5.15.2_5
[00:01:46] Deleting qt5-dbus-5.15.2_1.txz: missing dependency: qt5-core-5.15.2_5
[00:01:46] Deleting qt5-gui-5.15.2_4.txz: missing dependency: glib-2.66.8,2
[00:01:46] Deleting qt5-linguisttools-5.15.2.txz: missing dependency: qt5-core-5.15.2_5
[00:01:46] Deleting qt5-network-5.15.2_1.txz: missing dependency: qt5-core-5.15.2_5
[00:01:46] Deleting qt5-opengl-5.15.2_1.txz: missing dependency: qt5-core-5.15.2_5
[00:01:46] Deleting qt5-printsupport-5.15.2_1.txz: missing dependency: cups-2.3.3op2
[00:01:46] Deleting qt5-sql-5.15.2_1.txz: missing dependency: qt5-core-5.15.2_5
[00:01:46] Deleting qt5-testlib-5.15.2_1.txz: missing dependency: qt5-core-5.15.2_5
[00:01:46] Deleting qt5-widgets-5.15.2_1.txz: missing dependency: qt5-core-5.15.2_5
[00:01:46] Deleting qt5-x11extras-5.15.2_1.txz: missing dependency: qt5-core-5.15.2_5
[00:01:46] Deleting qt5-xml-5.15.2_1.txz: missing dependency: qt5-core-5.15.2_5
[00:01:46] Deleting shared-mime-info-2.0.txz: missing dependency: glib-2.66.8,2
[00:01:46] Deleting xcb-proto-1.14.1.txz: missing dependency: python38-3.8.10
[00:01:46] Deleting atk-2.36.0.txz: missing dependency: glib-2.66.8,2
[00:01:46] Deleting at-spi2-core-2.36.0.txz: missing dependency: glib-2.66.8,2
[00:01:46] Deleting avahi-app-0.8.txz: missing dependency: glib-2.66.8,2
[00:01:46] Deleting cairo-1.17.4,3.txz: missing dependency: glib-2.66.8,2
[00:01:46] Deleting dbus-glib-0.110.txz: missing dependency: glib-2.66.8,2
[00:01:46] Deleting gdk-pixbuf2-2.40.0.txz: missing dependency: glib-2.66.8,2
[00:01:47] Deleting stale symlinks... done
[00:01:47] Deleting empty directories... done
[00:01:47] Cleaning the build queue
[00:01:47] Sanity checking build queue
[00:01:47] Processing PRIORITY_BOOST
[00:01:47] Balancing pool
[main-default] [2021-06-13_02h43m08s] [balancing_pool:] Queued: 90 Built: 0  Failed: 0  Skipped: 0  Ignored: 0  Tobuild: 90  Time: 00:01:43
[00:01:47] Recording filesystem state for prepkg... done
[00:01:47] Building 90 packages using 2 builders
[00:01:47] Starting/Cloning builders
[00:01:48] Hit CTRL+t at any time to see build progress and stats
[00:01:48] [01] [00:00:00] Building lang/python38 | python38-3.8.10
[00:01:48] [02] [00:00:00] Building ftp/curl | curl-7.77.0
^C[00:02:17] Error: Signal SIGINT caught, cleaning up and exiting
[00:02:17] Cleaning up
[00:02:17] Unmounting file systems
root@mowa219-gjp4-8570p:~ #
 

grahamperrin

Daemon

Reaction score: 345
Messages: 1,166

Thanks. I wondered whether --quiet would produce a shortlist of names (list or line), instead of presenting metadata alongside each named package.

Instead, --quiet assumes --yes:

Code:
root@mowa219-gjp4-8570p:/tmp # file All
All: cannot open `All' (No such file or directory)
root@mowa219-gjp4-8570p:/tmp # pkg fetch -d -o /tmp/ hw-probe
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating poudriere repository catalogue...
poudriere repository is up to date.
All repositories are up to date.
The following packages will be fetched:

New packages to be FETCHED:
        ca_root_nss: 3.63 (276 KiB: 1.63% of the 17 MiB to download)
        curl: 7.76.1 (1 MiB: 7.88% of the 17 MiB to download)
        dmidecode: 3.3 (66 KiB: 0.39% of the 17 MiB to download)
        hw-probe: 1.6.b2 (94 KiB: 0.55% of the 17 MiB to download)
        hwstat: 0.5.1 (7 KiB: 0.04% of the 17 MiB to download)
        libnghttp2: 1.43.0 (123 KiB: 0.73% of the 17 MiB to download)
        lscpu: 1.2.0 (8 KiB: 0.05% of the 17 MiB to download)
        perl5: 5.32.1_1 (14 MiB: 85.96% of the 17 MiB to download)
        smartmontools: 7.2_1 (470 KiB: 2.77% of the 17 MiB to download)

Number of packages to be fetched: 9

The process will require 17 MiB more space.
17 MiB to be downloaded.

Proceed with fetching packages? [y/N]: N
root@mowa219-gjp4-8570p:/tmp # pkg fetch -d -o /tmp/ --quiet hw-probe && du -hs All
 17M    All
root@mowa219-gjp4-8570p:/tmp # rm -r /tmp/All
root@mowa219-gjp4-8570p:/tmp #



I wondered, because a few days ago I could not get an index for the ports tree but today, it works:

Code:
root@mowa219-gjp4-8570p:/tmp # poudriere ports -u
[00:00:00] Updating portstree "default" with git+https... done
root@mowa219-gjp4-8570p:/tmp # cd /usr/ports/sysutils/hw-probe/
root@mowa219-gjp4-8570p:/usr/ports/sysutils/hw-probe # make pretty-print-build-depends-list
pretty-print-build-depends-list requires an INDEX file (INDEX-14). Please run make index or make fetchindex.
root@mowa219-gjp4-8570p:/usr/ports/sysutils/hw-probe # make pretty-print-all-depends-list
make: don't know how to make pretty-print-all-depends-list. Stop

make: stopped in /usr/ports/sysutils/hw-probe
root@mowa219-gjp4-8570p:/usr/ports/sysutils/hw-probe # make pretty-print-run-depends-list
pretty-print-run-depends-list requires an INDEX file (INDEX-14). Please run make index or make fetchindex.
root@mowa219-gjp4-8570p:/usr/ports/sysutils/hw-probe # cd ../.. && make fetchindex
/usr/bin/env  fetch -am -o /usr/ports/INDEX-14.bz2 https://www.FreeBSD.org/ports/INDEX-14.bz2
/usr/ports/INDEX-14.bz2                               2305 kB  687 kBps    03s
root@mowa219-gjp4-8570p:/usr/ports # cd sysutils/hw-probe/ && make pretty-print-run-depends-list
This port requires package(s) "ca_root_nss-3.63 curl-7.77.0 dmidecode-3.3 hwstat-0.5.1 libnghttp2-1.43.0 lscpu-1.2.0 perl5-5.32.1_1 smartmontools-7.2_1" to run.
root@mowa219-gjp4-8570p:/usr/ports/sysutils/hw-probe #
 
Top