Thanks. I am thinking about reducing number of things I have to build, not increase. Poudriere looks interesting as a last resort option but I'd rather avoid more caching and more building if I can.
True. But mixing ports and packages is ugly too.
Why? That's what I can't understand yet. It doesn't have to be ugly. ...
Beauty lies in the eye of the beholder. I won't submit the following solution to a beauty contest, however, it has the big advantage that it works, which by its own weighs out any opinions about aesthetics.
Code:
#!/bin/sh
### the list of the ports that shall be updated from sources
### in the future include 'devel/pecl-pthreads' before 'textproc/pecl-yaml'
portslist="\
devel/apr1 \
devel/subversion \
lang/php71 \
mail/postfix \
mail/roundcube \
net/mDNSResponder \
net/netatalk3 \
www/mod_php71 \
www/squid \
\
textproc/pecl-yaml \
archivers/php71-phar \
archivers/php71-zip \
archivers/php71-zlib \
converters/php71-iconv \
converters/php71-mbstring \
databases/php71-pdo \
databases/php71-pdo_pgsql \
databases/php71-pgsql \
databases/php71-sqlite3 \
devel/php71-intl \
devel/php71-json \
ftp/php71-curl \
graphics/php71-exif \
math/php71-bcmath \
net/php71-sockets \
security/php71-filter \
security/php71-hash \
security/php71-openssl \
sysutils/php71-fileinfo \
textproc/php71-dom \
textproc/php71-xml \
www/php71-session"
### fetching updates of the FreeBSD ports tree
/usr/bin/printf "\nFetching updates of the FreeBSD ports tree...\n"
/usr/sbin/portsnap fetch update
/usr/sbin/pkg version -v
/usr/sbin/pkg updating -d `date -v-2w +%Y%m%d`
### ask and in case of y|Y run the updating processes
/usr/bin/printf "\nDo you want to continue (y/n)? "
save_stty_state=$(stty -g); stty raw -echo; answer=$(head -c 1); stty $save_stty_state
if echo "$answer" | grep -iq "^y" ; then
/usr/bin/printf "\n\n"
/usr/sbin/pkg update
portmake=""
pkgslist=`/usr/sbin/pkg version -o | /usr/bin/cut -f1 -w`
for port in $portslist ; do
for pkg in $pkgslist ; do
if [ "$pkg" == "$port" ] ; then
continue 2
fi
done
portmake="$portmake $port"
done
pkgslist=""
outdated=`/usr/sbin/pkg version -ol\< | /usr/bin/cut -f1 -w`
for outd in $outdated ; do
for port in $portslist ; do
if [ "$port" == "$outd" ] ; then
portmake="$portmake $port"
continue 2
fi
done
pkgslist="$pkgslist $outd"
done
pkgupgrd=""
outdated=`/usr/sbin/pkg version -RU -ol\< | /usr/bin/cut -f1 -w`
for outd in $outdated ; do
for pkg in $pkgslist ; do
if [ "$pkg" == "$outd" ] ; then
pkgupgrd="$pkgupgrd $pkg"
continue 2
fi
done
done
/usr/bin/printf "\nUpdating binary packages...\n"
if [ "$pkgupgrd" != "" ] ; then
/usr/sbin/pkg upgrade -U $pkgupgrd
else
echo "All installed packages are up-to-date."
fi
/usr/bin/printf "\nUpdating ports...\n"
if [ "$portmake" != "" ] ; then
cwd=$PWD
for port in $portmake ; do
cd /usr/ports/$port
/usr/bin/make clean deinstall
/usr/bin/make install clean
done
cd "$cwd"
else
echo "All installed ports are up-to-date."
fi
/usr/bin/printf "\nCleaning up...\n"
/usr/sbin/pkg clean -y
else
/usr/bin/printf "\n\n"
fi
The above script does what
pkg(8) is not able to do, namely, ignore some ports of a software installation in the
pkg upgrade process.
At the head of the script we need to inform the ports which must not be upgraded from the binary repository but shall be build from the ports tree -- of course the list of ports
$portslist needs to be adapted for other machines, and maintained whenever a new port with custom options is installed on the system.
In the first stage it fetches the latest updates of the FreeBSD ports tree by the way of
portsnap(8), and it shows a list of all installed software with indication of the version status. It will also show the entries in
/usr/ports/UPDATING related to the installed software of the last two weeks. Depending on that information the user might decide to stop or to proceed.
In case of proceed it updates the local copy of the repository catalogue(s). You want to let
pkg use the
latest repository, by default it works against the
quaterly repository which is with a higher probability out of synch with the ports tree.
Now the script compiles 3 lists, the list of all outdated software and more 2 divisions, namely outdated packages to be upgraded
$pkgupgrd and outdated ports to be made
$portsmake. BTW,
pkg lock does not work as advertised.
Finally it calls the command
pkg upgrade -U $pkgupgrd which takes care only for the outdated packages, and then it loops through the list of the outdated ports, stepping into each of the respective port's directory and calling the usual
make deinstall/make install combo.
This script works now on 6 FreeBSD servers very well. All well and good, what about the risks? If you are not satisfied, then continue reading. ...
... In the case of deviations from the common feature set, people recommend to built everything on the local machine from the port's sources, and those people continue telling you that it is too risky to intermix pre-built packages with your customized port-builds.
In a certain sense, this is even correct, namely, without a serious risk assessment the best educated guess of the experts is always that many things may go wrong. And of course the conclusion from this is that Dual Mode updating of ports, i.e. mixed installation from locally built and pre-built packages is too risky for even trying it.
Risk assessment and Risk separation
A package with the common set of options can be created locally from the corresponding port with the
make package command and by leaving the default options in the configuration dialog in place. The thus locally created package would be exactly identical to the corresponding package of the same version in the public repository of pre-built packages.
This identity of locally built vs. pre-built packages is one of the essential basis for the further discussion. The other basis is that the installation phase is agnostic to the origin of the package -- a package is a package is a package is ...
Any procedure of Dual Mode installation and updating of ports must stay on these basis, and given this constraints the risks can be named and classified into 2 groups:
1. First installation of a port/package
It may happen, that a selection of ports and options may conflict with installed packages. These conflicts must be resolved beforehand utilizing above script, by calling
pkg delete on the infringing package and adding it to the list of ports at the head of the script.
2. Updating of the installed ports/packages collections
Ususally (
amd64) the ports tree is ahead to the latest package repository by 2 to 3 days. In rare cases, it happens that a custom port depends on a new version of pre-built package which didn't made it yet into the remote repository. In this case we may wait for more 2 or 3 days and start the script again, or we manually run a local onetime build of the dependency before executing the software update script.
Special occasions, which usually show-up in the
/usr/local/UPDATING file, need special actions before we can let the script do its automatic job. For example the recent switch from
mail/dovecot2 to
mail/dovecot needed to be done outside of the normal updating sequence.
The command
pkg upgrade shows a list of actions to be done, and it asks whether to proceed
[y/n]. If
pkg wants to reinstall some of your custom ports because of changed options, then we need to tell
n(o) here, because this means, that package dependencies have changed compared to our first installation. In order to reolve this conflict, we would need to customize the package, i.e. call
pkg delete on it and put it into the list of the ports at the head of our script.
