I need help understanding pkg upgrade

P

pschmehl

I'm tired of running portmaster -ad and taking two or three days to build all the ports, resolving so many issues that pop up along the way. But pkg scares the hell out of me. I manage two servers. I have no backup or test systems to try things out before going into production. Both of these systems are live on the internet and in production. So, it's critical that I get updates right.

But, when I run pkg upgrade, it wants to uninstall ports that I absolutely need.

For example, one server is the mail server. The other is the webserver. Since web is more critical, and mail can be down for five days before we start losing mail, I always run upgrades on the mail server first. That helps me iron out any bugs, discover which major ports (like perl, php, python, etc.) need to be upgraded (and upgrade all the dependent ports) before moving to the web server and risking taking it down.

Tonight, I ran pkg upgrade, and it wanted to uninstall postfix. Why would it do that? So, I locked postfix, and ran it again. Then it wanted to uninstall 10 apps. I really don't understand what's going on. Can someone explain it to me?

This is the first run. Obviously, I don't want to uninstall postfix, so I canceled the upgrade. But why on earth does pkg want to uninstall postfix? That's the most important app on this server. I also run mailman on this server, and pkg wants to reinstall it because its options have changed. How can I find out what options have changed so I can determine if I have to lock it as well?
Installed packages to be REMOVED:
postfix: 3.5.6,1
Click to expand...

New packages to be INSTALLED:
freeglut: 3.2.1
jpeg-turbo: 2.0.6
libGLU: 9.0.2_1
libevent: 2.1.12
libglvnd: 1.3.3
pinentry-curses: 1.1.1
py27-setuptools44: 44.1.1
ruby26: 2.6.8,1
ruby27-gems: 3.0.8
xxhash: 0.8.0
zstd: 1.5.0

Installed packages to be UPGRADED:
GentiumPlus: 5.000_1 -> 6.001
alsa-lib: 1.1.2_2 -> 1.2.2
apache-ant: 1.10.7 -> 1.10.8
avahi-app: 0.7_3 -> 0.8
bsdstats: 7.0 -> 7.0_2
bzip2: 1.0.7 -> 1.0.8
cairo: 1.16.0,2 -> 1.17.4,3
check: 0.15.1 -> 0.15.2
cmake: 3.17.3_1 -> 3.21.0
courier-authlib-base: 0.71.0 -> 0.71.3
courier-authlib-userdb: 0.71.0 -> 0.71.3
courier-imap: 5.0.11,2 -> 5.1.4,2
courier-unicode: 2.1_2 -> 2.2.3
cups: 2.3.3_1 -> 2.3.3op2
cyrus-sasl-saslauthd: 2.1.27_1 -> 2.1.27_2
dbus: 1.12.20 -> 1.12.20_5
dbus-glib: 0.110 -> 0.112
expat: 2.2.8 -> 2.4.1
fontconfig: 2.13.92_2,1 -> 2.13.93_1,1
fop: 2.3 -> 2.6
freetype2: 2.10.2 -> 2.10.4
gettext: 0.20.2 -> 0.21
gobject-introspection: 1.56.1_1,1 -> 1.66.1,1
gpgme: 1.14.0 -> 1.15.1
html2text: 1.3.2a -> 1.3.2a,1
icu: 67.1,1 -> 69.1,1
jasper: 2.0.16_1 -> 2.0.33
java-zoneinfo: 2020.a -> 2021.a
javavmwrapper: 2.7.5 -> 2.7.7
jbig2dec: 0.18 -> 0.19
json-c: 0.14 -> 0.15_1
jsoncpp: 1.9.3 -> 1.9.4
lcms2: 2.11_1 -> 2.12
libX11: 1.6.9_2,1 -> 1.7.2,1
libXaw: 1.0.13_3,2 -> 1.0.14,2
libXt: 1.2.0,1 -> 1.2.1,1
libarchive: 3.4.3,1 -> 3.5.1,1
libassuan: 2.5.3 -> 2.5.5
libdrm: 2.4.102,1 -> 2.4.107,1
libepoll-shim: 0.0.20200602 -> 0.0.20210418
libgd: 2.3.0,1 -> 2.3.1,1
libksba: 1.4.0 -> 1.6.0
liblz4: 1.9.2_1,1 -> 1.9.3,1
libmaxminddb: 1.4.2 -> 1.6.0
libunwind: 20200331 -> 20201110
libuv: 1.38.1 -> 1.41.0
libxcb: 1.13.1 -> 1.14_1
libxshmfence: 1.3 -> 1.3_1
llvm90: 9.0.1_2 -> 9.0.1_3
mDNSResponder: 1096.40.7 -> 1310.120.71
mesa-libs: 19.0.8_2 -> 21.1.5_1
minixmlto: 0.0.2_1 -> 0.0.3
mpfr: 4.1.0 -> 4.1.0_1
neon: 0.30.2_4 -> 0.31.2
oniguruma: 6.9.5.r1_1 -> 6.9.7.1
open-motif: 2.3.8_1 -> 2.3.8_2
opendkim: 2.10.3_11 -> 2.10.3_12
openjdk8: 8.262.10.1 -> 8.302.08.1
p5-Crypt-OpenSSL-Guess: 0.11 -> 0.13
p5-Data-Dump: 1.23_1 -> 1.25
p5-Digest-HMAC: 1.03_1 -> 1.04
p5-ExtUtils-CBuilder: 0.280234,1 -> 0.280236,1
p5-File-Listing: 6.04_1 -> 6.14
p5-HTTP-Cookies: 6.08 -> 6.10
p5-HTTP-Message: 6.25 -> 6.33
p5-IO-Socket-IP: 0.39 -> 0.41
p5-IO-Socket-SSL: 2.068 -> 2.071
p5-JSON-PP: 4.05 -> 4.06
p5-LWP-Protocol-https: 6.09 -> 6.10
p5-Mail-AuthenticationResults: 1.20200331.1 -> 2.20210112
p5-Mail-DKIM: 0.58 -> 1.20200907
p5-Mail-Tools: 2.19 -> 2.21
p5-Mozilla-CA: 20180117 -> 20200520
p5-Net-CIDR-Lite: 0.21_1 -> 0.22
p5-Net-DNS: 1.25,1 -> 1.32,1
p5-Net-HTTP: 6.19 -> 6.21
p5-URI: 1.76 -> 5.09
p5-libwww: 6.46 -> 6.55
patch: 2.7.6_1 -> 2.7.6_2
pciids: 20200624 -> 20210725
pflogsumm: 1.1.5,1 -> 1.1.5_1,1
pinentry: 1.1.0_6 -> 1.1.1
pinentry-tty: 1.1.0 -> 1.1.1
pixman: 0.40.0 -> 0.40.0_1
png: 1.6.37 -> 1.6.37_1
portlint: 2.19.2 -> 2.19.7
portmaster: 3.19_25 -> 3.19_31
py27-Babel: 2.8.0 -> 2.9.1
py27-cython: 0.29.15 -> 0.29.24
py27-pytz: 2020.1,1 -> 2021.1,1
py27-six: 1.14.0 -> 1.16.0
py36-setuptools: 44.0.0 -> 57.0.0
py37-cython: 0.29.15 -> 0.29.24
py37-setuptools: 44.0.0 -> 57.0.0
python36: 3.6.11_1 -> 3.6.14
python37: 3.7.8_1 -> 3.7.11
rhash: 1.3.9 -> 1.4.2
rsync: 3.1.3_1 -> 3.2.3_1
ruby: 2.6.6_1,1 -> 2.7.4,1
rubygem-rdoc: 6.1.2_1 -> 6.3.2
screen: 4.8.0 -> 4.8.0_3
sudo: 1.9.2 -> 1.9.7p1
sysconftool: 0.17 -> 0.18
tiff: 4.1.0 -> 4.3.0
trousers: 0.3.14_2 -> 0.3.14_3
unbound: 1.10.1 -> 1.13.1
w3m: 0.5.3.20200507 -> 0.5.3.20210424
wayland: 1.18.0_4 -> 1.19.0_1
wayland-protocols: 1.20 -> 1.21
webp: 1.1.0 -> 1.2.0
wget: 1.20.3 -> 1.21
xcb-proto: 1.13_1 -> 1.14.1
xorg-macros: 1.19.2 -> 1.19.3
xorgproto: 2020.1 -> 2021.4

Installed packages to be REINSTALLED:
OpenSP-1.5.2_3 (options changed)
apr-1.7.0.1.6.1_1 (direct dependency changed: db5)
autoconf-2.69_3 (ABI changed: 'freebsd:11:x86:64' -> 'freebsd:11:*')
autoconf-wrapper-20131203 (ABI changed: 'freebsd:11:x86:64' -> 'freebsd:11:*')
db5-5.3.28_7 (options changed)
dejavu-2.37_1 (options changed)
docbook-xsl-1.79.1_1,1 (options changed)
dsssl-docbook-modular-1.79_1,1 (ABI changed: 'freebsd:11:x86:64' -> 'freebsd:11:*')
flex-2.6.4_2 (options changed)
gettext-tools-0.21 (options changed)
gnupg1-1.4.23_2 (needed shared library changed)
iso-schematron-xslt-20130313_1 (ABI changed: 'freebsd:11:x86:64' -> 'freebsd:11:*')
jbigkit-2.1_1 (options changed)
llvm80-8.0.1_4 (direct dependency changed: python38)
mailman-2.1.34 (options changed)
netpbm-10.91.01 (options changed)
p5-HTTP-Date-6.05 (direct dependency removed: p5-Time-Local
p5-Net-DNS-Resolver-Programmable-0.009 (options changed
p5-Net-IP-1.26_1 (ABI changed: 'freebsd:11:x86:64' -> 'freebsd:11:*')
p5-Test-NoWarnings-1.04_2 (ABI changed: 'freebsd:11:x86:64' -> 'freebsd:11:*')
p5-type1inst-0.6.1_6 (options changed)
py27-dnspython-1.16.0 (direct dependency changed: py27-setuptools44)
py27-future-0.18.2 (direct dependency changed: py27-setuptools44)
py27-pycparser-2.20 (direct dependency changed: py27-setuptools44)
python27-2.7.18_1 (needed shared library changed)
sqlite3-3.35.5_3,1 (options changed)
texi2html-5.0_2,1 (ABI changed: 'freebsd:11:x86:64' -> 'freebsd:11:*')

Number of packages to be removed: 1
Number of packages to be installed: 11
Number of packages to be upgraded: 114
Number of packages to be reinstalled: 27
Click to expand...
 
So just to be clear - these are machines that USED to be using ports/portmaster and you want to try and move to binary packages from that old set-up?

They are not clean set-ups with binary packages already?
pschmehl said:
I manage two servers. I have no backup or test systems to try things out before going into production. Both of these systems are live on the internet and in production. So, it's critical that I get updates right.
Click to expand...
Save yourself a world of pain and change this situation.

I don't have the answer for you, but I'm going through the same migration process - but I'm leaving old machines on portmaster, and starting (on test machines and then a new production machine) using binary packages. Trying the migration live is a recipe for maxium stress. Find an old machine, cheap VM, whatever - there must be something you can experiment on!

And sort out backups. You'll feel better for it.
 
OP, this is part of the experience. Even big companies are highly reluctant to upgrade their Linux servers precisely because of the issues you describe. So they end up stuck in the morass "I am so scared of upgrading, if I don't do it right, everything will break!"
I'm still in the process of setting up Poudriere - but even lining up the details so that only KDE gets properly upgraded (just fresh tarballs downloaded, compiled with same settings as before) - even that is a lengthy process for me.

One suggestion I have - Get another server, install up to date stuff on it, and then replicate the functionality you need. yeah, that will take more time, but that's a safer way to do it. I did exactly that on my first IT job at a small shop. The shop's boss didn't want to buy another server, though, so I scrounged up a discarded PC from the back room, and set it up as a dedicated email server running Dovecot and FreeBSD 6.0 (yeah, it was a long time ago). The move was very clean. And, I used pkg at the time.
 
richardtoohey2 said:
So just to be clear - these are machines that USED to be using ports/portmaster and you want to try and move to binary packages from that old set-up?

They are not clean set-ups with binary packages already?

Save yourself a world of pain and change this situation.

I don't have the answer for you, but I'm going through the same migration process - but I'm leaving old machines on portmaster, and starting (on test machines and then a new production machine) using binary packages. Trying the migration live is a recipe for maxium stress. Find an old machine, cheap VM, whatever - there must be something you can experiment on!

And sort out backups. You'll feel better for it.
Click to expand...
Yes, these are machines that have always used ports/portmaster. They are not clean set-ups with binary packages. The reason for that is that pkg doesn't install the correct options for certian ports that I use. But 99% of the ports installed could use packages with no problem. In fact, it's been my practice, when a port fails to upgrade with portmaster, to install the package instead, where possible.

I don't understand all the talk about not mixing binary packages with portmaster-built packages. What is the difference? Portmster actually builds packages.

Backups are a separate issue. There is no money for software. This is a strictly volunteer, bandaid and bailing wire setup.
 
astyle said:
OP, this is part of the experience. Even big companies are highly reluctant to upgrade their Linux servers precisely because of the issues you describe. So they end up stuck in the morass "I am so scared of upgrading, if I don't do it right, everything will break!"
I'm still in the process of setting up Poudriere - but even lining up the details so that only KDE gets properly upgraded (just fresh tarballs downloaded, compiled with same settings as before) - even that is a lengthy process for me.

One suggestion I have - Get another server, install up to date stuff on it, and then replicate the functionality you need. yeah, that will take more time, but that's a safer way to do it. I did exactly that on my first IT job at a small shop. The shop's boss didn't want to buy another server, though, so I scrounged up a discarded PC from the back room, and set it up as a dedicated email server running Dovecot and FreeBSD 6.0 (yeah, it was a long time ago). The move was very clean. And, I used pkg at the time.
Click to expand...
The problem with this suggestion is that I'm trying to do LESS work, not MORE. I'm 74 and retired. I don't get a dime for this work. The last thing I want to do is scrounge up a cheap server and try to replicate the setup we have. There are no old PCs laying around.
 
pschmehl said:
not mixing binary packages with portmaster-built packages.
Click to expand...

Defocusing from portmaster: if you build packages for yourself, then you should avoid mixing with packages from quarterly.

So, for example, I have:

Code: 
% uclcmd get --file /etc/pkg/FreeBSD.conf FreeBSD.url
"pkg+http://pkg.FreeBSD.org/${ABI}/latest"
%

– latest.

pschmehl said:
… Is that now a problem?

Code: 
]# pkg info -x courier
courier-authlib-base-0.71.0
courier-authlib-userdb-0.71.0
courier-imap-5.0.11,2
courier-unicode-2.1_2
Click to expand...

From what's listed, those four, I don't see a conflict.

Consider the other possible conflicts.
 
grahamperrin said:
Defocusing from portmaster: if you build packages for yourself, then you should avoid mixing with packages from quarterly.

So, for example, I have:

Code: 
% uclcmd get --file /etc/pkg/FreeBSD.conf FreeBSD.url
"pkg+http://pkg.FreeBSD.org/${ABI}/latest"
%

– latest.



From what's listed, those four, I don't see a conflict.

Consider the other possible conflicts.
Click to expand...
So, if I'm understanding you correctly, the problem with mixing portmaster-built ports and pkg-installed ports is that the pkg-installed ports are possibly older than the portmaster-built ports?
 
pschmehl said:
The problem with this suggestion is that I'm trying to do LESS work, not MORE. I'm 74 and retired. I don't get a dime for this work. The last thing I want to do is scrounge up a cheap server and try to replicate the setup we have. There are no old PCs laying around.
Click to expand...
In that case, try to learn about ZFS, creating snapshots and rolling them back. And also, ZFS boot environments. They make it really easy to recover from mistakes. FWIW, I'm in the process of teaching myself to do that.
 
So why exactly do you want to move from portmaster? If you just want to keep this fragile set-up going and there's no time or budget for anything else - carry on with portmaster.

Why is it taking 2 days, though? Are the machines very old/limited? Or you have a bucket-load installed? Doesn't look like a huge list from your first post.

I don't think there's going to be an easy way to go from what you have to a trouble-free low maintenance set-up without spending time & money and you don't appear to have either. So you're a bit stuck.
 
astyle said:
In that case, try to learn about ZFS, creating snapshots and rolling them back. And also, ZFS boot environments. They make it really easy to recover from mistakes. FWIW, I'm in the process of teaching myself to do that.
Click to expand...
I'm going to be frank with you. ZFS is beyond my understanding or capability. I simply don't understand how it works. (And I've read a lot about it.)
 
One has to weigh the cost of having a solid backup and recovery strategy against the cost of not having it, factoring in the cost of allowing the whole system to fail irrecoverably and become unusable. Everything has a cost.
 
richardtoohey2 said:
So why exactly do you want to move from portmaster? If you just want to keep this fragile set-up going and there's no time or budget for anything else - carry on with portmaster.

Why is it taking 2 days, though? Are the machines very old/limited? Or you have a bucket-load installed? Doesn't look like a huge list from your first post.

I don't think there's going to be an easy way to go from what you have to a trouble-free low maintenance set-up without spending time & money and you don't appear to have either. So you're a bit stuck.
Click to expand...
Every time I run portmaster (which is infrequently) I run into problems with ports that won't build for one reason or another. Then I have to google to chase down the problem and get it fixed so I can find the next port that won't build. The equipment isn't that old, but it's definitely not new. The mail server is a dual core opteron with 8 MB of memory and 125GB hard drive. The web server is newer.

pkg is much faster, but you're stuck with the options someone else chose.

When I run portmaster, I use screen. I get it started and watch it for errors until I go to bed. Then the next day I reattach and I do some more. Finally, but the third day I'm usually done. I'm just looking for a quicker way to get up to date without blowing up the main apps I run. PHP is always a problem, because pkg uninstalls extensions that I need.

Frankly, I'm beginning to think it's time to hand all this off to a hosting company and let them handle all the backend work. I just hate leaving FreeBSD because it works so well.
 
Vull said:
One has to weigh the cost of having a solid backup and recovery strategy against the cost of not having it, factoring in the cost of allowing the whole system to fail irrecoverably and become unusable. Everything has a cost.
Click to expand...
Well, sometimes people can't afford to go the dentist, so their teeth rot. Not everyone can afford the best of everything. For backups, I use scripts that upload the files to Dropbox and delete them after seven days. So, I always have one week's worth of the most recent backups. Yet, it's not ideal, but it also doesn't cost much.
 
pschmehl said:
Well, sometimes people can't afford to go the dentist, so their teeth rot. Not everyone can afford the best of everything. For backups, I use scripts that upload the files to Dropbox and delete them after seven days. So, I always have one week's worth of the most recent backups. Yet, it's not ideal, but it also doesn't cost much.
Click to expand...
Sounds to me like a backup and recovery strategy. Here's hoping your upgrade goes well.
 
Keep using ports only. Make a full backup, restore this backup on your computer inside virtual machine and test the upgrade process there while reading /usr/ports/UPDATING and taking notes of everything during the test update. Then perform the actual update on your production server.
 
I feel your pain. At work we have a build server and our own package repository and builds can very take a long time on a system with 16 cpus!

I also know that the ports system is broken. How to fix it, I do not know, it's above my station and frankly I don't care.

Let me give you an example: I recently wanted to build doxygen and so issued make config-recursive. After about 20 minutes of answering questions for over 50 ports I abandoned it when it said it needed to build gcc!! Seriously, it wanted to build gcc!. This is by no means an isolated or unique incident.

I then downloaded doxygen, fixed two things with the CMakesfiles.txt to get it to build and built it in 15 minutes, no 50+ dependencies except flex & bison & maybe 1 other. (This was on an Armv7 device as well).

So, I'm convinced on this: ports is broken. Use pkg only. Always!
 
mark_j said:
I feel your pain. At work we have a build server and our own package repository and builds can very take a long time on a system with 16 cpus!

I also know that the ports system is broken. How to fix it, I do not know, it's above my station and frankly I don't care.

Let me give you an example: I recently wanted to build doxygen and so issued make config-recursive. After about 20 minutes of answering questions for over 50 ports I abandoned it when it said it needed to build gcc!! Seriously, it wanted to build gcc!. This is by no means an isolated or unique incident.

I then downloaded doxygen, fixed two things with the CMakesfiles.txt to get it to build and built it in 15 minutes, no 50+ dependencies except flex & bison & maybe 1 other. (This was on an Armv7 device as well).

So, I'm convinced on this: ports is broken. Use pkg only. Always!
Click to expand...
Probably one of the deps wasn't able to build without gcc for whatever reason (some packages have this issue, being there, done that) or maybe a more recent version is able to build without it and the maintainer didn't tested again if it builds without gcc.
 
You are forced to use ports when you need a custom option which is not build by default configuration of the port which is valid only for prebuild pkg.
 
mark_j If you are building from ports then, if gcc is a dependency, don't be surprised when it needs to be built. It doesn't mean ports is broken. I also question your need to answer questions when you were using config-recursive.
 
pschmehl said:
… a strictly volunteer, bandaid and bailing wire setup.
Click to expand...

pschmehl said:
… no old PCs laying around.
Click to expand...

pschmehl said:
… 125GB hard drive. …
Click to expand...

Could a small drive be added?

A 16 GB USB flash drive might be ample, although in your volunteer scenario you might find someone to donate an old spare hard drive (small HDDs are throwaway items, these days).

You could:
  1. use gpart(8) or whatever to create a single ZFS partition e.g. /dev/da0p1
  2. create a ZFS pool e.g. zpool create portsdrive da0p1
  3. install ports-mgmt/poudriere-devel
  4. change three lines in a preconfigured poudriere.conf
  5. poudriere ports -c
  6. poudriere jail -c -j thirteen -v 13.0-RELEASE
  7. poudriere ports -u
  8. poudriere bulk -v -j thirteen mail/courier-imap
That's mail/courier-imap as one example; build as few or as many ports as you want. The speed with which poudriere can build things into a repository will be a breath of fresh air to you.

Steps (1)–(6) can be one-off, need not be repeated. Step (7) whenever you want to update the ports tree.

For step (4), these three lines in your /usr/local/etc/poudriere.conf:

Code: 
ZPOOL=portsdrive
DISTFILES_CACHE=/usr/ports/distfiles
PACKAGE_FETCH_BRANCH=latest

Your /usr/local/etc/pkg/repos/poudriere.conf:

JSON: 
{
    "poudriere": {
        "url": "file:///usr/local/poudriere/data/packages/thirteen-default",
        "enabled": true
    }
}

As a side note, that's ZFS at its simplest; no learning curve. (Just be careful about specifying the partition to give to ZFS. da0p1 above is just an example.)

PS corrected a handful of typos above.


pschmehl said:
… pkg-installed ports are possibly older than the portmaster-built ports? …
Click to expand...

True. Compare, for example, the latest and quarterly columns at <https://www.freshports.org/lang/gcc/#packages>.

Whilst things such as pkg-install(8) are reasonably good at working with dependencies, mixing latest with quarterly will make things unnecessarily difficult. If you're accustomed to using any utility to build and install from ports – from latest – then (on the same computer) constraining /etc/pkg/FreeBSD.conf to quarterly is unlikely to add value; change it to latest.
 
pschmehl said:
Backups are a separate issue. There is no money for software.
Click to expand...
Backup is a concept, not a piece of software. There's nothing more needed than disk space on another machine. Same for a virtual machine to play araound with those updates (and maybe step back an try another way again). Otherwise you're playing russian roulette with a live systems - may cost much more than some time and disk space.

I've seen far to many companies with not the smallest two-digit million euro turnovers without backups and without development machine losing complete servers. Said "hey, we have raid", and the technician took out the wrong of the two remaining disks after noticing all other disks died a long time ago (and the last two also sounding … weird). Or crashing raid controllers - can be fun, too. And I never get the "why". No backup, no mercy.

You've (or your company) decided to go high risk. So you're even going high risk on updates.
 
You must log in or register to reply here.
Back
Top