Dual Mode Maintenance of Ports and Packages

[obsigna]

I am operating a FreeBSD 9.3-RELEASE headless server, having 212 ports installed from source and maintained via ports-mgmt/portmaster.

• Most of these ports are dragged-in dependencies, and for these I always leave the configure options at there respective default values.
• Most of the ports that were requested to being install are also built using the default configure options.
• I need a handful (ca. 5) of ports having customized configure options.
• In two additional ports, I placed local patch files in order to modify certain functionalities to my needs.

Ideally, I would like to maintain only the 7 ports having customized options and/or local patches using ports-mgmt/portmaster, and the other 195 by pkg upgrade.

For experimenting with the Dual Mode Maintenance, I set up a FreeBSD 10-RELEASE test system, that got for the time being much less ports/packages installed. My approach so far is to maintain the small list of customized/patched ports in a text file, and in the update cycle feeding this list into portmaster --update-if-newer, instead of utilizing the -a command line option. Then, in a second step, I run pkg upgrade, expecting that it leaves the ports already updated by portmaster alone. Instead it is always complaining about that port on the test system that got local patches applied:
# pkg upgrade

Updating repository catalogue
FreeBSD repository is up-to-date
All repositories are up-to-date
Checking for upgrades (17 candidates): 100%
Checking integrity... done (0 conflicting)
The following 1 packages will be affected (of 0 checked):

Installed packages to be REINSTALLED:
	mDNSResponder-544_2 (needed shared library changed)

The process will require 318 B more space

Proceed with this action [y/N]: n

I guess, what I am looking for is a way to tell ports-mgmt/pkg to ignore/exclude some certain ports in the course of its upgrade processing, however, I cannot find that kind of feature of pkg. Does this feature not exist? Is there another way? Anyway, is Dual Mode Maintenance feasible without major hassle?

 

[obsigna]

See pkg() or pkg help lock.

Yes, that did trick, as far as pkg upgrade is concerned. Many thanks for your quick reply.

I assume, that a lock is effective also in the course of portmaster updates, and that I need to unlock the packages in question before running the source updates and lock it again before running the succeeding binary updates. Anyway, this would not impose a big problem, since it can easily be scripted.

I will check the whole procedure in the course of the next days. Today I ran already the source update and the binary upgrade sequences, and therefore, portmaster will for the time being find nothing to re-install.

 

[obsigna]

craftsman's toolbox gets bigger ...
just for completness: Portmaster has the option -x for excluding.

Ha, I saw this already, however, wanting to be a good craftsman :-D, I chose the other tool, namely positively defining 7 ports to include, instead of 195 ports to exclude.

Many thanks again!

 

[obsigna]

Unfortunately pkg lock <pkg-name> does locking too much for my purpose.

For example I got a custom net/samba41 installed, and I left all the ports it depends on with the respective default options. The idea is to maintain samba41 together with some other ports with customized build options using portmaster --no-term-title --update-if-newer -yBD <LIST OF CUSTOMIZED PORTS>. Then I lock these ports in order to call pkg upgrade for everything else, and the customized ports are not upgraded, OK - as expected, however the dependencies are not upgraded either, not OK! So, pkg (un)locking is not exactly what I need.

If I unlock samba41, then pkg upgrade wants to install the default version together with all the dependency bloat:

Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
The following 25 packages will be affected (of 0 checked):

New packages to be INSTALLED:
	cups-client: 1.7.3_2
	avahi-app: 0.6.31_2
	expat: 2.1.0_1
	gnome_subr: 1.0
	pcre: 8.35
	libdaemon: 0.14_1
	gobject-introspection: 1.36.0_2
	gnomehier: 3.0
	glib: 2.36.3_3
	libiconv: 1.14_3
	dbus-glib: 0.100.2_1
	dbus: 1.8.6
	libX11: 1.6.2_2,1
	xproto: 7.0.26
	libxcb: 1.10_2
	libXdmcp: 1.1.1_2
	libXau: 1.0.8_2
	libxml2: 2.9.1_1
	libpthread-stubs: 0.3_5
	kbproto: 1.0.6
	libSM: 1.2.2_2,1
	libICE: 1.0.9,1
	gdbm: 1.11_2
	gamin: 0.1.10_7

Installed packages to be REINSTALLED:
	samba41-4.1.11 (options changed)

The process will require 113 MB more space.
29 MB to be downloaded.

Proceed with this action? [y/N]: N

In order to circumvent the problem, I wrote the following shell script, that I call for semi-automatic dual mode updating of ports/pkgs :

#!/bin/sh

portslist="net/mDNSResponder\
           net/samba41\
           net/netatalk3\
           security/ipsec-tools"

pkgslist=`pkg info -oa | cut -f2 -w`

for port in $portslist ; do
   pkg unlock -qy $port
done

portmaster --no-term-title --update-if-newer -yBD $portslist

upgrdlist=""
for pkg in $pkgslist ; do
   for port in $portslist ; do
      if [ "$port" == "$pkg" ] ; then
         continue 2
      fi
   done

   upgrdlist="$upgrdlist $pkg" 
done

pkg upgrade $upgrdlist

for port in $portslist ; do
   pkg lock -qy $port
done

This seems to work quite well, for the ports/pkgs that I installed so far on the test box.

I still use pkg (un)locking in order to prevent to pkg upgrade a customized port by accident.

 

[pvanulden]

This seems like a pretty big over sight? I mean, everything works great if you are just using the default binary packages but, like the OP, I too have had to do a few customizations which required building the port. Hopefully in future there will be some way to mark a package source as binary or built from ports and then have pkg and portmaster key on that.

 

[SirDice]

I highly recommend setting up your own repository using ports-mgmt/poudriere. Even for one server this will pay off. The entire building is done "offline" so you're not touching your production machine. Any build issues can be resolved without stress. Options can be set on every port and you still have the benefit of using packages. No build dependencies are installed on your production machine and you get to keep track of when to update the ports tree and/or your own packages. Best of both worlds really.

 

[arp242]

I highly recommend setting up your own repository using ports-mgmt/poudriere. Even for one server this will pay off. The entire building is done "offline" so you're not touching your production machine. Any build issues can be resolved without stress. Options can be set on every port and you still have the benefit of using packages. No build dependencies are installed on your production machine and you get to keep track of when to update the ports tree and/or your own packages. Best of both worlds really.

I find it a sad state of affairs that we need to resort to this simply because of a defective package manager.

All that is required for this to work, is a "don't update dependencies unless strictly required"-option... This would solve the entire issue, and doesn't force people with a single home server (ie. me) to use these extensive workarounds (or ditch binary packages and spend several hours building from source, which is an even sadder state of affairs).

 

[ANOKNUSA]

All that is required for this to work, is a "don't update dependencies unless strictly required"-option... This would solve the entire issue, and doesn't force people with a single home server (ie. me) to use these extensive workarounds (or ditch binary packages and spend several hours building from source, which is an even sadder state of affairs).

Once ports-mgmt/portmaster gets updated to fully utilize pkg (and I imagine it will), it'll be possible to use binary packages for build dependencies, and only build the target port and run-time dependencies from source. ports-mgmt/poudriere also has several features that cut down on compile times, like parallel build processes.

Yes, I'd personally like to see an option for pkg which skips over any packages with custom options during an upgrade. Yet one has to wonder: If pkg is capable of knowing that certain installed packages have build-time options that differ from the defaults, doesn't that suggest that the devs have already considered this and are likely to implement it in the future? What other purpose could that functionality serve?

 

[SirDice]

Yet one has to wonder: If pkg is capable of knowing that certain installed packages have build-time options that differ from the defaults, doesn't that suggest that the devs have already considered this and are likely to implement it in the future? What other purpose could that functionality serve?

Not exactly. It detects the options have changed compared to the version that's installed. But yes, it's one of those things that are planned but not yet implemented. There's still quite a lot of work being done on it. You can tell by the breakneck speed of the amount of updates to pkg(8) itself.