How to keep some ports and many binary packages up to date

daudo

daudo

Hi,

I am quite new to FreeBSD, so please excuse, if this is a FAQ, but unfortunately my research so far hasn't made me 100% happy and so I decided to try here :)

Most of my packages are pkg installed binaries, yet some are ports based.

In general, I am aware how to keep both pkg based and ports based packages up to date, but unfortunately I fail to understand, how to prevent pkg based updates from breaking my ports packages.

So lets assume that all but one packages are pkg based and I want to update my pkg based packages.

Situation *before* update:

PackageA:
- installed from ports, version 1.0
- pkg knows the package as well, with version 1.0

Let's assume that sometime after my last update, a new version 1.1 of PackageA is published to the pkg repositories.

If I later decide to update my system, I run:
pkg update
pkg upgrade

Among the other updates, this would install PackageA version 1.1 from the binary repositories, which is exactly what I need to prevent.

From http://unix.stackexchange.com/questions/110167/how-to-update-a-mix-of-packages-and-ports-on-freebsd I learned, that the best way to prevent Package A from being updated by pkg seems to be to lock it using pkg lock.

This works to a certain degree, but unfortunately this locks the package for any other ports oriented tool as well (like portupgrade) and they complain, that the package is locked.

While unlocking it and rerunning portupgrade again might work for one or only a few packages, it becomes difficult, if you have a significant number of ports based packages that you had to unlock each time you update.

So can I configure pkg not to interfere with my ports based packages and vice versa?
 
Last edited by a moderator:
daudo said:
So can I configure pkg not to interfere with my ports based packages and vice versa?
Click to expand...
No, because there is no difference between a port and package once they're installed.

The best solution is to set up your own repository using ports-mgmt/poudriere. That way you get the best of both worlds, the configuration options of ports and the ease of management of packages.
 
Perhaps a request for an extra feature to portmaster/portupgrade is in order: a flag to say "if the pkg is locked, go ahead and automatically unlock it since I really want to update it".
 
What's stopping you from unlocking it yourself beforehand?
 
Nothing at all. I just think an option like that would provide a better integration of the Portmaster/Portupgrade tools with the underlying package installation tool, and a way to enforce the -f option of those tools without having to do an extra step.
 
I can unlock myself beforehand, but like I said, this is quite an effort if you have more than just a few ports based packages. Unlocking (and relocking) has to be done before/after each update, so it would indeed be nice to have this "somehow" automated.

BTW, after closer inspection on what pkg info tells me about a package, it seems that it can differentiate between pkg binary packages and ports packages:

info for dovecot2, one of my ports packages:
Code: 
% pkg info dovecot2
[...]
Annotations  :
   cpe  : cpe:2.3:a:dovecot:dovecot:2.2.21:::::freebsd10:x64:1
[...]

info for vim, pkg based:
% pkg info vim
[...]
Annotations  :
   cpe  : cpe:2.3:a:vim:vim:7.4:::::freebsd10:x64
   repo_type  : binary
   repository  : FreeBSD
[...]
So as it seems, the "repo_type" annotation is populated with "binary" when you install a package using pkg(8) and it remains unpopulated, when you build & install a ports package.

I will investigate ports-mgmt/poudriere however, maybe that's the answer to my problem :)
 
Last edited by a moderator:
You are the target audience of ports-mgmt/synth.
locking/unlocking doesn't work well in practice

ljboiler said:
Nothing at all. I just think an option like that would provide a better integration of the portmaster/portupgrade tools with the underlying package installation tool, and a way to enforce the -f option of those tools without having to do an extra step.
Click to expand...
It's a fairy tale I think. Until yesterday Portmaster didn't have a maintainer. Now he's going to be consumed with just getting it maintained. I don't think new features are coming for either. Does Portmaster even interact directly with pkg(8)?

SirDice prefers Poudriere which is a fine tool, but he should be suggesting Synth at the same time whenever he suggests Poudriere rather than not mention it. It's mature enough to mention since it's going to be releases at 1.00 any day now.
 
marino@ said:
SirDice prefers poudriere which is a fine tool, but he should be suggesting synth at the same time whenever he suggests poudriere rather than not mention it. It's mature enough to mention since it's going to be releases at 1.00 any day now.
Click to expand...
I'm sure it is, but I can't recommend something I've never used myself and have no experience with ;)
 
daudo said:
I can unlock myself beforehand, but like I said, this is quite an effort if you have more than just a few ports based packages. Unlocking (and relocking) has to be done before/after each update, so it would indeed be nice to have this "somehow" automated.
Click to expand...
It would defeat the whole purpose of the lock if applications can simply override it at will. So I think this is a bad idea.
 
Back on focus to the original question:
How to keep some ports and many binary packages up to date?
Click to expand...

I had exactly the same question more than a year ago, and others asked for this as well. The answers to use locking somehow or to setup ports-mgmt/poudriere didn't really satisfy my needs. And ports-mgmt/synth also doesn't seem to solve the issue by 100% as it still builds the binaries from the ports.

I WANT to update hundreds of unmodified vanila stock binaries with pkg upgrade and the irreducible rest having customized built settings by source using ports-mgmt/portmaster. This is the question, and the answers poudriere and synth(1) are perhaps helpful in other respects, but not exactly for the given question, because somewhere in YOUR system all the updates are actually built.

I resolved the issue for me by using a simple shell script:
Code: 
#!/bin/sh

/usr/sbin/portsnap fetch update

portslist=\
"archivers/php56-zip\
databases/php56-pdo\
databases/php56-pdo_pgsql\
databases/php56-pgsql\
converters/php56-iconv\
converters/php56-mbstring\
devel/php56-json\
devel/subversion\
games/minecraft-server\
lang/php56\
mail/dovecot2\
mail/postfix\
mail/roundcube\
net/mDNSResponder\
net/netatalk3\
net/samba42\
security/cyrus-sasl2\
security/php56-filter\
security/php56-hash\
security/php56-openssl\
sysutils/php56-fileinfo\
textproc/php56-dom\
textproc/php56-xml\
www/apache24\
www/mod_php56\
www/php56-session\
www/squid"


/usr/sbin/pkg update

pkgslist=`/usr/sbin/pkg info -oa | /usr/bin/cut -f2 -w`
upgrdlist=""
excldlist=""
for pkg in $pkgslist ; do
   for port in $portslist ; do
      if [ "$port" == "$pkg" ] ; then
         continue 2
      fi
   done

   upgrdlist="$upgrdlist $pkg"
   excldlist="$excldlist -x $pkg"
done


# upgrade binary packages
if [ "$upgrdlist" != "" ] ; then
   /usr/sbin/pkg upgrade -U $upgrdlist
else
   echo "All installed packages are up-to-date."
fi

# upgrade the ports
/usr/local/sbin/portmaster --no-term-title --update-if-newer $excldlist -yBD $portslist

# cleaning up
/usr/local/sbin/portmaster -y --clean-distfiles
The basic idea is to positively identify the few ports that got customized built settings by $portslist. The complement of this list is compiled into two other lists of sightly different formats, namely $upgrdlist and $excldlist, and these lists are actually feeded into pkg(8) and portmaster(8) respectively.

I use this script on a weekly basis, and it works very well. In order to keep the version differences between port sources and binaries small, I added the file /usr/local/etc/pkg/repos/FreeBSD.conf with the directive telling pkg(8) to use the latest packages instead of the quarterly builds:
Code: 
FreeBSD: {
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest"
}
 
SirDice said:
I'm sure it is, but I can't recommend something I've never used myself and have no experience with ;)
Click to expand...
Of course I would not expect you to recommend it -- only mention that is an option (one that you have no experience with if you like)

I've said elsewhere that Synth is not aimed at current Poudriere users. If they are happy with Poudriere then there is no need to change tools. However, I would say Synth could be recommended for somebody new looking to make a switch to a repository builder. Poudriere is aimed at clusters (in my opinion, that's the primary purpose) while ports-mgmt/synth is aimed at regular users.
 
obsigna said:
I WANT to update hundreds of unmodified vanila stock binaries with pkg upgrade and the irreducible rest having customized built settings by source using ports-mgmt/portmaster. This is the question, and the answers poudriere and synth are perhaps helpful in other respects, but not exactly for the given question, because somewhere in YOUR system all the updates are actually built.
Click to expand...

The truth is that Poudriere and Synth are the tools that are appropriate for this task, not Portmaster. It's like saying you want to use a VW beetle to plow a field, that's what you really want, even though a tractor is the correct tool. So respectfully, I think you are wrong, it's the exact correct answer to that given question. Actually, Poudriere isn't set up to use prebuilt packages at all so you'd have to use multiple repositories and let pkg(8) figure it out whereas Synth figures it out and let's pkg(8) just install for it's repository. Both approaches would work to combine vanilla + custom packages WITHOUT the locking (which will break at some point)
 
marino@ said:
Of course I would not expect you to recommend it -- only mention that is an option (one that you have no experience with if you like)

I've said elsewhere that synth is not aimed at curent poudriere users. If they are happy with poudriere then there is no need to change tools. However, I would say synth could be recommended for somebody new looking to make a switch to a repository builder. Poudriere is aimed at clusters (in my opinion, that's the primary purpose) while synth is aimed at regular users.
Click to expand...

I am a FreeBSD user from version 6? and all the time I use Portmaster but when Synth came in the ports I gave it a try and I love it. It works on my system very good and I don't have any problems. And BTW I am no computer educated.
 
marino@ said:
The truth is that poudriere and synth are the tools that are appropriate for this task, not portmaster. It's like saying you want to use a VW beetle to plow a field, that's what you really want, even though a tractor is the correct tool. So respectfully, I think you are wrong, it's the exact correct answer to that given question. Actually, poudriere isn't set up to use prebuilt packages at all so you'd have to use multiple repositories and let pkg(8) figure it out whereas synth figures it out and let's pkg(8) just install for it's repository. Both approaches would work to combine vanilla + custom packages WITHOUT the locking (which will break at some point)
Click to expand...

Actually my script is proven to do exactly what I want (WITHOUT the locking), so how can I be wrong? Respectfully, your comparison between a VW beetle and a tractor does not fit quite well. IMHO, You and SirDice suggest to take a sledgehammer to crack a nut.
 
obsigna said:
Actually my script is proven to do exactly what I want (WITHOUT the locking), so how can I be wrong? Respectfully, your comparison between a VW beetle and a tractor does not fit quite well. IMHO, You and SirDice suggest to take a sledgehammer to crack a nut.
Click to expand...
Your script might look nice and working but eventually it will create problems in your software. You do not take into account the /UPDATING which is the main reason why mixing ports and packages is a wrong recipe.
 
gkontos said:
... You do not take into account the /UPDATING which is the main reason why mixing ports and packages is a wrong recipe.
Click to expand...
My script is not meant to be run by a cron job, and nothing prevents me to look into /usr/ports/UPDATING, if the script bails out. Actually, I run pkg updating -d `date -v-2w +%Y%m%d` quite frequently.

Anyway, I agree that the script leaves room for improvements, for example I put said pkg updating together with a (y/n) confirmation prompt into it before it starts upgrading packages and ports.
Code: 
#!/bin/sh

portslist="\
archivers/php56-zip \
databases/php56-pdo \
databases/php56-pdo_pgsql \
databases/php56-pgsql \
converters/php56-iconv \
converters/php56-mbstring \
devel/php56-json \
devel/subversion \
games/minecraft-server \
lang/php56 \
mail/dovecot2 \
mail/postfix \
mail/roundcube \
net/mDNSResponder \
net/netatalk3 \
net/samba42 \
security/cyrus-sasl2 \
security/php56-filter \
security/php56-hash \
security/php56-openssl \
sysutils/php56-fileinfo \
textproc/php56-dom \
textproc/php56-xml \
www/apache24 \
www/mod_php56 \
www/php56-session \
www/squid"

/usr/bin/printf "Updating the ports tree...\n"
/usr/sbin/portsnap fetch update
/usr/sbin/pkg version -v
/usr/sbin/pkg updating -d `date -v-2w +%Y%m%d`

/usr/bin/printf "\nDo you want to continue (y/n)? "
save_stty_state=$(stty -g); stty raw -echo; answer=$(head -c 1); stty $save_stty_state
if echo "$answer" | grep -iq "^y" ; then
   /usr/bin/printf "\n\n"
   /usr/sbin/pkg update

   pkgslist=`/usr/sbin/pkg info -oa | /usr/bin/cut -f2 -w`
   upgrdlist=""
   excldlist=""
   for pkg in $pkgslist ; do
      for port in $portslist ; do
         if [ "$port" == "$pkg" ] ; then
            continue 2
         fi
      done

      upgrdlist="$upgrdlist $pkg"
      excldlist="$excldlist -x $pkg" 
   done

   /usr/bin/printf "\nUpgrading binary packages...\n"
   if [ "$upgrdlist" != "" ] ; then
      /usr/sbin/pkg upgrade -U $upgrdlist
   else
      echo "All installed packages are up-to-date." 
   fi

   /usr/bin/printf "\nUpgrading the ports...\n"
   /usr/local/sbin/portmaster --no-term-title --update-if-newer $excldlist -yBD $portslist

   /usr/bin/printf "\nCleaning up...\n"
   /usr/local/sbin/portmaster -y --clean-distfiles

else

   /usr/bin/printf "\n\n"

fi
Regarding mixing ports and packages, I would like to remember to what SirDice said in one of his previous messages:
SirDice said:
..., because there is no difference between a port and package once they're installed. ...
Click to expand...

Note also, that it does not exactly disturb me when during a port update a dependent port gets updated by source because it didn't made it yet into the package repository. Usually, I let it go. If I am in a hurry, I cancel port updating, and run the script the other day again, so the dependency can be resolved from an updated package repository.
 
obsigna said:
Regarding mixing ports and packages, I would like to remember to what SirDice said in one of his previous messages:
Click to expand...
The reason it's not advised to mix ports and packages is because packages (the official ones) are built using the default options whereas ports are not. Using ports more or less implies you deviate from the default. Mixing various default settings can and will cause problems. When a package depends on, for example, MySQL 5.5 and a port is built for MySQL 5.6 you're going to have a hard time getting the dependencies straight. Hence the advice not to mix them.
 
SirDice said:
The reason it's not advised to mix ports and packages is because packages (the official ones) are built using the default options whereas ports are not. Using ports more or less implies you deviate from the default. Mixing various default settings can and will cause problems. When a package depends on, for example, MySQL 5.5 and a port is built for MySQL 5.6 you're going to have a hard time getting the dependencies straight. Hence the advice not to mix them.
Click to expand...
Of course, the mixture must be chosen with care, however, once this is done, it can easily be maintained by the script. For example, I configured apache24 with the threaded worker MPM, and for this reason, I need a threaded mod_php56 build, and that implies that php56 itself and all of its modules are build from sources as well, even, with the standard settings – and for this reason the whole php56-hell is in the $portslist of my script. In the case of your MySQL example, I need to put the package that by default depends on MySQL55 into the $portslist as well, in order it gets build from the sources against MySQL56. Anyway, these kind of choices don't need to be changed everyday. In this respect gkontos is right, /usr/ports/UPDATING needs to be checked for relevant instances.
 
I have been using FreeBSD since 6.0-RELEASE and I have always used ports. The new package manager system combined with Poudriere has given me the opportunity to deliver reliable package updates to my servers. I also mentioned in another thread that Synth is an excellent tool that does not require the overhead and complexity of Poudriere, making it ideal for standalone systems.
In my experience, mixing ports with packages is a bad idea. The main reason is that the upgrade procedure is totally different. Given the fact that you only need to build everything once, I don't see any reason why someone would take that risk.
 
gkontos, if I would need to maintain software installations on several servers, let's say at least one more than one, then of course I would choose ports-mgmt/poudriere. For maintaining the software installation updated on a single machine, Poudriere and Synth are the total overkill. From 2009 to 2014, I simply used portmaster for everything, then I successfully switched to the mixed maintenance, i.e. 27 ports from source out of 229 packages in total, using a simple and comprehensible script, which does nothing strange behind the scenes. That makes my solution completely on-topic, while Poudriere and Synth are perhaps somehow related, aren't it?

Risk? FUD, nothing else!
 
Synth is aimed at single servers as well.)
It does what you attempt to do correctly with no risk.
You are free to choose to believe that your technique (which requires a hell of a lot more work than a single synth prepare-system) is equivalent or superior, but it's just not.
There are technical reasons why portmaster building is described both as "unsafe" and "dirty".

So with two alternatives:
1A) issue single install command, come back and it's done *correctly*
1B) issue single prepare command, come back and manually run pkg install, and it's done *correctly*
2) Do all the work involved in your process and assume risk and build under conditions described as unsafe.

Why would anybody with their ego not vested do option #2?
What are you trying to save? Time? (why it is time critical, come back in the morning) Bandwidth? (Is this significant)? Why would somebody do a lot more effort for a result they can't (or at least *shouldn't*) be confident in?
 
it is possible, obsigna, that you don't know that Synth has an option to download and use packages built by FreeBSD instead of building all them. It automatically checks and if the remote package is deemed suitable, it gets fetched instead of built. So it's capable of this "mixed" mode as a standard option. So again, if you have that with a single command, why would anybody try to recreate that manually with a script and the additional work that goes along with it.
 
Thanks for all the suggestions. I must admit, that my very first impulse was to produce a shell script much like obsigna did. Sounds nice in terms of KISS, but I am afraid that this approach might break if applied to many servers.

I will give both proderie and synth a try. From what I have now read about them, both seem to be as close to a "best practice" as can be for situations like the growing numbers of FBSD servers soon to be managed.
 
So today I spent some time playing with ports-mgmt/synth and so far I am quite impressed.

Two questions, however:

* I see that synth produces a separate repository in /var/synth/live_packages/. I guess I this is also intended to be served to other servers?

* And if I see it correctly, you can only build packages/ports for the host OS version where synth is running. Is that correct? Background is that eventually I may have different FreeBSD versions to provide packages for.
 
You must log in or register to reply here.
Back
Top