Using packages and ports at the same time?

HL1234

Active Member

Reaction score: 8
Messages: 133

Hello folks,
I'm using FreeBSD as email and web server - no desktop, no users, weak hardware - but it works since a long time.

I'm using mixing ports and packages for installing my applications. Because of the weak hardware I won't install everything over the ports, just the software for eMail (like sendmail, etc. and web server like apache, php-extensions, etc..)

I get a question, because I run into a lot of problems every time I'm doing updates:

After some times, there are ports sign as "vulnerable" for hacking - so I get something to do.
I do
cd /usr ports and
portsnap fetch update
or / and
pkg update

Then I check new versions with

pkg version -l "<"
or
pkg upgrade -n

I have try to update with
portupgrade -r abc.. or every package it needs an update and when fail
portupgrade -a and the rest with
portupgrade -r abc.. or pkg install abc... until everything is new.
By the way, it take me sometimes days to solve that.

or with
pkg install abc... for every package it needs update or when fail
pkg install upgrade

Every time I got a lot of conflicts and every time I must work through my documentation to reconfigure the eMail and webserver settings until they works again.

So, ..... Can I have packages and ports Installations at the same time? And how do I manged them without get conflicts.

Can I configure the package system that it don't install applications that I explicit install about the ports?

Thank you for your ideas!
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 9,783
Messages: 34,911

So, ..... Can I have packages and ports Installations at the same time? And how do I manged them without get conflicts.
You can but you have to be extremely diligent as port options that may have been enabled may be overuled by packages (which are always built using the default options).

The best solution really is to set up your own repository with ports-mgmt/poudriere. That way you have the flexibility of ports and the ease of management of packages.

Can I configure the package system that it don't install applications that I explicit install about the ports?
No, because there's no difference between a package and port (from the system's point of view). A port builds a package and it's that package that gets installed.
 

Dies_Irae

Active Member

Reaction score: 58
Messages: 137

No, because there's no difference between a package and port (from the system's point of view). A port builds a package and it's that package that gets installed.

What about pkg-lock(8)? He can keep a package locked to prevent updating, and use pkg-unlock(8) when he wants to eventually update the package using ports.
But I never tried this, it's just an idea - maybe a wrong one.
 

obsigna

Daemon

Reaction score: 743
Messages: 1,158

...
Can I configure the package system that it don't install applications that I explicit install about the ports?

I also was looking for Dual Mode Maintenance of Ports and Packages. I ended up writing my own updating shell script which works pretty well for some time now. The basic idea was exactly what you asked for, namely, explicitly identify the ports that should be updated from the configurable sources and not by the way of statically configured binary packages.

The most recent incarnation of this script is:
Code:
#!/bin/sh

portslist=\
"archivers/php55-zip\
databases/php55-pdo\
databases/php55-pdo_pgsql\
databases/php55-pgsql\
converters/php55-iconv\
converters/php55-mbstring\
devel/php55-json\
devel/pecl-intl\
devel/subversion\
lang/php55\
mail/dovecot2\
mail/postfix\
mail/roundcube\
multimedia/libdvdcss \
net/mDNSResponder\
net/netatalk3\
security/cyrus-sasl2\
security/php55-filter\
security/php55-hash\
security/php55-openssl\
security/strongswan\
sysutils/apcupsd\
sysutils/php55-fileinfo\
textproc/php55-dom\
textproc/php55-xml\
www/apache24\
www/mod_php55\
www/php55-session\
www/squid"

pkg update

pkgslist=`pkg info -oa | cut -f2 -w`
upgrdlist=""
excldlist=""
for pkg in $pkgslist ; do
   for port in $portslist ; do
      if [ "$port" == "$pkg" ] ; then
         continue 2
      fi
   done

   upgrdlist="$upgrdlist $pkg"

   pkg=`echo $pkg | cut -f2 -d / | cut -f1 -d .`
   excldlist="$excldlist -x $pkg"
done


# upgrade binary packages
if [ "$upgrdlist" != "" ] ; then
   pkg upgrade -U $upgrdlist
else
   echo "All installed packages are up-to-date."
fi

# upgrade the ports
portmaster --no-term-title --update-if-newer $excldlist -yBD $portslist

This script is the result of a lot of trial and errors. I also started with pkg lock, but it turned out that this locks also all dependencies of a port, most of which could be updated using binary packages. In addtion, it is important to upgrade the binary packages, i.e. those in the $upgrdlist, before updating the ports in the $portslist. Finally, it turned out, that the script needs to compile an $excldlist for the portmaster command. Otherwise portmaster would update packages that are already new in the ports but still didn't made it into the binary package repository.
 

kpa

Beastie's Twin

Reaction score: 1,831
Messages: 6,318

What about pkg-lock(8)? He can keep a package locked to prevent updating, and use pkg-unlock(8) when he wants to eventually update the package using ports.
But I never tried this, it's just an idea - maybe a wrong one.

That's still only bandaid for a bigger problem that is the hard set-in-stone dependencies that is the only way to express dependencies at the moment. You can lock certain packages so that they won't be replaced but eventually you will run into a situation where the dependency solver can not find any solutions because of conflicting demands. There is work going on to implement so called abstract dependencies with a PROVIDES/REQUIRES mechanism (like what Debian's DPKG does) but that's going to be serious overhaul of the ports system when it's implemented.
 

obsigna

Daemon

Reaction score: 743
Messages: 1,158

The script in my previous post was not complete. By accident, I removed one line when transferring it from my machine into the code block of my forum message. The missing line was:

Code:
...
      upgrdlist="$upgrdlist $pkg"
...


I corrected the script in my post #4.
 

ANOKNUSA

Aspiring Daemon

Reaction score: 373
Messages: 675

What about pkg-lock(8)? He can keep a package locked to prevent updating, and use pkg-unlock(8) when he wants to eventually update the package using ports.

Package "locking" should ideally only be used in corner cases, as doing so requires even more vigilance than usual. Lock a package, and you'll need to lock any packages it depends on as well. Lock those dependencies, and any packages that share those dependencies may need to be locked... And so on. Slip up anywhere in the chain and you'll inevitably break something. If you have a certain package that absolutely, positively must not be upgraded at the moment, it's better to forego upgrading all ports/packages until you have more leeway (or you really, really know what you're doing).

If you want to change the options on some, but not all ports, use poudriere(8) as SirDice suggested. You'll have to build all packages from source, but it's more flexible that using only ports or only packages and pretty fast compared to other methods (it builds all packages in parallel and can build in RAM).
 
OP
HL1234

HL1234

Active Member

Reaction score: 8
Messages: 133

Thank You all!
The answer was No! but this is better then no answer. I see, I'm not alone with this problem.

The best solution I thing is the script of [USER=41855]obsigna[/USER]. I will give them a try next time after I had collected out all ports and there dependencies, I build from source.

With ports-mgmt/poudriere I have no experience. And what does that mean: "set up your own repository with it" How can I do that?

For refreshing the ports I would like further use: portsnap fetch update. Is this possible when using poudriere?
Is there any tip where I should read about it?
 

gkontos

Daemon

Reaction score: 486
Messages: 2,160

[USER=16769]HL1234[/USER], I think that instead of clarifying things we made them more complicated for you. Bottom line, use ports or packages but avoid mixing them unless you have a lot of experience. I personally don't mix them, at least on production systems.

ports-mgmt/poudriere is totally useless for a couple of servers. It is like suggesting to use puppet or chef for managing a couple of servers. Why? Because you need to have a dedicated +1 system just for this.

@All, experienced *nix administrators, site contributors. Please do not suggest over and over and over and over.... the use of **poudriere** as if it was a chewing gum.
 

Dies_Irae

Active Member

Reaction score: 58
Messages: 137

You'll have to build all packages from source, but it's more flexible that using only ports or only packages and pretty fast compared to other methods (it builds all packages in parallel and can build in RAM).
Yes, but in the end, he still need to compile from sources. Plus the effort of setting up ports-mgmt/poudriere.

@All, experienced *nix administrators, site contributors. Please do not suggest over and over and over and over.... the use of **poudriere** as if it was a chewing gum.
Totally agree.
 

ANOKNUSA

Aspiring Daemon

Reaction score: 373
Messages: 675

I'm not looking to start a fight here, but perhaps people would be less apt to recommend poudriere so much if you explained what potential detriments outweigh the potential benefits. I spent one hour setting it up half a year ago, have been using it without fail since, and near as I can tell it's the most efficient use of my system resources compared to the alternatives. I admit I probably shouldn't recommend poudriere to beginners, but anyone familiar with both the ports system and package manager can make effective use of it in any case.
 

gkontos

Daemon

Reaction score: 486
Messages: 2,160

I'm not looking to start a fight here, but perhaps people would be less apt to recommend poudriere so much if you explained what potential detriments outweigh the potential benefits. I spent one hour setting it up half a year ago, have been using it without fail since, and near as I can tell it's the most efficient use of my system resources compared to the alternatives. I admit I probably shouldn't recommend poudriere to beginners, but anyone familiar with both the ports system and package manager can make effective use of it in any case.

The issue has to do with how many servers the system will support. If someone has only one server then the cost of adding another just for the sake of building custom packages is not worth it. On the other hand, if you maintain a server farm then the benefits that you get are tremendous. So, my observation has to do with the fact that suggesting a tool like that must meet certain criteria.

Setting up poudriere is not the easiest thing but this has nothing to do with my objection.
 

PacketMan

Aspiring Daemon

Reaction score: 163
Messages: 949

I use ports exclusively now, though for most of what I do I would rather use pkg. I noticed an option with ports command line is package, and also package-recursive. So say I wanted to install a port of 5 desktop based machines, say OpenOffice for example. Could I do a make package-recursive clean which includes the install I understand. After it's finished could I copied the newly created .tbz file(s) to the remaining 4 machines then then use the pkg add to install?
 

gkontos

Daemon

Reaction score: 486
Messages: 2,160

pkg_add is deprecated. In your case the use of ports-mgmt/poudriere is highly recommended. Especially in an environment where you have many desktop machines. Unless of course you only use default port settings. In that case you can just use pkg install.
 

Dies_Irae

Active Member

Reaction score: 58
Messages: 137

I admit I probably shouldn't recommend poudriere to beginners, but anyone familiar with both the ports system and package manager can make effective use of it in any case.
In my opinion, this is the central point. If I am familiar with both the ports system and package manager, then probably I am skilled enough to resolve a conflict between ports and packages, and I don't need to setup poudriere.
But there's nothing wrong in trying new solutions.
 

kpa

Beastie's Twin

Reaction score: 1,831
Messages: 6,318

In my opinion, this is the central point. If I am familiar with both the ports system and package manager, then probably I am skilled enough to resolve a conflict between ports and packages, and I don't need to setup poudriere.
But there's nothing wrong in trying new solutions.

If you have the skills then yes go ahead and resolve the conflicts manually. However, if there is an automated way to achieve the same ends why not use it?
 

Dies_Irae

Active Member

Reaction score: 58
Messages: 137

However, if there is an automated way to achieve the same ends why not use it?
I agree, and as I said there's nothing wrong in trying new things - I often hang myself with over-complicated setups just for the shake of curiosity, and poudriere is on my "soon-or-later-I-must-try-it" list.
What I'm trying to say is that in a single-machine scenario poudriere may be simply "too much". Funny and instructive to setup as an exercise, but too much for something like this:
I'm using FreeBSD as email and web server - no desktop, no users, weak hardware
Said that, tomorrow I will try it at work! :D
 

kpa

Beastie's Twin

Reaction score: 1,831
Messages: 6,318

I agree, and as I said there's nothing wrong in trying new things - I often hang myself with over-complicated setups just for the shake of curiosity, and poudriere is on my "soon-or-later-I-must-try-it" list.
What I'm trying to say is that in a single-machine scenario poudriere may be simply "too much". Funny and instructive to setup as an exercise, but too much for something like this:

Said that, tomorrow I will try it at work! :D

You and some others are assuming that ports-mgmt/poudriere is this massive resource hog that can not be used on low power machines. This not true at all, it is mostly just a clever wrapper around the existing tools that doesn't use that much extra computing resources (jails(8) for example take almost no extra memory or CPU time) and usually the time it takes to build ports with it (especially larger batches) is very compareable to building the same ports with just make install or ports-mgmt/portmaster. Disk space is obviously a concern with poudriere but if that's a problem on the system you're using you shouldn't be building ports on the machine at all but dedicate another more suitable system for that purpose.
 

gkontos

Daemon

Reaction score: 486
Messages: 2,160

You and some others are assuming that ports-mgmt/poudriere is this massive resource hog that can not be used on low power machines. This not true at all, it is mostly just a clever wrapper around the existing tools that doesn't use that much extra computing resources (jails(8) for example take almost no extra memory or CPU time) and usually the time it takes to build ports with it (especially larger batches) is very compareable to building the same ports with just make install or ports-mgmt/portmaster. Disk space is obviously a concern with poudriere but if that's a problem on the system you're using you shouldn't be building ports on the machine at all but dedicate another more suitable system for that purpose.

I really don't understand why you insist so much here. You also claim that ports should not be build at all on a system that is not dedicated for that. Some people may install tons of software but most serious sysadmins install only the necessary. So upgrading via ports is really something that takes usually less that 15 min on a properly configured and maintained server. The only reason why a dedicated machine would come handy in these situations is when you upgrade to a major version and you require a full rebuild of all ports.

I am not saying that using/having such a system is bad. But you need to realize that many people do take cost into account very seriously. So, if they have a monthly budget of $150 for hardware, then it is very difficult to raise it for adding a system that is dedicated for building packages.
 

kpa

Beastie's Twin

Reaction score: 1,831
Messages: 6,318

I really don't understand why you insist so much here. You also claim that ports should not be build at all on a system that is not dedicated for that. Some people may install tons of software but most serious sysadmins install only the necessary. So upgrading via ports is really something that takes usually less that 15 min on a properly configured and maintained server. The only reason why a dedicated machine would come handy in these situations is when you upgrade to a major version and you require a full rebuild of all ports.

That's not what I mean. All I'm saying is that the requirements for using poudriere are lower than most people seem to expect. If you can build ports on a system without poudriere it is also suitable for using poudriere unless it's seriously limited in disk space. Using it on a single machine is also not useless at all. You get the nice benefit of separating the ports building from the live system, something that anyone who's been used to the chaos of FreeBSD ports should appreciate.

I should have said this instead to put your mind to ease:

Disk space is obviously a concern with poudriere but if that's a problem on the system you're using you shouldn't be building ports on the machine at all but make use of another more suitable system for that purpose.
 

Dies_Irae

Active Member

Reaction score: 58
Messages: 137

You and some others are assuming that ports-mgmt/poudriere is this massive resource hog that can not be used on low power machines.
Mmh no, this is not what I meant. I'll try to explain it better.
Example scenario: a home desktop machine used by an average (read: not-so-skilled) user. Usually he use packages because it's simpler and faster to install and update software. One day he want/need (for any reason) to install some software from the ports tree, but this mix is causing conflicts between ports and packages. Now he has 2 choices:

- Understand how the ports and the packages systems works and resolve the conflict
- Understand how the ports and the packages systems works and configure poudriere

Do you see the common factor? ;)

I'm not saying that poudriere is bad/heavy/complicated, and there's nothing wrong in suggest its usage; I just wanted to point out the fact that the use of poudriere to resolve conflicts it's not mandatory, expecially in simple cases like this. If I am able to use poudriere, I can resolve the conflict myself. Just that, nothing more.

Obviously, with many machines and/or conflicts poudriere is a must-have, with no doubt.

Hope to have been clearer this time.
 

Zirias

Daemon

Reaction score: 495
Messages: 1,015

The issue has to do with how many servers the system will support. If someone has only one server then the cost of adding another just for the sake of building custom packages is not worth it. On the other hand, if you maintain a server farm then the benefits that you get are tremendous.

And in the scenario described by the OP with the "weak" server, there might be a much more powerful desktop machine avaiblable that could, from time to time, run a vm with poudriere for updating. This would be quite a gain.
 
Top