Solved Synth, Poudriere, Python, DEFAULT_VERSIONS?

ANOKNUSA

ANOKNUSA

Hey, folks. I posted this same topic to freebsd-questions@, but haven't gotten a reply yet, so I figured I'd try here. When building a custom repository, both Synth and Poudriere will skip over any port that requires Python 3. Poudriere will explicitly state that this is because one would have to put
Code: 
DEFAULT_VERSIONS+= python=python3.x
in /etc/make.conf, while Synth will simply claim that "there are no packages that require building." Changing the default Python version breaks the build of any Python 2 ports. However, installing a Python 3 package---in my case, archivers/borgbackup---with plain old make install clean works fine, building and installing the port and all dependencies without any Python 2 vs. Python 3 conflicts.

So what's the deal? Why can't these two programs achieve the same result as manually installing a port, and what (if anything) can I do to build archivers/borgbackup into a package? I'd like to handle all package upgrades on my home server using Synth; as it is, having an entire ports tree and custom repo on my laptop just to have access to one package is overkill.
 
I don't have problems building borgbackup using poudriere with python3.5

This is my /etc/make.conf:
Code: 
DEFAULT_VERSIONS          += python=3.5 python3=3.5 perl5=5.24 ssl=libressl clang=38

All of my packages which I use are built by poudriere to avoid conflicts or problems such as openssl and libressl, and different python or perl versions.
 
he's saying you have to pick one version of python or the other (2.x or 3.x) and he's right. While the pkgsrc tree can support both together, the freebsd ports tree never get "multiversion" capability. It's not just python, but ruby, perl, etc. I give a huge chunk of the blame to the python project because the 2x/3x thing is an abomination caused by poor design/adherence to backwards compatibility.

anoknusa mentioned /etc/make.conf a couple of times which is confusing because neither poudriere nor synth use it. They use jail-specific make.conf files, so any setting in the real /etc/make.conf would never be honored.
 
This a little trick I use on my system to make both ports-mgmt/poudriere-devel and the command line agree on settings and dependencies no matter how they are set for poudriere. My /etc/make.conf:

Code: 
DISTDIR=/var/cache/distfiles
WRKDIRPREFIX=/var/tmp
BATCH=1

.include "/usr/local/etc/poudriere.d/make.conf"

With those in place I can go to a port directory under /usr/ports (on the host, no need to start the poudriere jail) and check the dependencies and options and they will be the same ones poudriere is going to use for builds. Note that I never use saved port options, I set all them in the /usr/local/etc/poudriere.d/make.conf file. That's why I also have BATCH set in /etc/make.conf.

With ports-mgmt/synth you would .include the /usr/local/etc/synth/LiveSystem-make.conf file instead.
 
Remington said:
I don't have those settings in my host /etc/make.conf. Only ezjail's jails have those settings but poudriere's jail seems to work with host's /etc/make.conf as it hasn't run into any problems.
Click to expand...

What I am saying is that poudriere doesn't look at /etc/make.conf at all.

see: https://www.freebsd.org/cgi/man.cgi?query=poudriere&sektion=8&manpath=freebsd-release-ports

specifically:
Code: 
  Create optional make.conf
     You can also specify a global make.conf which will be used for all the
     jails.  Any of the following are allowed and will all be used in the
     order shown:

           /usr/local/etc/poudriere.d/make.conf
           /usr/local/etc/poudriere.d/<setname>-make.conf
           /usr/local/etc/poudriere.d/<tree>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<tree>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<setname>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<tree>-<setname>-make.conf
 
marino@ said:
What I am saying is that poudriere doesn't look at /etc/make.conf at all.
Click to expand...

That's why I made a soft-link in poudriere.d to look at /etc/make.conf since I use it for buildworld and ports as well.

I rarely use FreeBSD packages because many of their dependencies run into problems with my custom built packages especially openssl vs libressl.
 
Remington said:
That's why I made a soft-link in poudriere.d to look at /etc/make.conf since I use it for buildworld and ports as well.
Click to expand...

kpa@'s method is much better.
You said you have poudriere conf pull in /etc/make.conf, but it's much safer to do the reverse (have /etc/make.conf pull in poudriere's make.conf)
 
to clarify, poudriere's make.conf is strictly limited to ports while the global /etc/make.conf can and usually does contain settings for building world and kernel, often opposing (e.g. you might want to build world with debug symbols but not your ports). The fact that both world and ports use the same make.conf is kind of serious design flaw, a bit stemming from design of make itself.
 
Perhaps I haven't been entirely clear. I have several Python 2 applications installed, as well as applications that have Python 2 ports/packages as dependencies. I use these to perform the day job that lets me do things like eat and get online so I can communicate with all you fine folks. These applications are all available in the official repo, and can easily be built by Synth and Poudriere with Python 2 set as the default (which it is by default). Python 3 applications, however, cannot be built. You won't find archivers/py-borgbackup or others in the official repository.

Now, yes, one can change "DEFAULT_VERSIONS+=" to make Python 3 applications build, but go ahead and do so, and then try to build sysutils/feather or sysutils/ranger or mail/offlineimap. Those builds will now fail. As marino@ said, I need to choose one or the other, limiting what I'm able to build with these two tools. It's a choice between my preferred backup solution, and a half-dozen other things I regularly use.

Curiously, while it's impossible to use Synth or Poudriere to build both Python 2 and Python 3 ports that can co-exist in a single package repository, it's perfectly possible to just manually install all of them from the ports tree. So clearly, the ability to mix Python 2 and Python 3 applications on a single system exists, and the ports system itself will do so. I'm curious as to why Poudriere and Synth can't build co-existing packages for these interpreted languages, or if they can, how I could do so. Perhaps more importantly, I'm wondering if I'm just stuck doing what I've been doing and manually installing Borg after installing everything else. I use the quarterly repo and ports tree, so manually managing a single port alongside a mix of custom and default packages won't cause any problems. But it's an annoyance.

marino@ said:
anoknusa mentioned /etc/make.conf a couple of times which is confusing because neither poudriere nor synth use it.
Click to expand...

Synth will use the host's make.conf if it's being used to directly install and upgrade ports/packages on a single system, right? Unless you changed that recently. I could have sworn that at one point Poudriere used the host's make.conf in the absence of jail-specific configurations, but I haven't tried in a long time.
 
marino@ said:
to clarify, poudriere's make.conf is strictly limited to ports while the global /etc/make.conf can and usually does contain settings for building world and kernel, often opposing (e.g. you might want to build world with debug symbols but not your ports). The fact that both world and ports use the same make.conf is kind of serious design flaw, a bit stemming from design of make itself.
Click to expand...

I don't have flags or debug symbols for buildworld in /etc/make.conf so I'm not worried. If I need to have customized buildworld or buildkernel then I'll use /etc/src.conf.
 
Remington said:
I don't have flags or debug symbols for buildworld in /etc/make.conf so I'm not worried. If I need to have customized buildworld or buildkernel then I'll use /etc/src.conf.
Click to expand...

It was one example out of potentially hundreds.
The point is that you can later add things to make.conf without thinking about the consequences to the ports building.
Why even risk it?
It's clear that kpa's way is better. At the very best, it's the same, but it only goes down from there.

so do what you want, but for everyone else reading: If you want to use common make.confs, use kpa's method.
 
ANOKNUSA said:
Curiously, while it's impossible to use Synth or Poudriere to build both Python 2 and Python 3 ports that can co-exist in a single package repository, it's perfectly possible to just manually install all of them from the ports tree. So clearly, the ability to mix Python 2 and Python 3 applications on a single system exists, and the ports system itself will do so. I'm curious as to why Poudriere and Synth can't build co-existing packages for these interpreted languages, or if they can, how I could do so.
Click to expand...

Actually, it's not really supported, you just found a loophole.
Now, if you say "It should be supported!", then I would agree with you and frankly I don't understand the technical limitations as to why it's not.
All I can say is the DEFAULT_VERSION+=python is a global setting that both poudriere and synth honor, so the conflict is inherent in the ports tree itself.
Personally I'd happily move the default to python3 and say "tough cookies" to anybody that uses s/w that refused to move off of EOL'ed version of python 2.x but that won't fly. At some point in the hopefully near future that will be the case, but python 2.x will have to be EOL'ed officially first.

Synth will use the host's make.conf if it's being used to directly install and upgrade ports/packages on a single system, right?
Click to expand...

Nope. make.conf is only used for building. pkg(8) doesn't look at it for installation. The packages are created as a result of poudriere/synth run so pkg(8) manipulates those directly per repository metadata.
 
ANOKNUSA said:
Now, yes, one can change "DEFAULT_VERSIONS+=" to make Python 3 applications build, but go ahead and do so, and then try to build sysutils/feather or sysutils/ranger or mail/offlineimap. Those builds will now fail. As marino@ said, I need to choose one or the other, limiting what I'm able to build with these two tools. It's a choice between my preferred backup solution, and a half-dozen other things I regularly use.
Click to expand...

ranger builds fine with python3 but offlineimap and feather failed. I don't trust any packages that are still stuck with python2 and I'll consider them outdated or no longer maintained. Python3 has been out for 8 years and what does that say about the maintainers or developers who are still stuck with python2? Personally, I think any packages that still depends on python2 should be purged from the repository. I agree with marino about python team making python3 backward-incompatible a mess and they should force python2 into EOL but they extended it until 2020. Purely stupidity on their decision.
 
to be extra super clear, neither poudriere nor synth ever look at host's /etc/make.conf. There is a generated make.conf and if a jail make.conf exists, its appended to the end of that (both builders work the same way)
 
marino@ said:
It was one example out of potentially hundreds.
The point is that you can later add things to make.conf without thinking about the consequences to the ports building.
Why even risk it?
It's clear that kpa's way is better. At the very best, it's the same, but it only goes down from there.

so do what you want, but for everyone else reading: If you want to use common make.confs, use kpa's method.
Click to expand...

Okay for the sake of the argument. I'll make changes to /etc/make.conf to keep options for buildworld and ports separate. ;)
 
marino@ said:
Actually, it's not really supported, you just found a loophole.
Click to expand...

Alright, so I'll probably be going with Plan "C" then, and going straight to the source. Not ideal, but it works. I've been meaning to move away from mail/offlineimap anyway (it's slow and the documentation sucks), and I can write a script to replace sysutils/feather, but there are a handful of important applications not even written in Python that still have Python 2 dependencies, keeping me from switching to Python 3 completely.

marino@ said:
Now, if you say "It should be supported!", then I would agree with you and frankly I don't understand the technical limitations as to why it's not.
Click to expand...

I certainly won't pretend to know what it would take to get side-by-side Python versions working, but having to choose between one version or the other is a pretty tight restriction.

Thanks for the input, everyone. I'm not sure this is really "solved," but I'll mark it as such. ;)
 
You must log in or register to reply here.
Back
Top