Rough seas in the ports tree?

S

semi-ambivalent

I have been doing a complete re-install of a 12.1 box. I'm having a lot of problems getting ports to compile; libreoffice with FreeBSD patches that fail to apply, QT5-core failing to compile, smaller things that are apparently needed by several applications. I've even installed some pkgs just to get some ports to finish. A mess I attribute to being after the release of 11.4 and lots of new changes being committed. (Along with things like thunderbird sharing code with firefox, but I won't mention that.)

What I want to do is roll all of this back to a point where the ports pool environment appeared nice and calm. Can I do a while-in-use point-in-time with, say, portmaster, that doesn't include "rm -rf /usr/local/*"? Or using any other ports manager? origin ports count is 134, and I can prune a bit first. I'd like to do this live.

thx
 
semi-ambivalent said:
What I want to do is roll all of this back to a point where the ports pool environment appeared nice and calm
Click to expand...
So, that's never. The ports tree is continuously updated and changed, there's never a dull moment. You can checkout a quarterly branch though, that's more or less stable for 3 month periods (only receives security/stability updates during that period).

svnlite co https://svn.freebsd.org/ports/branches/2020Q2 /usr/ports

You may want to hold off for a few days though, first days of July a new quarterly branch will get branched off (2020Q3).
 
SirDice said:
So, that's never. The ports tree is continuously updated and changed, there's never a dull moment.

svnlite co https://svn.freebsd.org/ports/branches/2020Q2 /usr/ports

You may want to hold off for a few days though, first days of July a new quarterly branch will get branched off (2020Q3).
Click to expand...

True, but there are times of less-dullness. I suppose it doesn't matter if a particular port is or is not a train wreak if I never need it. I'll keep the quarterlies in mind since I no longer need the latest.

thx
 
SirDice said:
So, that's never. The ports tree is continuously updated and changed, there's never a dull moment. You can checkout a quarterly branch though, that's more or less stable for 3 month periods (only receives security/stability updates during that period).

svnlite co https://svn.freebsd.org/ports/branches/2020Q2 /usr/ports

You may want to hold off for a few days though, first days of July a new quarterly branch will get branched off (2020Q3).
Click to expand...
Which one is the ports tree installed during installation -- the quarterly one or the head branch?
 
mjollnir said:
Which one is the ports tree installed during installation -- the quarterly one or the head branch?
Click to expand...
If you refer to the ports.txz package offered during installation, then (outdated now) head branch, assuming the release(7) documentations is up to date and the Builders Release Engineering Team is following the described procedure:

http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.1-RELEASE/BUILDDATE :

20191101

release(7):
Code: 
For some users, it may be desirable to provide an absolutely clean build
     environment, with no local modifications to the source tree or to
     make.conf(5), and with clean checkouts of specific versions of the doc,
     src, and ports trees.  For this purpose, a script
     (src/release/release.sh) is provided to automate these checkouts and then
     execute “make release” in a clean chroot(8).
/src/release/release.sh:
Code: 
 # The default svn checkout server, and svn branches for src/, doc/,
        # and ports/.
        SVNROOT="svn://svn.FreeBSD.org/"
        SRCBRANCH="base/head@rHEAD"
        DOCBRANCH="doc/head@rHEAD"
        PORTBRANCH="ports/head@rHEAD"
 
mjollnir said:
Which one is the ports tree installed during installation -- the quarterly one or the head branch?
Click to expand...
The one that's installed from a ports.txz on the install media? That's a snapshot of head at the time the -RELEASE was made. A tool like portsnap(8) also grabs head, aka latest. For pkg(8) however the default is set to quarterly on -RELEASE versions and latest on -STABLE.
 
mjollnir said:
I'm for shure not the only one who suggests not to use ports-mgmt/portmaster (which builds ports in the main system) but poudriere or synth to build your ports. Installing devel/ccache will save you a great amount of time.
Click to expand...
I have used portmaster for a very long time now and it has always worked well -- except in these periods where updates are touching complex software. No doubt there are newer solutions but even with those there will be things like this which will cause headaches:

Code: 
bixby [105] [11:11am] [/usr/ports/devel/qt5-core]# make
===>  License LGPL21 accepted by the user
===>   qt5-core-5.14.2_3 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by qt5-core-5.14.2_3 for building
===>  Extracting for qt5-core-5.14.2_3
=> SHA256 Checksum OK for KDE/Qt/5.14.2/qtbase-everywhere-src-5.14.2.tar.xz.
===>  Patching for qt5-core-5.14.2_3
===>  Applying extra patch /usr/ports/devel/qt5/files/extrapatch-configure
===>  Applying extra patch /usr/ports/devel/qt5/files/extrapatch-mkspecs_features_create__cmake.prf
===>  Applying extra patch /usr/ports/devel/qt5/files/extrapatch-mkspecs_features_qt__module.prf
===>  Applying extra patch /usr/ports/devel/qt5/files/extrapatch-mkspecs_common_bsd_bsd.conf
===>  Applying extra patch /usr/ports/devel/qt5/files/extrapatch-mkspecs_freebsd-clang_qmake.conf
===>  Applying FreeBSD patches for qt5-core-5.14.2_3 from /usr/ports/devel/qt5-core/files
Ignoring previously applied (or reversed) patch.
1 out of 1 hunks ignored--saving rejects to src/corelib/global/qfloat16.cpp.rej
===>  FAILED Applying FreeBSD patch-src_corelib_global_qfloat16.cpp
===> Cleanly applied FreeBSD patch(es)  patch-git_276fa83 patch-src_corelib_corelib.pro
===> FAILED to apply cleanly FreeBSD patch(es)  patch-src_corelib_global_qfloat16.cpp
*** Error code 1

Stop.
make: stopped in /usr/ports/devel/qt5-core

I will use ccache in the future.

thx all
 
I use poudriere and switch to a stable branch, when I have a problem with unstable port tree, This is an example for branches/2020Q1. I show these steps in 2020Q1 branch, because I've tested it, and it was stable. Change it to more recent branch. And I use some name conventions, refer to the end of the post (Note section).

Install ports-mgmt/poudriere and devel/ccache
pkg install ports-mgmt/poudriere devel/ccache

Make few directory, copy settings and change persmissions:
mkdir -p /usr/ports/distfiles
mkdir -p /var/cache/ccache
cp /usr/local/etc/poudriere.conf.sample /usr/local/etc/poudriere.conf
chmod 640 /usr/local/etc/poudriere.conf

Minimal edit to /usr/local/etc/poudriere.conf
Code: 
FREEBSD_HOST=https://download.FreeBSD.org
CCACHE_DIR=/var/cache/ccache
CHECK_CHANGED_DEPS=yes
CHECK_CHANGED_OPTIONS=verbose

List all your packages here: jf121x64-p2020Q1-set0
e.g. I want to build security/libressl and security/ca_root_nss with poudriere. jf121x64-p2020Q1-set0 is going to look like this:

/usr/local/etc/poudriere.d/jf121x64-p2020Q1-set0
Code: 
security/libressl
security/ca_root_nss

Create a jail
poudriere jail -c -j jf121x64 -v 12.1-RELEASE -a amd64

Create a port tree
poudriere ports -c -p 2020Q1 -B branches/2020Q1 -m svn

Edit FreeBSD.conf to refelect poudriere packages:
/usr/local/etc/pkg/repos/FreeBSD.conf
Code: 
Poudriere:{url:"file:///usr/local/poudriere/data/packages/jf121x64-p2020Q1-set0"}
FreeBSD:{enabled:no}

View, keep or change ports options:
poudriere options -j jf121x64 -p 2020Q1 -z set0 -f /usr/local/etc/poudriere.d/jf121x64-p2020Q1-set0

Build the ports:
poudriere bulk -j jf121x64 -p 2020Q1 -z set0 -f /usr/local/etc/poudriere.d/jf121x64-p2020Q1-set0

Update and install packages from poudriere build:
pkg update -f
pkg install security/libressl security/ca_root_nss

Notes:
  • jf121x64: it's just a name for jail (Jail FreeBSD 12.1 amd64)
  • 12.1-RELEASE: FreeBSD version
  • amd64: arch
  • 2020Q1: name for port tree
  • branches/2020Q1: is the name of branch
 
drhowarddrfine said:
I still use portmaster. Haven't had any issues in a long time.
Click to expand...
This might be because you most likely only build ports with a fairly straight/easy set of dependencies. Portmaster is broken by design concerning handling of dependencies. In other words: it does not deserve it's name, because this is an essential preliminary to build ports correctly. If you ever try to build a full KDE workspace/distribution with it, you'll know this is true. Portmaster tries to calculate by-feet what should be done by integrating make(1), and give it the complete set of dependencies all at once (i.e. all Makefiles of all ports involved). See the well known Recursive Make Considered Harmful -- a must-read for every developer and people involved in maintaining/building software. Obviously, there might be dubious rules in some under-maintained Makefiles, which can be another source of failure; that's human and can always happen. IMHO such thing like portmaster should not be allowed to be available in the FreeBSD ports tree. When following the recipe given in the mentioned paper, you'll not have such problems (except caused by incomplete rules), and as a nice side-effect, parallel make works correctly because make(1) stratifies the dependency graph. Spread the word: do not use so-called portmaster until it's fixed.

Besides that, the issues of the OP seem to come from an unclean build environment, i.e. there is some /wrkdir left over from a previous build ("Ignoring previously applied (or reversed) patch"). Poudriere or synth together with ccache will avoid such, because they always build in a fresh jail.
 
drhowarddrfine said:
I still use portmaster. Haven't had any issues in a long time.
Click to expand...
Same here - works for me looking after Apache, MySQL and PHP and the things they pull in. Occasional glitch but mostly fine.

There are definitely better and cleaner and more liked tools, and if you are starting with FreeBSD - definitely look at those first.
 
I've always hesitated to use synth because, when it first came out, many on this forum stated it was more for people who were not technically inclined so I never bothered with it. Poudriere I never bothered with because portmaster worked fine for me and my impression was always that it was more useful when you had a port that needed distribution among several servers, it needed ZFS, and there was more configuration involved. I also question how much space this uses as some of the servers I used were limited. So I never bothered since I don't run my company anymore.

I won't use KDE or Gnome for a desktop and I only install developer-ish software like servers and Gimp and browsers.
 
mjollnir said:
Portmaster is broken by design concerning handling of dependencies.
Click to expand...
Although I'd never use portmaster as poudriere is way superior, this statement isn't true. The ports tree and make framework in /usr/ports/Mk already handles dependencies correctly. Portmaster doesn't add much here, while poudriere takes a completely different approach by evaluating the dependencies itself, building each port in a clean jail and installing exactly the required dependencies from packages built earlier. The key difference is the "clean environment" build.

mjollnir said:
Portmaster tries to calculate by-feet what should be done by integrating make(1), and give it the complete set of dependencies all at once (i.e. all Makefiles of all ports involved). See the well known Recursive Make Considered Harmful -- a must-read for every developer and people involved in maintaining/building software. Obviously, there might be dubious rules in some under-maintained Makefiles, which can be another source of failure; that's human and can always happen. IMHO such thing like portmaster should not be allowed to be available in the FreeBSD ports tree. When following the recipe given in the mentioned paper, you'll not have such problems (except caused by incomplete rules), and as a nice side-effect, parallel make works correctly because make(1) stratifies the dependency graph.
Click to expand...
And this statement is, in fact, FUD. A port build is "recursive", yes, because that's the only choice you have when you want to build third-party software using all kinds of build-systems from your running make job. But it includes all the steps from downloading the distribution file to packaging the results of the build, so this is nothing like the traditional "recursive make"* and doesn't affect any dependencies. The ports tree itself is designed recursive as well, but that doesn't matter, as the Makefile of any port you build will include all the framework and become the "master" for this build.

The problems you typically run into with a tool like portmaster are all related to unclean build environments -- you could have dependencies installed in non-matching versions, you could run into "automagic" dependencies, etc.

* (As a side note, it's very well possible to get recursive make to operate correctly as well, although I peref to avoid it if possible.)
 
Zirias said:
Although I'd never use portmaster as poudriere is way superior, this statement isn't true. The ports tree and make framework in /usr/ports/Mk already handles dependencies correctly.
Click to expand...
So then make -j 4 fetch-recursive in a big meta-ports directory to fetch all tar balls needed to build this port with 4 parallel downloads, will succeed and not try to download anything twice? (A message like "xyz-1.2.3.tar.gz already in distfiles; skipping" indicates a broken dependency graph: it is not stratified correctly)
Portmaster doesn't add much here
Click to expand...
Try portmaster -F x11/kde5 and be very patient...
And this statement is, in fact, FUD. A port build is "recursive", yes, because that's the only choice you have when you want to build third-party software using all kinds of build-systems from your running make job.
Click to expand...
I don't know what FUD means. Unfortunately, in practice: yes. The sound solution would be to require the maintainers to rewrite all external make dialects into BSD Makefiles, which is no practical alternative.
The problems you typically run into with a tool like portmaster are all related to unclean build environments -- you could have dependencies installed in non-matching versions, you could run into "automagic" dependencies, etc.
ossible.)
Click to expand...
See the above example.
 
mjollnir said:
So then make -j 4 fetch-recursive in a big meta-ports directory to fetch all tar balls to build this port with 4 parallel downloads, will succeed and not try to download anything twice?
Click to expand...
It won't if you remove that -j 4. You're trying to run the same thing in 4 parallel jobs, that's bound to cause race conditions.

Job 1: checks if file is there. File isn't there and starts the download
Job 2: checks if file is there. File isn't there (job 1 hasn't finished downloading yet), and starts the download.
 
SirDice said:
It won't if you remove that -j 4. You're trying to run the same thing in 4 parallel jobs, that's bound to cause race conditions.

Job 1: checks if file is there. File isn't there and starts the download
Job 2: checks if file is there. File isn't there (job 1 hasn't finished downloading yet), and starts the download.
Click to expand...
If the dependency graph was stratified correctly, make would not start a 2nd job to fetch the very same target.
This is the same case like a parallel make will not start a 2nd compile job on the very same target, iff the rules given to it via the Makefiles are correct - which is not always the case due to human mistakes. That's normal. Maybe you're refering to these errors as race conditions. My understanding of this term does not match this kind of failure.
 
mjollnir said:
So then make -j 4 fetch-recursive in a big meta-ports directory to fetch all tar balls needed to build this port with 4 parallel downloads, will succeed and not try to download anything twice? (A message like "xyz-1.2.3.tar.gz already in distfiles; skipping" indicates a broken dependency graph: it is not stratified correctly)
Click to expand...
In the ports tree, make is used as a tool to control jobs, and the dependencies for that work correctly (by using some "stamp files", but they are only created after finishing a job). Make avoids races when the targets are files, they are either there (so make can check the timestamp) or they aren't. This isn't possible in a usecase like the ports tree, distfiles have to be downloaded, checksummed, there are some scripts involved. You can't make a distfile itself a make target, this would kill all the flexibility of the ports framework.

So, yes, the ports tree isn't designed for parallel make, and never was. But it can of course build the actual ports using parallel make if their build system supports it. All of this has nothing to do with portmaster.
 
Zirias said:
[...] So, yes, the ports tree isn't designed for parallel make, and never was. But it can of course build the actual ports using parallel make if their build system supports it. All of this has nothing to do with portmaster.
Click to expand...
I am aware that the ports collection and it's handling has evolved over time, and it might add too much complexity to the Makefiles if the inter-port dependencies were reflected in strictly correct rules. I do not agree that this does not have to do with portmaster: contrast fetching the distfiles for a huge project like kde5 done by
  • portmaster -F x11/kde5 will ponder the dependencies for hours and download nothing after 1 hour. A newbie is tempted to do this, once s/he took the (IMHO wrong) decision to use portmaster to handle the ports
  • vs. letting make handle the dependencies: make -f /usr/ports/x11/kde/Makefile fetch-recursive will start fetching the distfiles after a few seconds to construct the correct dependency graph (and it will not attempt to fetch the same file twice if the deps rules are correct, even in parallel mode)
 
Origins go back to the '80s, and it was originally coined to describe IBM's behavior
Defined by Gene Amdahl after he left IBM to found his own company: “FUD is the fear, uncertainty, and doubt that IBM sales people instill in the minds of potential customers who might be considering [Amdahl] products.” The idea, of course, was to persuade them to go with safe IBM gear rather than with competitors' equipment. This implicit coercion was traditionally accomplished by promising that Good Things would happen to people who stuck with IBM, but Dark Shadows loomed over the future of competitors' equipment or software. See IBM. After 1990 the term FUD was associated increasingly frequently with Microsoft, and has become generalized to refer to any kind of disinformation used as a competitive weapon.
Click to expand...

FUD

www.catb.org
 
Ah ok, thanks for the history lesson :) I normally blame Microsoft for everything so they get a reprieve on this one...IBM was probably as evil as Microsoft is/was though (thinking of Compaq rise to power).
 
You must log in or register to reply here.
Back
Top