editors/libreoffice disappeared from ports

[astyle]

The fact that pkg says that it is going to remove a package and asks yes/no is hardly relevant.

If you say "no", then you will prevent security updates for all other packages. An operating system advertising with security can't rely on that.

Funny... OpenBSD suffers from dependency hell just as much, and still relies on it to manage its own ports/packages ?

But yeah, this thread is a great use case for Poudriere - we should be able to just download the Libreoffice port (via git) and compile it against the existing stuff so that upgrading is not so damn scary. I'm trying to pull off that idea with KDE, although that project got shelved for me (real life got in the way).

 

[mer]

This is odd. audio/meterbridge builds from ports just fine and didn't have any changes lately. Few dependencies.

Up to date ports tree?

It's been a while since I've built from ports, but when you "cd /usr/port/audio/meterbridge && make clean && make" doesn't that build against what you've already got installed? If a build dependency needs to be updated it would update that and then return to the main build?

But poudrie basically starts with a clean tree and builds everything? That could imply parallel builds failing as said earlier here.

One can use poudrie to create a local repo, only install from that repo, one can control exactly what's in that git repo: selectively git update, pull in some commits and not others which would create a local repo that doesn't exactly track as a whole to something upstream.
Kind of like the OpenBSD policy of "if you have an issue with a custom kernel, retry with Generic and then talk to us".

 

[T-Daemon]

audio/meterbridge builds from ports just fine

I wanted to investigate meterbridge on portsfallout.com, but it's unknown.

Same on freshports.org. No current or deleted port named audio/meterbridge.

Nothing on https://cgit.freebsd.org/ports/tree/audio either.

 

[cracauer@]

I wanted to investigate meterbridge on portsfallout.com, but it's unknown.

Same on freshports.org. No current or deleted port named audio/meterbridge.

Nothing on https://cgit.freebsd.org/ports/tree/audio either.

Errrr.

That might be one of my own ports that I want to commit in the future. Ooops.

But there's more. Here is the list of stuff that was removed in this run:

        gitklient: 0.3_1
        libdmx: 1.1.4_2
        meterbridge: 0.9.2
        postgresql13-client: 13.11

I can imagine a big KABOOM for some use cases when the postgresql client just goes away. And in any case, there is still no explanation why my installed meterbridge package was removed instead of ignored.

 

[astyle]

Up to date ports tree?

It's been a while since I've built from ports, but when you "cd /usr/port/audio/meterbridge && make clean && make" doesn't that build against what you've already got installed? If a build dependency needs to be updated it would update that and then return to the main build?

But poudrie basically starts with a clean tree and builds everything? That could imply parallel builds failing as said earlier here.

One can use poudrie to create a local repo, only install from that repo, one can control exactly what's in that git repo: selectively git update, pull in some commits and not others which would create a local repo that doesn't exactly track as a whole to something upstream.
Kind of like the OpenBSD policy of "if you have an issue with a custom kernel, retry with Generic and then talk to us".

My idea was to:

1. start with a fresh ports tree,
2. Use it to compile EVERYTHING on the system from scratch.
3. and then, WITHIN that existing ports tree, update just the editors/libreoffice, and ignore the rest of the ports.
4. Then compile just the updated editors/libreoffice port against the existing stuff.

In this scenario, all dependencies would be already satisfied, there'd be no need to remove stuff. Just run make reinstall and be done with it. It does take a LOT of setup work to make sure it works right... so yeah, pretty much what mer said...

You build EVERYTHING only once, and then a subset of it every time you wanna upgrade. And Poudriere can be fine-tuned for parallel builds or not. Poudriere is also pretty resilient - it will just pick up and re-do the fails on its own, not much input from user required.

 

[sidetone]

I don't understand why people didn't build the port, if it meant disabling the vulnerability. When something won't build due to a security vulnerability, this can be disabled in make.conf, and it can be built.

There's too much attitude of, fix it now, instead of troubleshooting, even as non-developers are capable of troubleshooting and being helpful for the problem. This is what FreeBSD users have done before, and shared information. The ones who complain, and haven't offered solutions (aside from mainly telling devs to fix it) or tried to build it, aren't very apt at FreeBSD or at being under the hood of opensource operating systems. That's what this forum is about. For other problems, people have pointed to bug reports, pointed to other information, and people have asked around how it can be build on our local computers, about weighing the risks of building a vulnerable port.

When it's an upstream issue, due to a vulnerability, that means it's for all operating systems. The difference is, on FreeBSD, you decide whether to build it anyway. On mainstream Linuxes, they make the decision for you, by either ignoring the vulnerability, or finding a way around that. People who know how to use FreeBSD, already know to disable the vulnerability lock to build it, if they urgently need that program, or wait a few days.

 

[drhowarddrfine]

That's why we're here.

No. That's not why we're here. We're here, and the other more current thread about the same topic, because libreoffice disappeared from the package repository and you couldn't upgrade or install it no matter how hard you tried. iow, pkg would reply that there is no such package as libreoffice. So what that has to do with any issues involving the uninstallation or removal of libreoffice or any other software, I don't know but it's not the topic of this or the other thread.

From the OP's first post here:

coming from the freebsd support channel on liberachat, I've been told to report here that editors/libreoffice can't be installed with pkg, the binary distribution seems to be absent.

 

[gpw928]

No. That's not why we're here. We're here, and the other more current thread about the same topic, because libreoffice disappeared from the package repository and you couldn't upgrade or install it no matter how hard you tried. iow, pkg would reply that there is no such package as libreoffice. So what that has to do with any issues involving the uninstallation or removal of libreoffice or any other software, I don't know but it's not the topic of this or the other thread.

My original post on the parallel thread begins with the words "A package upgrade just deleted libreoffice-7.5.5.2_3".

I really don't understand why you keep trying to insist that this didn't happen, or that it's a problem that does not deserve consideration.

 

[gpw928]

I find myself at odds with people whose views on the behaviour of Unix are pretty well atuned to my own. I like the "no mercy" approach, and despise software that repeatedly asks "are you really sure?".

However, I'm reminded of a talk given last century by Stuart Feldman, author of make(1). I can't remember whether it was Ritchie or Thompson who had a bunch of valuable files accidentally deleted by make. Feldman's argument was that they had tacitly agreed. Nevertheless the subsequent ire (read s#!t fight) led to implementation of the ".PRECIOUS" directive.

My point is that there is a long history of making Unix less hostile to "ordinary" users!

I understand the technical difficulties here. But I need a recently patched web browser for my desktop. And I am fully confident that many others do as well. Currently, the best option to get that is to be on the latest packages.

So I think it would make sense to modify the pkg upgrade command to default to fail if it would have to remove a package that was explicitly installed by the user.

That way ordinary users could upgrade packages from cron(8) with confidence.

Failures could be dealt with as a relatively rare exception (and the handbook could be updated to recommend waiting a day or two).

 

[PMc]

I understand the technical difficulties here. But I need a recently patched web browser for my desktop. And I am fully confident that many others do as well. Currently, the best option to get that is to be on the latest packages.

Yes, there is some fancy of always needing the very newest browser. I don't know why, I am happy with ESR, and updating quarterly only because otherwise the amount of changes might get unintellegible.

So I think it would make sense to modify the pkg upgrade command to default to fail if it would have to remove a package that was explicitly installed by the user.

That way ordinary users could upgrade packages from cron(8) with confidence.

That's a different usecase. pkg upgrade is designed to calculate the updates, ask the user if that would be fine, download them and calculate again, and if there are changes, maybe ask again, and only then run the upgrade.
This interaction can be switched off, that is mainly for scripted use where the script already knows which specific package it wants to upgrade (and why).
But running blanket change upgrades from cron without an idea of what's going to happen, I think that is a horrortrip. It results in an undefined system where things are continuousely morphing, making it practically impossible to obtain solid information for any debugging.

 

[drhowarddrfine]

gpw928 This is the problem with resurrecting an old thread which this one is and cross posting to the other newer one. One gets confused as to what one is responding to and thinking they're answering one topic, not the other. Your thread appeared on the same day as the disappearance of libreoffice in the package repository and the same day it was posted as a PR.

 

[scottro]

Heh, I didn't even know it till I saw this thread--I use it for very few things, but when I saw this thread, I was able to reinstall with pkg. I should probably use quarterly on my main workstation.

I see no mention of this though in /usr/ports/UPDATING. Possibly because it's already fixed, but it does seem to me that UPDATING isn't as reliable a source of information as it used to be. Might just be the packages I use that gives me that impression and I can't even give an example off the top of my head, but it does seem to me that at times, the documentation's fallen behind, or rather than the handbook, it's on the wiki, and so on.

 

[PMc]

but it does seem to me that UPDATING isn't as reliable a source of information as it used to be.

Me too. There is things that I would like to read there, but don't. But this one most likely isn't there because nobody considered it noteworthy or is aware of it at all.

Possibly because it's already fixed,

I'm quite sure the actual problem is not yet fixed.
To subsume: libreoffice has a bug that results in occasional build failures. Most likely with high core count builds (>16). Usually, when that happens, we just re-run the build. Apparently, when that happens on the pkgrepo build site, then the pkg is missing for a while.

So, maybe this time we will tackle it and figure out what is actually going wrong, and fix it. Or maybe not.

 

[rrbrussell]

I find myself at odds with people whose views on the behaviour of Unix are pretty well atuned to my own. I like the "no mercy" approach, and despise software that repeatedly asks "are you really sure?".

However, I'm reminded of a talk given last century by Stuart Feldman, author of make(1). I can't remember whether it was Ritchie or Thompson who had a bunch of valuable files accidentally deleted by make. Feldman's argument was that they had tacitly agreed. Nevertheless the subsequent ire (read s#!t fight) led to implementation of the ".PRECIOUS" directive.

My point is that there is a long history of making Unix less hostile to "ordinary" users!

I understand the technical difficulties here. But I need a recently patched web browser for my desktop. And I am fully confident that many others do as well. Currently, the best option to get that is to be on the latest packages.

So I think it would make sense to modify the pkg upgrade command to default to fail if it would have to remove a package that was explicitly installed by the user.

That way ordinary users could upgrade packages from cron(8) with confidence.

Failures could be dealt with as a relatively rare exception (and the handbook could be updated to recommend waiting a day or two).

The underlying technical problem is that pkg does not encode soname dependencies. It must assume that any change to a dependency breaks the dependent package.

I know people don’t like Linux around here but the rpm format has been able to track soname dependencies for a few years and I believe the deb package format can do it as well. I have no idea if distributions like Slackware or Alpine can track that information.

Let us assume we have two packages: A and B. A depends on B. Also let us assume the simple case of A not caring about which options B is built with. To resolve the issue we need to know the following information:

• The soname of the library in B that A depends on. Specifically we should try to link with as non specific a soname as possible.
• The version of B that A was compiled and linked against. This defines our lower version bound.
• The version of B that had or likely has ABI changes sufficient to break A. This defines our upper version bound.

Between the upper and lower bounds pkg should just install any upgrades to B. A should not need to be recompiled.

No, keeping this information isn’t a perfect solution. I do expect it to require breaking changes to pkg and the pkg data/file format.

Anyway that is my estimation of the minimum technical information needed to fix the problem.

 

[shkhln]

The underlying technical problem is that pkg does not encode soname dependencies. It must assume that any change to a dependency breaks the dependent package.

Pkg does encode that information. Thankfully, it doesn't use it for dependency resolution, because that's an imbecile feature anyway.

 

[rrbrussell]

Pkg does encode that information. Thankfully, it doesn't use it for dependency resolution, because that's an imbecile feature anyway.

Where? It encodes the dependency ports and versions but none of the actual shared library dependencies. At least it doesn’t seem to. I was chasing a missing libssl.so.111 earlier this week and given the amount of disk activity and cpu load when I asked pkg to check for missing shared libraries I think it parsed ldd’s output for every file in /usr/local/{bin,sbin,lib}.

Why is that feature imbecilic? Upgrading a shared library in a way that is compatible with already built and linked software is the point of soname versioning standards.

 

[shkhln]

Where?

% pkg info chromium | grep -i shared -A 10
Shared Libs required:
    libxslt.so.1
    libxshmfence.so.1
    libxml2.so.2
    libxkbcommon.so.0
    libxcb.so.1
    libwebpmux.so.3
    libwebpdemux.so.2
    libwebp.so.7
    libsndio.so.7.2
    libsnappy.so.1
--
Shared Libs provided:
    libvulkan.so.1
    libvk_swiftshader.so
    libVkICD_mock_icd.so
    libGLESv2.so
    libEGL.so
Annotations    :
    FreeBSD_version: 1302001
    cpe            : cpe:2.3:a:google:chrome:117.0.5938.88:::::freebsd13:x64
    repo_type      : binary
    repository     : FreeBSD

I doubt you'll want to use libvulkan.so.1 from chromium, though.

Why is that feature imbecilic? Upgrading a shared library in a way that is compatible with already built and linked software is the point of soname versioning standards.

There are quite a few dependencies that are not ELF shared libs: Python/Ruby/Java libs and interpreter versions, interfaces to the kernel modules, D-Bus interfaces. All this shit has to be tracked for the partial upgrades to work and shared libs are a huge distraction from this (mostly from pkg ignoring package versions). You might say it's worth it because it's simply an additional information which could be derived fully automatically not costing any maintenance time, but as you can see from the libvulkan.so.1 example above it really can't be.

Don't copy features from RPM, you'll be disappointed.

 

[zirias@]

I doubt you'll want to use libvulkan.so.1 from chromium, though.

This could probably be solved easily by excluding any "private" libs, those are not installed into the standard paths for obvious reasons.

There are quite a few dependencies that are not ELF shared libs: Python/Ruby/Java libs and interpreter versions, interfaces to the kernel modules, D-Bus interfaces. All this shit has to be tracked for the partial upgrades to work and shared libs are a huge distraction from this (mostly from pkg ignoring package versions). You might say it's worth it because it's simply an additional information which could be derived fully automatically not costing any maintenance time, [...]

Exactly. One might actually think about such a feature. It would certainly take some development effort, but "doable". And in theory, it would make perfect sense.

I still agree with you that it shouldn't be done, but for yet another reason: The SONAME of a library is meant to name an ABI. Extensions are fine, but whenever existing pieces of a library ABI change, the SONAME must change. A feature handling lib dependencies based on the SONAME would rely on all upstream projects getting it right. In practice, it's broken too often. Some upstreams get it completely wrong, having all sorts of breaking changes and still keeping the SONAME (and that's even common practice for "pre-release" libs which you might need as well). Some attempt to get it right, but actually only keep the API stable. A classic example would be an API exposing e.g. a struct type. Adding a member to the end of the struct doesn't break the API (the same consuming code still compiles fine), but it does break the ABI, because the size information of any type is baked into the binaries. There's a reason that it's common practice in the ports tree to "bump" all consumers of some shared lib whenever that lib is updated (regardless of its SONAME), so they are rebuilt as well. This avoids issues caused by upstreams getting it wrong.

 

[rrbrussell]

% pkg info chromium | grep -i shared -A 10
Shared Libs required:
    libxslt.so.1
    libxshmfence.so.1
    libxml2.so.2
    libxkbcommon.so.0
    libxcb.so.1
    libwebpmux.so.3
    libwebpdemux.so.2
    libwebp.so.7
    libsndio.so.7.2
    libsnappy.so.1
--
Shared Libs provided:
    libvulkan.so.1
    libvk_swiftshader.so
    libVkICD_mock_icd.so
    libGLESv2.so
    libEGL.so
Annotations    :
    FreeBSD_version: 1302001
    cpe            : cpe:2.3:a:google:chrome:117.0.5938.88:::::freebsd13:x64
    repo_type      : binary
    repository     : FreeBSD

I doubt you'll want to use libvulkan.so.1 from chromium, though.

--snipped some other stuff--

Pkg needs some extra documentation over its internal structure then. I understood that information came from running ldd iteratively on the files listed in the manifest, it certainly didn't appear to be anywhere in the .pkg file itself and the provided shared libs list makes me suspicous about how Chromium was built/packaged.

This could probably be solved easily by excluding any "private" libs, those are not installed into the standard paths for obvious reasons.

Exactly. One might actually think about such a feature. It would certainly take some development effort, but "doable". And in theory, it would make perfect sense.

I still agree with you that it shouldn't be done, but for yet another reason: The SONAME of a library is meant to name an ABI. Extensions are fine, but whenever existing pieces of a library ABI change, the SONAME must change. A feature handling lib dependencies based on the SONAME would rely on all upstream projects getting it right. In practice, it's broken too often. Some upstreams get it completely wrong, having all sorts of breaking changes and still keeping the SONAME (and that's even common practice for "pre-release" libs which you might need as well). Some attempt to get it right, but actually only keep the API stable. A classic example would be an API exposing e.g. a struct type. Adding a member to the end of the struct doesn't break the API (the same consuming code still compiles fine), but it does break the ABI, because the size information of any type is baked into the binaries. There's a reason that it's common practice in the ports tree to "bump" all consumers of some shared lib whenever that lib is updated (regardless of its SONAME), so they are rebuilt as well. This avoids issues caused by upstreams getting it wrong.

I can certainly understand your point of view. I am on a first name basis with the C footgun. I can write useable python, it works for now php and shell, and decent Rust code. My C/C++ is a disaster.

More generally here, I see two general paths forward. The first is call this expected behaviour, but a footgun, and improve the handbook and probably the pkg manual as well. The second path is improve pkg's dependency tracking, the cheaper options are failing if a manually installed package would be removed or reorder the output from pkg to put manually installed but to be removed packages closer to the confirmation request. The expensive version of the second path is a possible full rehab of how pkg tracks and possibly how poudriere handles building ports.

I can have a rough draft done for the documentation changes in a day or two. I am assuming the documentation patches need to go to the freebsd-docs mailing list?

The other two options woud be better handled by someone other than me, well if you don't want the result in Rust or Python anyway.

 

[shkhln]

Pkg needs some extra documentation over its internal structure then. I understood that information came from running ldd iteratively on the files listed in the manifest, it certainly didn't appear to be anywhere in the .pkg file itself and the provided shared libs list makes me suspicous about how Chromium was built/packaged.

It's shlibs_required and shlibs_provided in +MANIFEST in the package root.

 

[PMc]

It just happened again in my build engine!

Here it is running, and it is running forever:

Build in print/tex-xetex ... ok
Build in security/xmlsec1 ... ok
Build in editors/libreoffice ...

The instance (last one) consumes a low amount of compute

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
83044 root         28 134  i10    16G  8838M kqread   4 198.2H 953.06% bhyve
84561 root         20 134  i10  6301M   234M kqread  10  55:36 198.15% bhyve
  639 root         20 134  i10  6301M   463M kqread  10  10:48 198.12% bhyve
 3610 root         20  21    0   624M   470M kqread   8 195:08  35.59% bhyve
81236 root         24 134  i10    11G  5355M kqread   6 117.6H  16.94% bhyve

Lets look inside:

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
85059 root         10  20    0  1674M    22M uwait    0  19:48   6.44% javac
20045 root          1  20    0    14M  3024K CPU3     3   0:01   0.68% top
   11 root         20 -52    -     0B   320K WAIT    -1  12:09   0.56% intr
    0 root        108 -16    -     0B  1728K swapin   5  59:30   0.16% kernel

So this is our endless running process: javac

UID   PID  PPID  C PRI NI     VSZ    RSS MWCHAN   STAT TT        TIME COMMAND
0 85059 71727 0 20 0 1713960 22856 uwait I - 19:52.43 /usr/local/openjdk11/bin/javac -J-Xmx128M -encoding utf8 --release 8 -Xlint:-options -g -d /var/local/ports/usr/ports/editors/libreoffice/work/libreoffice-7.6.2.1/workdir/CustomTarget/jvmfwk/jreproperties/ /var/local/ports/usr/ports/editors/libreoffice/work/libreoffice-7.6.2.1/jvmfwk/plugins/sunmajor/pluginlib/JREProperties.java

Looking inside: it runs ten threads

# ps axH | grep 85059
85059  -  I       0:00.02 /usr/local/openjdk11/bin/javac -J-Xmx128M -encoding u
85059  -  I       0:00.05 /usr/local/openjdk11/bin/javac -J-Xmx128M -encoding u
85059  -  I       0:00.00 /usr/local/openjdk11/bin/javac -J-Xmx128M -encoding u
85059  -  S       6:46.50 /usr/local/openjdk11/bin/javac -J-Xmx128M -encoding u
85059  -  I       0:00.00 /usr/local/openjdk11/bin/javac -J-Xmx128M -encoding u
85059  -  S       6:43.71 /usr/local/openjdk11/bin/javac -J-Xmx128M -encoding u
85059  -  S       6:32.78 /usr/local/openjdk11/bin/javac -J-Xmx128M -encoding u
85059  -  S       0:00.68 /usr/local/openjdk11/bin/javac -J-Xmx128M -encoding u
85059  -  I       0:00.00 /usr/local/openjdk11/bin/javac -J-Xmx128M -encoding u
85059  -  I       0:00.03 /usr/local/openjdk11/bin/javac -J-Xmx128M -encoding u

And truss shows that it apparently has a problem with interlocking of these threads:

85059: 0.000094693 0.000155326 _umtx_op(0x18c56bee2038,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x18,0x18c59ef43db8) ERR#60 'Operation timed out'
85059: 0.011988157 0.012034650 _umtx_op(0x18c56bee2068,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x18,0x18c5a8e0ddb8) ERR#60 'Operation timed out'
85059: 0.014322564 0.014362192 _umtx_op(0x18c56bee2080,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x18,0x18c5a9b08db8) ERR#60 'Operation timed out'
85059: 0.014734241 0.000229620 _umtx_op(0x18c56bee2038,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x18,0x18c59ef43db8) ERR#60 'Operation timed out'
85059: 0.015662659 0.001040724 _umtx_op(0x18c56bee2068,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x18,0x18c5a8e0ddb8) ERR#60 'Operation timed out'

There is nothing else going on than these calls to _umtx_op.


I found a somehow similar problem description for a different product here:

Workers sometimes get stuck due to a deadlock in Java's process handling · Issue #1250 · bazelbuild/bazel

Compilation hangs, please see the last console messages before hanging. Repeatable. Version: 0.2.2b OS: FreeBSD Java: openjdk8-8.66.17_1 ---example 1--- INFO: From Building src/main/java/com/google...

github.com

(bugreport 274162 updated)

 

[PMc]

Madness strikes.

 

[astyle]

Madness strikes.

You mean PR 274162 ([PR]274162[/PR]) ?

 

[PMc]

I mean, the libreoffice port does not just "magically" disappear from ports.
It disappears because occasionally, during the build, an invocation of javac goes into an endless loop. And it does this on my build system also.

So, why do I bother to figure things out so far, if then the bug report is just closed? That thing will probably disappear again at some occasion, unless the actual problem gets fixed.

 

[astyle]

As long as we keep atttracting new users, we'll be faced with the task of explaining the difference between packages and ports... ? I was at one point accused of splitting hairs by someone who didn't know a thing about software compiling.... ?