Disappearance of Desktop Applications (firefox/thunderbird ...) from Repositories

The problem was a changed patch file that was pulled from an external url by multimedia/x265. Due to the change the checksum did not match any more and the patch still isn't pulled into the quartely branch.

So to build all ports that require x265 from quarterly, one needs to cherry-pick commit 88c12b53a49ce5d7686cfcda9e2409ed3c1bd0f2 (or edit the distfile by hand / copy it from latest)
 
NapoleonWils0n said:
...

i think im going to have to install Ubuntu on my back up computer a Macbook Pro Retina 2015

All 3 of my computers run Freebsd

Dell Xps 15 2019
Macbook Pro Retina 2015
Macbook air 2011

Not sure its wise to keep all my eggs in one basket
Click to expand...

I'm thinking about following the same route but I will wait for the final release of PopOS 24.04 to make a decision. I know that btrfs works fine and that the system can be installed on an encrypted LVM device but for me the real hard part will be running a system without ZFS.
 
ubuntu 24.04 does have experimental zfs on root

but its zfs on an encrypted lvm and not zfs encryption

i used fedora for a while which ships with btrfs
but i didnt like fedora for the following reasons

1 - you have to add repositories for non-free software like video codecs

2 - the package manager is awful

and when you add a repositories it really slows down searching for packages
you go to search for a package and it then has to sync all of the external repos before it returns a simple search result

so you end up waiting 30 seconds to a minute just searching for a package

3 - constant kernel updates every couple of days

nothing more annoying than switching on the computer and having to install a kernel update and reboot every day
before you can get on with some work

4 - btrfs

i think you have to install a btrfs grub menu to replicate freebsd boot environments
 
NapoleonWils0n said:
ubuntu 24.04 does have experimental zfs on root

but its zfs on an encrypted lvm and not zfs encryption

i used fedora for a while which ships with btrfs
but i didnt like fedora for the following reasons

1 - you have to add repositories for non-free software like video codecs

2 - the package manager is awful

and when you add a repositories it really slows down searching for packages
you go to search for a package and it then has to sync all of the external repos before it returns a simple search result

so you end up waiting 30 seconds to a minute just searching for a package

3 - constant kernel updates every couple of days

nothing more annoying than switching on the computer and having to install a kernel update and reboot every day
before you can get on with some work

4 - btrfs

i think you have to install a btrfs grub menu to replicate freebsd boot environments
Click to expand...
If you take a look at openzfs issues you will never use ZFS on Ubuntu because of the patches they add. I agree that Fedora is not an option for many reasons, GNOME be another one (I really despise their attitude).
 
If you take a look at openzfs issues you will never use ZFS on Ubuntu because of the patches they add.
Click to expand...
i guess that applies to zfs on linux in general and not just to ubuntu

i have used nixos in the past which has the option to use zfs
but i guess you would have the same problems
 
NapoleonWils0n said:
i guess that applies to zfs on linux in general and not just to ubuntu

i have used nixos in the past which has the option to use zfs
but i guess you would have the same problems
Click to expand...
I'm talking about custom patches that Ubuntu ships _over_ openzfs. I have used Archlinux with root-on-ZFS for long time before moving to FreeBSD and it worked perfectly fine with the standard kernel module.

https://github.com/openzfs/zfs/issues/16227 for example.
 
fmc000 said:
I'm thinking about following the same route but I will wait for the final release of PopOS 24.04 to make a decision. I know that btrfs works fine and that the system can be installed on an encrypted LVM device but for me the real hard part will be running a system without ZFS.
Click to expand...
btrfs working fine? No it does not. It kind of works most of the time if you are not using striped RAID and are reasonably lucky with single device or mirror setups.
 
recluce said:
btrfs working fine? No it does not. It kind of works most of the time if you are not using striped RAID and are reasonably lucky with single device or mirror setups.
Click to expand...
I know that, I will be using it on my laptop with a single disk, of course. But I also think that we're way off-topic so it's better if we leave Linux somewhere else :)
 
to summarize the situation

using ffmpeg as an example

1 - you already have packages like ffmpeg installed

even though ffmpeg isnt in the repo at present
it wont be removed from your system by doing a update and upgrade

when ffmpeg builds in the next cycle
and is restored to the repo then you will get updates for the package ( if there are any )

2 - you dont have ffmpeg installed

then you have to wait until the package builds and is restored to the repo before you can install it

hope i got that right
 
NapoleonWils0n said:
to summarize the situation

using ffmpeg as an example

1 - you already have packages like ffmpeg installed

even though ffmpeg isnt in the repo at present
it wont be removed from your system by doing a update and upgrade

when ffmpeg builds in the next cycle
and is restored to the repo then you will get updates for the package ( if there are any )

2 - you dont have ffmpeg installed

then you have to wait until the package builds and is restored to the repo before you can install it

hope i got that right
Click to expand...
The problem is when one dependency or the final desired program fails to build and disappears and then a dependency used elsewhere gets updated. The old successful port is not compatible with the new dependency so you have to either not upgrade the dependency, upgrade the failed-to-build port, or have a broken program until the update.

To use ffmpeg as a final port example, if x265 fails to build (updated or not) and opus receives a major update, you have to choose between not upgrading anything, upgrading opus and having ffmpeg fail to be able to talk to it properly (possibly always crash, possibly crash only when doing opus related operations), or remove a package that is expected to no longer work with the new opus. Your old x265 can stay and work but the old ffmpeg cannot if you want to update opus. Users are given a choice by pkg to accept removing expected broken packages or not upgrade at all. The users who are hit hard by this are those who have not yet installed ffmpeg or who said, "yes, remove it" when they wanted to actually say no (mistakes are one reason for backups; did they not make one?).

Linux has several ways they get around this.
  • Sometimes there are package archives: FreeBSD has dvd1.iso containing a few packages (=not many for anyone who suddenly started to get excited) within it and those older isos are available for quite some time. Older archives should have away to remove or warn about vulnerable packages instead of just allowing them to be installed; maybe it has changed but I haven't seen that happen in my minimal Linux use.
  • Sometimes they statically link libraries: binaries grow, There are some dependencies that were not well built to be statically linked and some dependencies conflict with other dependencies on 1 program when trying to do so. Dependency security issues/bugfixes require specific tracking and leads to full package removal+reinstall if incremental/patch capability isn't part of the package manager.
  • Sometimes they bundle libraries into a package; this replaces statically linked compile issues with potential conflicts though it can have disk space and execution performance improvements. A good package manager will reject installs of 2 packages writing the same file unless it also accounts for counting installs per file if the file would be identical (has to reject it otherwise) and can make sure uninstalls cannot remove such a 'shared' file. Dependency issues/bugfixes are similar to the statically linked case.
  • Sometimes they use containers (or whatever they call them) to bundle a program and all of its dependencies into its own clean environment. This brings more space use and RAM inefficiencies than statically linked binaries which were already generally worse than shared libraries. Dependency tracking issues are again similar to that of statically linked though in my browsing this technique is where them being neglected is accepted most of all. FreeBSD users seeking such isolation and version/incompatible program mixing use jails for such task though setup and maintenance commands are different and it is not directly compatible to released containers of programs.
  • Projects often release packages in the forms stated above for Linux; this is faster than waiting for any 3rd party to see the update and build it themself and offloads the hosting to be per project too. This sets up a workflow that is even less likely to be looked at by the project they offer packages for which can lead to bad actors having an easier time getting in problematic changes and means each project has to maintain new compilation offerings whenever any of their dependencies change. FreeBSD doesn't have as many supported release as there are Linux distros; most distros have a limited set of packages they officially support and release and most projects officially support hardly any distros while FreeBSD rarely is supported by separate projects with binary releases and when they do happen they are often of fewer FreeBSD versions, older FreeBSD versions + compatibility api, and for fewer FreeBSD architectures than the ports are often found for. Improving in this area is up to each project and out of the FreeBSD project's control. Projects working with the maintainers, or being the maintainers themselves (non-maliciously is assumed) would also improve the ports tree.
There are tradeoffs, both good and bad, for how things are currently done but I doubt the project will take up maintaining a repository of every last working copy + dependencies for every port anytime soon when they already store packages for multiple supported releases and across multiple architectures and would now need to start storing many copies of some dependencies on each release/architecture just because of some breakage or another.
 
sods law

as long as my currently installed packages dont get removed by updating and upgrading
then thats ok

the only time i actually had packages removed was when i changed from quarterly to latest
and the packages werent in the latest branch and so were removed

i guess web browsers like firefox and chromium will also be affected
as well as every package that use ffmpeg

so will every package that depends on ffmpeg now be frozen until the issue is fixed
or do we continue to get updates for programs like Firefox that use ffmpeg in the backend

at the moment im still updating and upgrading packages
but making sure to read the output before pressing "y"

is the best course of action to upgrade packages carefully
or not update and upgrade until the issue is resolved
 
ffmpeg can still be built (given the fix for multimedia/x265 is in place); but it might still include the vulnerability. So there shouldn't be any "local fallout" from simply upgrading packages.

I haven't yet found a detailed explanation of the CVE (only 7272 should apply, because 7055 concerns versions <=6.0) to make a guess about the actual impact on a regular server or desktop... The only hint that this might not be remotely exploitable on just any system is that "user interaction is required" - so maybe you need to feed some malformed data to ffmpeg to trigger that heap overrun.
 
Whenever I try to use binary packages, issues like this bite me (and the lack of configurable options that pull in unwanted / unneeded dependencies). I guess the old-school approach of building from ports remains the only safe choice.
 
For the truly safe, you need to build packages from ports in a clean environment and install/upgrade from those. Spent many years installing/upgrading from ports directly, usually with tools like portupgrade and portmaster. Interesting things happened as a result of trying to build/install from such nonclean environment. Maintainers/committers still like to know about and fix such issues where possible but by default they have migrated to clean environment tools like poudriere for their building so its not tested/found by default.

I thought the default is vulnerable packages are still built+distributed but flagged as such through the vulnerability database. Building from ports will require defining DISABLE_VULNERABILITIES. In my experience, not every CVE makes it into FreeBSD's database; Some don't apply to FreeBSD, some aren't known to the security team, and the list is manually updated by humans who have missed known issues when they meant for it to be reported.
 
Mirror176 said:
Maintainers/committers still like to know about and fix such issues where possible but by default they have migrated to clean environment tools like poudriere for their building so its not tested/found by default.
Click to expand...
Some maintainers doesn't consider issues with non-cleanroom-built environment as problem and requires trying clean-room build then reporting back again. This may be because almost all maintainers are purely volunteers and having not enough time for investigating everything.

So having clean-room builder on hand would be helpful, even though not using as daily drivers. But, for example, poudriere[-devel] has massive functionalities and can do very complexed build conditions.

I have a casual use-case that does builds for single PC running it and don't require web server for built pkg repo by local poudriere.
 
bsdcode said:
explanations for why packages have to be removed from the repository when dependencies fail to build are sound and understandable
Click to expand...
TBH, not really, why "accept" the patches and push builds if you get that much fallout? "Oh, yeah, no, that round of pkg building isn't going live"
 
msplsh said:
TBH, not really, why "accept" the patches and push builds if you get that much fallout? "Oh, yeah, no, that round of pkg building isn't going live"
Click to expand...
If the patch/changes doesn't include security fix, and the updated ports are dependened upon by a plenty of ports (regardless directly or indirectly), maintainers are expected to request exp-run (thoroughly test builds), then, fix fallouts before committing.

An example of exp-run here.

But if the patch/change include security fix, especially for critical vulnerabilities, there's not at all such a time. Delay for exp-run is not accceptable here.
Fallouts, if any, SHALL be investigated and fixed after the security fix lands.
 
sko said:
The problem was a changed patch file that was pulled from an external url by multimedia/x265. Due to the change the checksum did not match any more and the patch still isn't pulled into the quartely branch.

So to build all ports that require x265 from quarterly, one needs to cherry-pick commit 88c12b53a49ce5d7686cfcda9e2409ed3c1bd0f2 (or edit the distfile by hand / copy it from latest)
Click to expand...

Thank you for finding this! I opened PR 280833 for it and emailed the mailing list. Hopefully we can get it to land soon.

Edit: This is in! Should just be a few days for package builders to catch up and all will be well again.

cracauer@ said:
... and ffmpeg has security problems today. This whole affair will probably drag on for a while.

CVE-2024-7055 CVE-2024-7272
Click to expand...

I think we are okay on this front. We have ffmpeg 6.1.2 in latest and quarterly.

6.1.2 and 7.0.2 addresses CVE-2024-7075 according to here: https://ffmpeg.org/security.html

CVE-2024-7272 is for older versions, 5.1.5 and earlier.
 
tennea9 said:
And yes, there are always solutions... which are absolutely not simple to implement for the majority of users. For the more experienced, it's a usual scenario, we wait patiently and implement workarounds.
Click to expand...

I think the bigger concern here is why isn't Mozilla targeting FreeBSD as a first party platform for their releases. Port maintainers can only do so much. Nor are they getting paid for their work.
 
Unfortunately it seems bitbucket didn't communicate that they were going to alter a patch we download; its patching result was unaltered but the checksum we match to it fails because it is different:
Code: 
--- /usr/ports/distfiles/x265-asm-warnings.patch.orig   2024-01-23 21:58:34.000000000 -0700
+++ /usr/ports/distfiles/x265-asm-warnings.patch        2024-08-13 20:45:00.000000000 -0700
@@ -142,5 +142,5 @@
      %elif %1==4
          SBUTTERFLY qdq, %3, %4, %5
 --
-2.42.0
+2.45.2
There was a comment concerning it could be an issue when 3.5 was brought into the ports tree. It could have been avoided with something like
Code: 
--- multimedia/x265/Makefile.orig       2024-07-10 16:18:58.395378000 -0700
+++ multimedia/x265/Makefile    2024-08-16 04:57:25.387238000 -0700
@@ -95,6 +95,9 @@
 CFLAGS:=       ${CFLAGS:N-O*} -O0 -g
 .endif
 
+post-fetch:
+       @${SED} '$d' x265-asm-warnings.patch
+
 .for b in ${OTHER_DEPTHS:C/HI([0-9]+)P/\1/}
 EXTRA_LINK_FLAGS+=-L${WRKSRC:H}/$bbit
(I copied from terminal; spacing is probably mangled) and the appropriate update to distinfo after applying this type of patch. This would cut off the last line of the patch and compare distinfo records after. As the last line was stated by someone else to be a git version, I don't see the purpose of it on that patch.

Alternatively we could bundle the patch into our ports tree or download it from somewhere that doesn't alter it. 3.6 should no longer need the patch when we update to it someday.
 
  • Thanks
Reactions: sko
You must log in or register to reply here.
Back
Top