pkg: repository FreeBSD contains packages for wrong OS version: FreeBSD:12:amd64

[byrnejb]

A couple or three, (or more) points:

1. If constraints of resources require then of course disposing of packages repositories for EOL releases can hardy be argued against. However, I point out that with the cost of online disk storage being what it is, and considering the bandwidth demanded would likely be little more than zero, it is hard to believe that resource constraints drive this policy. My own observation is that the current main branch of the ports tree amounts to 112Mb when downloaded as a zip file. Granted, these are source files and not binary packages.

2. The solution that I have applied, in the absence of any change in policy (which seems to me to be very unlikely and therefore not worth discussing), is to run pkg fetch -a periodically and keep things in my /var/db/cache/pkg/ directory on my workstation. From there I can move them to any system, local or remote, that needs a particular piece of software. I have not gone to the extent of sub-dividing the cache into OS versions, but that would be trivial to do.

3. As the ports tree is universal (in that it is not OS release specific) and is now in git it is possible to select a date, find the tag or commit nearest to, but before, a given date, download the packages needed from that point, and build them. It is not entirely straight forward but it can be done with a little effort, as I discovered through experiment. In fact, the github has quarterly branches going back to 2014. One can simply checkout the branch desired and go from there.

4. Respecting old hardware and software:

We have a bespoke business application system written in a 1980's 4GL. This runs on a computer host built in 1994. We are installing an emulator for the OS used by this system to run in a Rockly Linux VM guest of a BHyve hypervisor running on FreeBSD-12.3 or 13.1 and Intel hardware. That emulator will host the business application.

Moving the application to new hardware from the OEM requires an expenditure of more than $300,000.00 USD just for the licence transfer fee. And this would not solve the problem that the necessary hardware is no longer sold nor supported by the OEM. Rewriting the software to use more current technology would cost considerably more. We know because we tried. The cost of the emulator software is $40,000.00 CAD installed and verified. The Intel hardware cost ~$3,000.00 CAD.

By taking this approach the business also avoids the costs of retraining staff and the loss of business transaction history that would necessarily follow conversion to a different application system. It also preserves the investment in custom programming carried out over the course of ~40 years. That is long term. Five years is nothing.

And we have all of the source code maintained in a Git repository since October 29, 2009.

 

[tomaz]

I was also puzzled by the statement that continuing to store the packages costs space which costs money, because cloud storage is so cheap these days and the amount of space required is so small (and, as you say, the amount of bandwidth consumed by old versions would be very small). I think it's more a consequence of a deliberate "c'mon upgrade, it's not like it hurts" philosophy.

But it's not just about "it works, so why upgrade". No mission critical system should ever go through a live upgrade. It runs. Then when you have an upgraded version, it's an entirely new piece of hardware, with the new software, you test it and test it, and test it, and test it some more, then you replace the old system at a time when minimal downtime is acceptable, and be ready to put the old system back in place if the transition fails. You just can't upgrade, cross your fingers and hope - and suffer days of downtime if it goes wrong.

Thank you for your suggested solution, which is really what I'm looking for. I don't have the room on my system, unfortunately, for packages I'm not going to use (it's an embedded device), nor for development tools. I guess I'm going to have to install a new copy of the old system locally and try to build the missing package and copy it to the remote system and install it that way. But building software out of the package hierarchy can be a messy process, if you miss some dependency. I guess that's the whole point of the package system. I've not built FreeBSD packages before, how feasible is it to download the source code of the old packages and build the actual package on a separate (newly installed) development system based on the same old version I have on my remote hardware and then use the package system to try to install it, then see what dependencies it comes up with, then build those and so on?

 

[zirias@]

FreeBSD won't ever support you by hosting repositories for EOL systems. End of story here.

Anyways, If you really think you absolutely need this, you have all the tools at hand to do it yourself. Although the common practice for "live systems" nowadays is having them multiple times, taking one instance out for update, test that, and if everything is fine be done with it (I won't describe the whole process now...)

edit: I'll add my personal, non-official opinion why I think FreeBSD should never start supporting this. Less important, it would be an interesting oxymoron to support something not supported any more. More important, without real support (by at least delivering security patches), it would be very bad reputation if some consumer of such a repository would be hit by an exploit of a previously known vulnerability.

 

[tomaz]

I ran

setenv IGNORE_OSVERSION yes pkg install speedtest-cli

I was told that this required "pkg" to be also upgraded. The install of speedtest-cli failed and install ended with
Checking integrity... done (0 conflicting) [1/1] Upgrading pkg from 1.15.6 to 1.16.1... [1/1] Extracting pkg-1.16.1: 100% You may need to manually remove /usr/local/etc/pkg.conf if it is no longer needed. Shared object "libarchive.so.7" not found, required by "pkg"

Now when I run "pkg", it fails with
Shared object "libarchive.so.7" not found, required by "pkg"

So what now Mr Smart-Arse "minor version upgrades won't break anything"?

 

[shkhln]

because cloud storage is so cheap these days

If storage is so cheap, you can probably set up a mirror yourself. (That's probably somewhere around one new HDD per week.)

 

[Alexander88207]

So what now Mr Smart-Arse "minor version upgrades won't break anything"?

I'm sure this refers to after an upgrade and not when ignoring an upgrade.

 

[tomaz]

If storage is so cheap, you can probably set up a mirror yourself. (That's probably somewhere around one new HDD per week.)

No I can't because my time is not cheap. I doubt it's one HDD per week, but one HDD per week is a lot cheaper than the cost of everyone's wasted time (even my wasted time alone). At any rate, I'm not interested in arguing, I'm looking for a solution. I've already decided not to use FreeBSD in the future, but right now I'm stuck with the consequences of my past decisions. Enjoy congratulating each other in your echo chamber, I hardly think I'm alone among users in reaching the conclusion I've reached.

 

[elgrande]

Now when I run "pkg", it fails with
Shared object "libarchive.so.7" not found, required by "pkg"

Reminds me of this:
/*
* Your warranty is now void.
*
* I am not responsible for bricked devices, dead SD cards,
* thermonuclear war, or you getting fired because the alarm app failed. Please
* do some research if you have any concerns about features included in this ROM
* before flashing it! YOU are choosing to make these modifications, and if
* you point the finger at me for messing up your device, I will laugh at you. Hard. A lot.
*/

 

[shkhln]

If storage is so cheap, you can probably set up a mirror yourself. (That's probably somewhere around one new HDD per week.)

I don't want leave the question open to speculation, so

#!/usr/local/bin/ruby
# encoding: UTF-8

require 'json'

if not File.exist?('packagesite.yaml')
  system('fetch http://pkg.freebsd.org/FreeBSD:13:amd64/release_1/packagesite.txz && tar -xf packagesite.txz')
end

nbytes = 0

File.open('packagesite.yaml').each_line do |package|
  nbytes += JSON.parse(package)['pkgsize']
end

puts "#{nbytes.to_f / (1000 ** 3)} GB"

prints "only"

106.5492852 GB

Keep in mind this a snapshot of a single repo, though.

 

[tomaz]

Reminds me of this:
/*
* Your warranty is now void.
*
* I am not responsible for bricked devices, dead SD cards,
* thermonuclear war, or you getting fired because the alarm app failed. Please
* do some research if you have any concerns about features included in this ROM
* before flashing it! YOU are choosing to make these modifications, and if
* you point the finger at me for messing up your device, I will laugh at you. Hard. A lot.
*/

Yes, force people to either upgrade (which they don't want) or tell them to use the ignore flag, since it's only a minor version upgrade, so ABI hasn't changed so it should all work, but it's all unsupported, then when they do either thing and it goes horribly wrong, run away like a coward and hide.

Anyway, I fixed my problem. In case it helps anyone else, after the failed "pkg" upgrade, I fished out "pkg-1.15.6.txz" and "pkg-1.16.1.txz" from /var/cache/pkg. I examined the manifests. I manually deleted the files installed by the incompatible "pkg-1.16.1.txz", look what the post and pre uninstall scripts did and replicated their effect, then manually installed the contents of "pkg-1.15.6.txz" and did the same with the scripts, thus restoring my "pkg" to the known working version. For some reason, after this, pkg started working and is no longer stopping me installing new packages as it was in the same way the O.P. was prevented from installing new packages and as I was when I was first brought to this thread. I wonder why? I was then able to install "py37-speedtest-cli-2.1.2" which is the package I wanted to install. That's an out of date package known to be broken (with a known fix in 2.1.3), so I'm guessing it's not from the latest repo? Anyway, 2.1.2 can be made to work with a small patch, which I manually applied, and now everything's doing what I wanted it to - the original version of the O.S. is still there doing what it's supposed to do, and I installed my desired package.

 

[tomaz]

I don't want leave the question open to speculation, so

#!/usr/local/bin/ruby
# encoding: UTF-8

require 'json'

if not File.exist?('packagesite.yaml')
  system('fetch http://pkg.freebsd.org/FreeBSD:13:amd64/release_1/packagesite.txz && tar -xf packagesite.txz')
end

nbytes = 0

File.open('packagesite.yaml').each_line do |package|
  nbytes += JSON.parse(package)['pkgsize']
end

puts "#{nbytes.to_f / (1000 ** 3)} GB"

prints "only"

106.5492852 GB

Keep in mind this a snapshot of a single repo, though.

18TB WD181KRYZ is $450 on Amazon and can hold 170 such snapshots. To store the last version of each package of each EOL release surely can't be more than a few hundred releases' worth - in the entire history of FreeBSD. So the total cost of storage for everything would be about $1000. Nothing compared to even one person having to go through the hassle of manually installing their needed packages because they've been removed. The whole thing has nothing to do with the cost of storage and everything to do with control freakery.

 

[shkhln]

I doubt it's one HDD per week, but one HDD per week is a lot cheaper

You'd think so, but then somebody has to buy a yet another server to put them in. The foundation operates on a very tight budget.

than the cost of everyone's wasted time (even my wasted time alone).

Would that make "everyone" angry enough to stop donating money? Otherwise that comparison doesn't make any sense.

 

[shkhln]

To store the last version of each package of each EOL release surely can't be more than a few hundred releases' worth - in the entire history of FreeBSD.

I already wrote in this very thread, there are per-release snapshots of the repositories, although they aren't exactly supported and I don't think anyone actually promised to keep an archive of them.

 

[Jose]

No I can't because my time is not cheap.

18TB WD181KRYZ is $450 on Amazon and can hold 170 such snapshots. To store the last version of each package of each EOL release surely can't be more than a few hundred releases' worth - in the entire history of FreeBSD. So the total cost of storage for everything would be about $1000.

Let's summarize:

1. You don't want to work for it
2. You don't want to pay for it
3. You just want someone to spend their time and money and give it to you for free

Maybe it'll help if you bang your spoon on the high chair some more.

 

[monwarez]

I ran

setenv IGNORE_OSVERSION yes pkg install speedtest-cli
...

What is the reason to change this default, it is a default for a reason: to not shoot on your foot.
Just looking at the manpage:

IGNORE_OSVERSION: boolean [FreeBSD specific]
Ignore FreeBSD OS version check, useful on -STABLE and
-CURRENT branches. Default: NO.

You are neither running -STABLE nor -CURRENT so enabling this settings allow yourself to shoot on your foot.

 

[zirias@]

So what now Mr Smart-Arse "minor version upgrades won't break anything"?

You didn't do the minor version upgrade but instead tried something completely unsupported.

I would very much prefer if you could get around without insults. It's your fault you don't upgrade your systems as you're supposed to. FreeBSD support cycles are predictable and have overlaps that should be long enough for any admin worth their money. If you still disagree, maybe look for an OS that matches your expectations better?

 

[zirias@]

You are neither running -STABLE nor -CURRENT so enabling this settings allow yourself to shoot on your foot.

Just to complete this, it allows "foot shooting" on any branch... The thing is, users of -STABLE or -CURRENT might occassionally need it and are expected to know exactly what they're doing, otherwise they should run -RELEASE

 

[mer]

Initial cost of media do actually do the storage is not really the issue. There are also recurring costs for things like electricity to power the media, to power the cooling for the devices.
Cloud storage: ok for some things but often at the mercy of whomever is providing it. I can easily recall a few times over the past year or two where Amazon AWS had issues and put a whole bunch of websites off the air for a while. It's not bad if the website is only hosting pictures of your cats, but if it's your online store you are losing money.

I just don't understand the argument that FreeBSD should be keeping around a copy of applications that were built against a version of base that has been end of lifed.
Nothing prevents anyone from still running that EoL'd version, heck there are probably people still running Win95. But an expectation of applications being kept around? My opinion, no.
I also have a realistic expectation that it doesn't matter what I say to folks that truly believe the project should be keeping things forever.

 

[byrnejb]

I just don't understand the argument that FreeBSD should be keeping around a copy of applications that were built against a version of base that has been end of lifed.

And yet the entire FreeBSD ports tree, containing every change ever committed, is on github going back to 2014Q1.

 

[SirDice]

And yet the entire FreeBSD ports tree, containing every change ever committed, is on github going back to 2014Q1.

It goes way further back than that. 2014 was the first year the quarterly branches were introduced. The ports tree itself existed long before that, and so does its history. The entire subversion (and CVS before that) history was merged to git too.

But just because you have a history of the ports tree, that's just how a version control system works, doesn't mean you should keep everything that it ever built around.

 

[mer]

And yet the entire FreeBSD ports tree, containing every change ever committed, is on github going back to 2014Q1.

Sure, I'll play.
Because it's the source code and it's in a SCM tool and not "built binary packages for every single architecture that was supported against every single release".

Big difference and the discussions about the "cost of storage is so cheap", well someone still has to pay for it. If you want to pay for the storage and the recurring costs to keep a version of your desired ports tree around, then why not send a donation to the Foundation earmarked for that?

 

[tomaz]

There are people - like the OP and myself - who have FreeBSD installations in the wild, which it turns out are EOL, where upgrading the O.S. is not an option, and where we need to install new applications. We already know that if we upgraded the O.S., we could have access to repositories - because the computer itself tells us this. Telling us that again is completely unhelpful because upgrading the O.S. is not an option under our circumstances, and we've made that clear.

I've been using other open source O.S.'s but I've never come across a problem like this - package repositories disappearing - before, so it's not something I was even contemplating as a possibility until I was suddenly unable to install new packages. I'm sure it's written somewhere, but I - and I am sure most users - don't have the time to read every Wiki page before setting up an appliance to solve a problem.

Contrary to what some people seem to think, our time is not free and we're not necessarily using open source software because we don't have to spend money (we certainly have to spend money on hardware and our time - which is more valuable), but for other reasons such as security, privacy, auditability etc. One of the reasons for using open source software is that you are in control of your computer and what software runs on it, rather than the computer being in control of you. Obviously if the computer forces you to "upgrade" your O.S., then you're no longer in control of your own computer. Having said that, while I'm happy to spend money, I would certainly never give a dime to an entity, commercial or otherwise, whose techies think it's OK to insult me.

If you have something helpful to tell us (e.g. some of us may be able to build the deleted packages from source, time consuming as it is, it may be the least worst option - we may have developer experience but not FreeBSD specific knowledge as to how to do this or where to look for for the correct version of source code) - then please kindly do so. Otherwise, do you really have to speak?

Obviously FreeBSD is going to continue to delete packages that had already been made available in repositories, I get that. And that's OK, it does make FreeBSD totally unfit for purpose for many people, it certainly does for me, but it's your playground, make it inhospitable to other people and play in your tiny sand pit if that floats your boat.

I won't use FreeBSD in new installs in the future, but unfortunately in the meantime, I do have a few devices in the wild which run FreeBSD and FreeBSD derivatives which I cannot replace any time soon, which I do need to maintain, where I do need to install new software from time to time and where upgrading the O.S. is n-o-t a-n o-p-t-i-o-n.

Thanks.

 

[zirias@]

There are people - like the OP and myself - who have FreeBSD installations in the wild, which it turns out are EOL, where upgrading the O.S. is not an option, and where we need to install new applications.

Too bad it's not supported.

Thanks.

 

[shkhln]

What's next? Complaining versions of applications in the old repos (as I already said twice in this thread we actually have those) are not new enough?

 

[cy@]

Let's say for example a person is running FreeBSD 8.1 (I know of a company, not mine, still running this version of FreeBSD and an old employer of mine was running 4.4.0 until recently). Let's say they want to install a port. They could,

git clone https://github.../ports.git
git checkout release/8.1.0 or git checkout release/4.4.0

With this they have the ports collection as of the day 8.1.0 was released. Just cd to the port and make install.

Anticipating someone saying, "but how do I git clone without git on the box?" Easy. Find a system that has git, Linux, FreeBSD or whatever, then ./configure and make. It's not hard to do.

Anticipating: "but that's too hard." Easy. Doing builds by hand, using ftp or schlepping files over the network should be easy for any sysadmin. When I switched careers from IBM mainframe to UNIX in 1992, all we had was ./configure, if even that. We made it work. When running old unmaintained software, these re the things a person needs to do.

As for maintaining an old EOL system, a system that will never receive security updates again on one's network. How's that secure? Think about that for a while.

As for maintaining old versions of packages for releases that are EOLed just in case some random person might want some random package years down the road is ridiculous. Think about that a little.