Archiving FreeBSD with 3rd-party packages

K

Korger

Hello,

I'd like to know if there are any public servers that store legacy FreeBSD versions with the full package repository that was compiled for that system. If not, how could I create such an archive myself?

I find this important because there are many packages that become deprecated and eventually dropped from the repository, without any replacements. For example, I'm still using a few 10-15-year-old NVidia cards. nvidia-driver-304 no longer works with the current X server, and nvidia-driver-340 seems to be on its way out as well (under xorg-7.7 it's giving me an ABI incompatibility warning, which so far can be ignored, but probably not for long). Once it doesn't install any more, I can't use any of the old NVidia cards, so I'd like to keep around a full 3rd-party archive which I can use for those machines.

Similarly, compiz used to be my favorite window manager, but having been unmaintained for a long time, it no longer works under a recent X window system. It would be great if I could keep around an older archive, so I can install and continue to use legacy software.

What's the best way for this?
 
I understand that, that's why I was talking about archiving. As of now every package in the repository works with 12.4-RELEASE. So if I take a snapshot of this repository, and store it myself, in ten years I should be able to install 12.4 on an old computer that doesn't support whatever release is going to be current in ten years. But how can I do that? Is anyone doing it already?
 
But the packages are changing on a daily basis - the packages for 12.x today will be different tomorrow and the amount of change can be huge if some of the core dependancies have changed.

I suppose you could choose an arbitrary set of dates (one week after release) and try and take a snapshot of all the binaries at that date, then another a month later, and maybe a snapshot a week before EOL ... or something.

But what if the future you wanted something even more specific than that - 12.x with MySQL 5.6.x rather than 12.x with MySQL 5.6.y that's in your snapshots?
 
VladiBG said:
Click to expand...

Yes, but ...

Looking only at amd64, I've just explored lots of that. As far as archived packages are concerned, there's only those for releases 5.1 through 9.2.

Index of /mirror/FreeBSD-Archive/ports/amd64/

gpw928 said:
It's a moving target. The package management evolved over a long time, and changed radically in 2014.
Click to expand...

Indeed, though there was overlap before 2014 (9.3 & 10.0) between when what's now pkg was 'pkgng' and the older pkg_tools were pkg_add and friends.

I still have a laptop that was installed with 9.2, upgraded from source to 9.3-stable, still with old pkg_tools, that I daren't touch until my 12.4 laptop runs as well.

richardtoohey2 said:
I suppose you could choose an arbitrary set of dates (one week after release) and try and take a snapshot of all the binaries at that date, then another a month later, and maybe a snapshot a week before EOL ... or something.
Click to expand...

This was never a problem before; packages were made from the ports tree at the time the release was built, matching the ports archive included in that release.

Even now there's a separate package set used to assemble the 1500 or so packages included on the DVD, despite suffering from poor maintenance through at least 12.x and 13.x

So for releases from 5.1 through 9.2, archived package sets already exist, and from 9.3 and 10.0 on, releases contain ports and sources - but not distfiles - needed to build packages.
 
Old packages are unlikely to work with new FreeBSD releases, so you'll have to use old FreeBSD versions as well. In that case you can just checkout ports tree for that period of time, and build what you need yourself, but... I guess you'll have the same problem with old distfiles being deleted :)
 
VladiBG said:
The others can be found in the iso images. Look around in the ftp archive.
Click to expand...

Sorry, but I've just spent another hour down all those rabbit holes. There are no packages in any of the ISO directories, just .ISO and .img files for each release.
Code: 
Index of /mirror/FreeBSD-Archive/old-releases/ISO-IMAGES/
Index of /mirror/FreeBSD-Archive/old-releases/amd64/ISO-IMAGES/
Index of /mirror/FreeBSD-Archive/old-releases/amd64/amd64/ISO-IMAGES/

Again, I can only find release distribution files containing a (presumably complete) packages/ directory for 5.1 to 9.2 (i.e. pre-2014).

i386/ goes back to 1.0, then 2.0.5 to 13.2, but again with no packages/ beyond 9.2.

If you find any after 9.2, please post a link?
 
Probably because they think they'll need something they didn't think of later.

Interesting question. I would think you'd want to take a snapshot of ports right before the time UPDATING gets touched and then the final time ports is built to target a release.
 
If you want to create an archive of compiled packages, I can suggest looking at Poudriere and git. FreeBSD's sources git repo goes back to 1.1.1-RELEASE. Also, the ports repo is on git, as well. It is a LOT of work (and a lot of metal and processing power) to compile what you need, but not impossible. Good luck :rolleyes:

Not to mention you'll need a lot of storage space for a complete repo of both sources (which may or may not even be available) AND the resulting packages.
 
fetch https://pkg.freebsd.org/FreeBSD:12:amd64/latest/packagesite.txz
tar xzf packagesite.txz packagesite.yaml
sed -e 's#.*path":"#https://pkg.freebsd.org/FreeBSD:12:amd64/latest/#' -e 's/pkg".*/pkg/' packagesite.yaml >URLs
Then fetch packages.

or
set ABI and PKG_CACHEDIR in pkg.conf(5) and pkg fetch -a.

If you want to create packages from files installed on your computer, use pkg-create(8).
 
I did some digging and found this, to answer my own question: http://mirrors.xtom.com/freebsd-pkg/

This site has been archiving all the 3rd-party packages since FreeBSD 10. However, it's not an official site, and I don't know their retention policies, or if the packages can be trusted. I'm surprised more people don't take an effort to preserve this content.

The DVD iso found at Index of /mirror/FreeBSD-Archive/old-releases/ISO-IMAGES/ stores only a small fraction of the 3rd-party repository. However, I believe using the DVD together with the xTom repo should give us a complete system, at least back to FreeBSD 10 while it's still there.

I'm also curious how I could archive the current release myself, in case I don't trust that xTom will be around indefinitely? I'm looking to get the content under http://pkg.freebsd.org/FreeBSD:12:amd64/quarterly/, but when I click on the All directory, it gives me a Forbidden error. How does that work, since I can install from the same directory using pkg just fine?
 
Korger said:
how I could archive the current release myself,
Click to expand...
I told you about looking into git. Once you pull the git repo(s) onto your own disk, you can locally check out and compile both FreeBSD source AND packages at any point in time. Just find a commit that has a timestamp you like.

Yeah, it's a lot of work, including learning the proper git commands, and figuring out how to navigate the repos and making use of them.

Korger said:
I'm surprised more people don't take an effort to preserve this content.
Click to expand...
Because it's a lot of work, and FreeBSD is largely a volunteer effort by a tiny non-profit? It's amazing enough that the effort even lasted 30 years already, on par with Linux distros like Debian and Slackware...
 
Korger said:
I did some digging and found this, to answer my own question: http://mirrors.xtom.com/freebsd-pkg/
Click to expand...

Well dug.

Korger said:
This site has been archiving all the 3rd-party packages since FreeBSD 10. However, it's not an official site, and I don't know their retention policies, or if the packages can be trusted.
Click to expand...

Nothing "ain't nohow permanent" but going on their position and network peers internationally, I'd expect their mirrors would have to be reputable. They may have a FreeBSD mirror agreement?

Korger said:
I'm surprised more people don't take an effort to preserve this content.
Click to expand...

Using similar parsing to the above by Charlie_, or the list from xTom, totalling filesizes in a repo is straightforward. I'll try tomorrow if noone's faster - but clearly huge.

Korger said:
The DVD iso found at Index of /mirror/FreeBSD-Archive/old-releases/ISO-IMAGES/ stores only a small fraction of the 3rd-party repository.
Click to expand...

Very small; at 12.3 about 1130 packages, mostly xorg, KDE and gnome3. By 12.4, gnome missed out due to renaming so only some 930 packages, out of some 32k.

Code: 
bsdconfig packages

                    Package Selection
   Please select a category to display

   > Review           Review/perform pending actions
   All                32343 packages; 576 installed; 1 selected
   accessibility      51 packages
    afterstep         20 packages
    arabic            8 packages
    archivers         310 packages; 10 installed
    astro             149 packages; 1 installed; 1 selected
    audio             942 packages; 15 installed
   benchmarks         121 packages; 1 installed
    biology           247 packages
   cad                162 packages
    chinese           136 packages
    comms             233 packages
    converters        231 packages
   databases          1231 packages; 1 installed
    deskutils         293 packages
    devel             8698 packages; 28 installed
    dns               270 packages
    docs              22 packages
   editors            398 packages; 2 installed
    education         77 packages
    elisp             383 packages
    emulators         207 packages; 1 installed
    enlightenment     14 packages
   finance            166 packages
      (+)                                   25%
                 <Select>        <Cancel>

Install, Re-Install, or Un-install selected packages and dependencies
(page 1 of 4 at 11 Jan 2023)

Maybe I should try again to submit my bsdconfig patches to enable that, allowing browsing each category with oneline descriptions - but it's still incomplete WiP. It does work fully on the DVDs.

Korger said:
However, I believe using the DVD together with the xTom repo should give us a complete system, at least back to FreeBSD 10 while it's still there.
Click to expand...

And then the only release packages missing from 5.1 up (amd64) would be 9.3.

Korger said:
I'm also curious how I could archive the current release myself, in case I don't trust that xTom will be around indefinitely? I'm looking to get the content under http://pkg.freebsd.org/FreeBSD:12:amd64/quarterly/, but when I click on the All directory, it gives me a Forbidden error. How does that work, since I can install from the same directory using pkg just fine?
Click to expand...

I recall reading that fetching All/* was forbidden, maybe to prevent DoS? In any case it's all in packagesite.yaml as above.

The port distfiles are another matter. Many, maybe most will have moved from e.g. sourceforge to the scarily overcentralised github.

Hope you've got plenty of spare tebibytes <&^}=
 
Sadly, last time I moved, I threw out dozens of Wallnut Creek CD/DVDs, going back to FreeBSD 3.0.

There's some quite old ISOs in the Wallnut Creek collection. Back then the CD/DVDs came with lots of packages on additional media, as Internet access was not so common (or fast).
 
You must log in or register to reply here.
Back
Top