Introducing new project: Ravenports

Many of you already know me for various contributions to FreeBSD and FreeBSD Ports, most likely through the Synth repository builder that I wrote and still maintain.

I haven't made many commits to Synth over the last few months and there's an explanation for that. I've been developing a OS-agnostic ports and package builder system called Ravenports. Briefly there are 3 integrated components to the system:
  1. ravensource (files used to compile a port buildsheet)
  2. Ravenports (a collection of buildsheets, single file analog to a FreeBSD port)
  3. ravenadm (an administration / builder tool resembling synth)

The ravenports have major technical advantages over FreeBSD ports such as:
  • variant ports (similar to openbsd flavors, replaces freebsd master/slave ports)
  • subpackages (ports can create 1 or more subpackages, e.g. you can load just a fortran runtime library instead of pulling in the entire GCC)
  • multiversioning (you can use python2 and 3 simultaneously, php 56 and 71 simultaneously, perl 5.24 and 5.26 simultaneously etc, and build packages for all versions in the same build instead of picking just one default)
  • 2-4 orders of magnitude faster with regards to scanning and processing
  • due to compilation of ravensources into ravenports, syntax checking and linting are inbuilt, eliminating all sources of common contributor issues.
  • built-in support for alternative versions of stock ports, aimed at corporate users to truly tailor for their needs.
The other major advantage of course is that Ravenports is not anchored to a single operating system as FreeBSD ports and pkgsrc[1] are. It's a true "write once, build many" mechanism that require a minimal amount of platform-specific directives. This allows high-quality packages for all supported platforms, but the "virtual machine" approach means each supported OS/architecture combination has to be bootstrapped (probably by me) which is a long and complex procedure. This leads to the drawback of Ravenports only being currently available on FreeBSD/amd64 (11+), DragonFly, and Linux (all glib 2.6.32-based linux). Solaris/Illumos is next on my list.

I just revealed a new website: http://ravenports.ironwolf.systems/

General questions and commentary are welcome here. All specific questions are better sent to the two mail lists I set up. This is just a basic introduction for any people out there that are interested in following my new ports-based work.

-- John


[1] While pkgsrc is multisystem, it's NetBSD first as evidenced by numerous shims for other systems to use NetBSD headers, directories, clumsey boostrap process etc. Ravenports outclasses pkgsrc in terms of performance (several magnitudes) and non-NetBSD platform support.
 
John when do you expect DragonFly BSD to switch completely to Ravenports? I have seen few e-mails of yours on DF mailing list but nothing that will indicate that ravenports will replace D-Ports in 5.0 release which seems to be immanent (first one to feature HAMMER2 along sides of HAMMER1). How does ravenports compare to MirOS ports?

http://www.mirbsd.org/ports.htm

If it was me I would focus short term on port ravenports to OS X and possibly FreeBSD arm as well as making sure it works better than pkgsrc on some interesting Linux distro like Alpine Linux . I like the fact that you will attempt to boot strap raven ports to whatever is left of OpenSolaris. OpenSolaris lake of modern packaging system was a serious impediment to wider adoption.

Finally, what if any would be advantage of me using Ravenports on OpenBSD instead of native pkg_add? How would Ravenports play with syspatch? What about binary upgrades. Most of my current OpenBSD machines are installed before 5.5 release and I just keep binary updating them and doing pkg_add -u. Works like a charm.
 
John when do you expect DragonFly BSD to switch completely to Ravenports? I have seen few e-mails of yours on DF mailing list but nothing that will indicate that ravenports will replace D-Ports in 5.0 release which seems to be immanent (first one to feature HAMMER2 along sides of HAMMER1).

Hi Oko,
For sure, DPorts will be in place for the DragonFly 5.0 release. In fact, there's no stated objective to replace DPorts with Ravenports, even as a long-term goal. Obviously that's a logical assumption to have. I would assume to even consider such a thing, a number of other events have to happen, such as:
  • A large number of people use Ravenports exclusively (not as sandbox testing) for a while
  • The number of variants needs to probably triple (at ~1800 now, probably need at least 5000 variants)
  • specific popular packages need to come in (feedback welcome)
So if the dragonfly community provided a aggregate minimum port set, that set would need to be brought in first. At this point, people could choose their preferred ports system. M.Dillon has a medium preference to having a localbase base of "/usr/local" to match ports. Right now ravenports is using "/raven". Ravenports is designed to have any localbase, but a new system root and toolchain need to be generated. There are good reasons *not* to use /usr/local (obvious conflict avoidance; installing manually outside of ports systems, etc) though.

How does ravenports compare to MirOS ports?
I can't answer. I honestly have never heard of them before.
It seems that all BSD port system until now are "make"-based. Ravenports is not, so that's a major difference.
Just glancing at the link you provided, it seems that it would be the same difference as ravenports vs. openbsd ports

If it was me I would focus short term on port ravenports to OS X and possibly FreeBSD arm as well as making sure it works better than pkgsrc on some interesting Linux distro like Alpine Linux

The linux support isn't theoretical. It's already been tested on several like ubuntu, mint, arch, fedora, etc. One repository works on all as long as they are glib 2.6.32-based (most are, just exclude musl-based distros)

As far as "working better than pkgsrc" there's really no comparison. pretty much apples and oranges. A major difference is that ravenports is like freebsd: there are already prebuilt repositories for all platforms so building is not required. Moreover, due to "variants" there are way fewer build options because often a variant package is defined instead of an option, meaning both versions are available pre-built. Ravenports doesn't have a "bootstrap from source" option. You install it and go (again, only necessary if you actually want to build and not use the available packages).

Finally, what if any would be advantage of me using Ravenports on OpenBSD instead of native pkg_add?

This questions assumes enough demand and/or contributors to bring to OpenBSD. One obstacle is the gcc7 port with Ada support has been stagnant in its review. I've been waiting for that to be present because I need it for two reasons: 1) to bootstrap it and 2) to figure out which of the substantial number of patches to bring over. Anyway, assuming that all happens because you aren't the only OpenBSD person to ask:
  • it's not really important, but you'd be using freebsd's package manager here. You may or may not consider that an advantage, but currently that's the only package format that ravenports generates. Others are possible (e.g. the IPS format of OpenIndiana)
  • advantage: access to catalog which presumably have ports not available on openbsd or maybe not as new versions
  • same customization possibilities (either in-house or provided via commercial support). Ravenports is squarely aimed at corporate/enterprise needs which are often frustrated by the bleeding edge/low quality assurance aspect seen at both FreeBSD and pkgsrc
  • a disadvantage might omission of security aspects seen in openbsd packages but those in theory can be added back.
  • the ravenports website might have more advantages
How would Ravenports play with syspatch?
I'm unfamiliar with it. However, Ravenports provides almost everything it needs except for: libc, libm, libpthread, and a few other core system libraries. As long as the SONAME doesn't change, it should continue to work after system upgrades.

pkg(8) handles the binary package upgrades. In general it should be the same experience as freebsd. DragonFly and FreeBSD don't bump their system libraries often due to use of symbol versioning though. If OpenBSD bumps them every major release, each release will need its own repository, but pkg(8) can handle that too.
 
multiversioning (you can use python2 and 3 simultaneously, php 56 and 71 simultaneously, perl 5.24 and 5.26 simultaneously etc, and build packages for all versions in the same build instead of picking just one default)
Could Ravenports system solve the problem of 'forced' update of all dependencies during an update of one application?
For example, imagine that I have (relatively) old packages installed on my PC, and I want to update Firefox. Normally this update forces me to update many other packages and applications, that I'm not really want to update together with Firefox. And sometimes such update breaks some applications (for example, when the application is not available anymore in current portstree). With several versions of libraries installed, such 'massive' update could be skipped...
 
Could Ravenports system solve the problem of 'forced' update of all dependencies during an update of one application?

It depends.
case 1: you only use prebuilt packages and you don't build anything yourself.
While Ravenports are updated continuously (like FreeBSD ports), the updates are only published on a weekly basis. If you wait a few weeks, then try to update specifically firefox, chances are a number of packages will also get updated. This is normal and desirable. Trying to force replace just a single package will probably result in breakage, which is common to any system using freebsd's pkg(8) package manager.

Case 2: you build your own packages using only "stock" ports
You would update your local tree and command ravenadm to build firefox. It will build lots of packages (similar to synth). You'd generate a repo and install. Your updates are limited to only what is necessary for firefox. However, the result could be identical to case 1 above.

Case 3: you leverage Ravenport's custom port capability to lock in ports at specific versions
This is how individuals or companies would avoid unwanted churn and more importantly, unwanted newer versions. There are pros and cons, plus differences depending on if this is done personally or the port is maintained via service.

The bottom line is that pkg(8) is doing it's job and something complex like firefox will probably seen lots of dependencies updated depending on when the previous update occurred.
 
OK, thanks, for this answer.
So, the only case of 'multiversioning' would be the possibility to install two versions of software, present in the same version of ports tree (like php56 and php70), isn't it?
 
That's generally what I mean by "multi-version".
On pkgsrc, it's common to be able to install multiple versions of ruby, python, perl, etc., together without conflict but freebsd ports never had this ability. We're seeing a lot of python2 vs 3 issues for example. Variants are typically one or the others because they have a lot of common files. Multiversioning takes intentional work. In the case of PHP, ravenports might be the only one that has no conflict between installed PHP versions. (dunno for sure)
 
Hi John,

looking at the Ravenports configuration menu, the options A-J appear to set a build environment. How compatible is it with synth(1)? I was wondering if it would be possible to set two profiles, one for Ravenports, and another for the native ports, and switch from the former to the latter if an application would not be found in Ravenports.

Kindest regards,

M
 
mefizto, what I think you're really asking is if ravenports and freebsd ports can coexist. They can, but they'll be duplication (which is okay). Ravenports does not use ldconfig, so the realtime linker relies on the DT_RUNPATH/DT_RPATH flags being properly set in the ELF dynamic files, which they are. So Ravenports executables and libraries link to other libraries properly (not to ports libraries or system libraries by mistake).

However, using both ports means you'll have a lot of duplicates installed which is harmless. The first will probably be at /usr/local and the second copy will be based at /raven.

The ideal situation is that one ports system has everything you need though.
 
Hi John,

thank you for the reply. Actually, I sort of concluded that they can co-exist, so it was a hidden assumption.

What I was contemplating is how to bridge the time before - as I ardently hope - the Ravenports are adopted. So if there were a mechanism in the ravenadm, used as a primary source, that would notify the user that an application is not available as a ravenport, the user could then simply switch, ideally in the ravenadm, to build the application using the native ports.

But, this is just a wish as I do not understand all the intricacies.

Kindest regards,

M
 
mefizto,
The systems weren't meant to work together (ravenports is system-agnostic whereas ports are only available on FreeBSD and DragonFly). What I would recommend is to influence which software is ported first by stating your wishes here: https://groups.google.com/forum/#!forum/ravenports (in the only topic that currently exists). I have an idea which ports are most popular, but "hints" will be taken into account.
 
Can I use your solution parallel with Linux From Scratch to build my own hobbyist Linux Distro?
Sure, as long as you're using glibc. Musl-based systems aren't supported yet (and no work foreseen). However, supporting musl-systems isn't as hard as supporting a new platform from scratch.
 
Hi, some questions i have
1.
/raven/bin/ravenadm build fish
returns a lot of
"failed option check".
Any idea what is the problem ?
2.
How do I list all available ravenport packages.
Or all available ravenport shells ?
Normally on freebsd I just do : "pkg search ."
3.
How do I unpack or install the "ravenport builds"?
4.
/raven/bin/ravenadm build ada-util
Invalid port namebase: 'ada-util'
 
Hi, some questions i have
1.
/raven/bin/ravenadm build fish
returns a lot of
"failed option check".
Any idea what is the problem ?
2.
How do I list all available ravenport packages.
Or all available ravenport shells ?
Normally on freebsd I just do : "pkg search ."
3.
How do I unpack or install the "ravenport builds"?
4.
/raven/bin/ravenadm build ada-util
Invalid port namebase: 'ada-util'

You use the prebuilt packages --> /raven/sbin/pkg.

There is a on-line CATALOG too.
 
No idea, but how many lines of cobol are currently running on this planet and how many persons feel a KDE desktop environment is essential to have ?
 
Question, how to list all available packages in Ravenports and store them in a text file in order to grep , organize them by category ?
 
Ravenports don't have categories but keywords, and in short you can use pkg search[1] to find packages.

[1] Using the builtin ravenports pkg.
 
Back
Top