What are the freebsd O.S. things you don't understand ?

ralphbsz said:
I'm sure any of these problems can be worked around within Bazel.
Click to expand...

I haven't used it myself, but I heard an interesting podcast about a build system called Pants 2. It's intended to address some of the issues you brought up - a couple things that come immediately to mind are 1) heterogeneous toolchains in a monorepo 2) multi-platform configurations 3) build caching.

ralphbsz said:
I happen to have a functioning (barely functioning, overly complex, and brittle) system of makefiles with if statements that call other makefiles, and maintaining that is currently easier than finishing switching.
Click to expand...

One thing I don't understand is how we don't see lots of build problems with the makefile system. I read Recursive Make Considered Harmful and it's clear to me how recursive make can lead to indeterminate DAGs among the dependencies - which would particularly cause problems when doing parallel builds. But, we make -j4 buildkernel all the time... so apparently src isn't running into those kinds of issues. Maybe because so many devs are running those commands so many times, they've worked out any conflicts?
 
The build systems situation reminds me of this xkcd: https://xkcd.com/927/

Every so often I learn about "new", "modern" build system written in currently overhyped language that will fix all the shortcomings of all previous build systems, and in the end there's usually yet another half-done solution for someone's pet project.
 
yuripv79 said:
Every so often I learn about "new", "modern" build system written in currently overhyped language that will fix all the shortcomings of all previous build systems, and in the end there's usually yet another half-done solution for someone's pet project.
Click to expand...
Yep.
I'm all for actually trying to advance things, but one has to really be advancing not simply reinventing.
A pattern that works for one language, one directory structure may not work for another or at best be difficult to fit things in.

Simplistically, a build system takes a list of inputs, does stuff and creates outputs.
The devil in the details comes when you try to avoid rebuilding things that don't need to be rebuilt (dependencies), pull from multiple directory structures, build in parallel, use same source tree for multiple outputs (architectures).
Language to specify sources/dependencies/outputs needs to be consistent and understandable.

I've used a few different ones over the years, they all have their issues, they all have good points.
The one consistent point I've come across is:
pick the build system before you start writing code. They all seem to be easier to use/maintain if you start that way. Trying to backfit something in a multi-thousand line source tree is a lot of work.
 
It looks like we've fully switched over to build systems now?

Yeah, I was in love with Bazel for a while, but it turned out to be too complicated. The thing I liked about it is how it zapped transitive dependencies. I think it's been fully Mavenized now, unfortunately.

Some years ago some sheltered souls left Google and discovered the mess of Java build systems the rest of us had been struggling with. They immediately missed the Google internal build tool, Blaze that is gee-whiz cool from what I've heard. At least three build tools were born of their wailing and gnashing of the teeth: Bazel (a spoonerism of "Blaze"), Pants, and Buck.

Unfortunately the bad habits acquired from using Maven and its Evil successor, Gradle won out, and I believe all three engage in the same criminal download-the-Internet dance to build stuff. I believe the Java compiler is at least partially to blame. For some reason, the folks at Sun decided to break the one text file -> one compiled file invariant that had held for all previous compilers. This means that when I javac foo.java, I could actually be compiling several files. This breaks the source freshness check that relies on comparing the creation time of foo.class with the modified time of foo.java. Let's not even get into anonymous classes. Children could be reading this.

Even worse, this crappy approach has now infected the build tools for languages that have sane compilers, like Rust. Even though the name "cargo" in a none-too-subtle dig at the cargo-cult programming approach it encourages, it's still the de-facto preferred way to build and distribute Rust programs. I wonder if anyone builds Rust with plain ol' rustc and makefiles.

I was into Tup for a while, but I ran into some problems with it on Freebsd that I couldn't solve. FUSE programming is documented pretty badly in general.
 
  • Like
Reactions: mer
patmaddox said:
3. The development process. I see commits, and discussion on bugs.freebsd.org and reviews.freebsd.org, and on hackers@ mailing list. It seems to me that there’s not much clear discussion of the things people will be working on - not much direction, or roadmap. My understanding is that there are devs-only mailing lists (is that true?) and I wonder if some of that discussion takes place there. Are there people with a general sense of the direction FreeBSD is headed? If so, who are they communicating that with, and how? I would love to get better insight into the development process. I follow bugs and reviews a bit, but there’s a lot of action and it’s overwhelming and not sure where to focus.
Click to expand...

There is a mailing list, but it isn't technical. That kind of planning you mention does not apply to volunteers and those who get paid for FreeBSD hacking follow their company's preferences. There is no central planning for FreeBSD's direction. Nor for Linux. Somebody writes some code for some reason, that is all that matters.
 
Jose said:
Yeah, I was in love with Bazel for a while, but it turned out to be too complicated. The thing I liked about it is how it zapped transitive dependencies. I think it's been fully Mavenized now, unfortunately.

Some years ago some sheltered souls left Google and discovered the mess of Java build systems the rest of us had been struggling with. They immediately missed the Google internal build tool, Blaze that is gee-whiz cool from what I've heard. At least three build tools were born of their wailing and gnashing of the teeth: Bazel (a spoonerism of "Blaze"), Pants, and Buck.
Click to expand...

FWIW I hated Blaze when I was there. But then I had to use it with Lisp, and Lisp has build tools of its own. I really dislike build systems where a 1-line change takes more than 7 second to have running code for debugging. To make it more off-topic:
medium.com

Software development at 1 Hz

People keep saying that a one line source code change should take one second to be “effective”. “Effective” meaning that you get some form…
medium.com medium.com

That's my take on the issue of build systems. For me the minimum build time matters more than the time used to build the whole system.
 
I understand and master all aspects. No? Really? :p

I choose FreeBSD because you have a chance to understand an aspect if you pay time for that. I think it's more complicated with other OSes, not to mention Windows, but also Linux. I left this last more or less for this reason.
 
cracauer@ said:
There is a mailing list, but it isn't technical. That kind of planning you mention does not apply to volunteers and those who get paid for FreeBSD hacking follow their company's preferences. There is no central planning for FreeBSD's direction. Nor for Linux. Somebody writes some code for some reason, that is all that matters.
Click to expand...
Understood. I’m not asking about a formal roadmap, but more like what the Project Model doc says about new feature development (emphasis added):

As the requests are prioritised by the individual developers on the basis of doing what they find interesting, necessary, or are funded to do, there is no overall strategy or prioritisation of what requests to regard as requirements and following up their correct implementation. However, most developers have some shared vision of what issues are more important, and they can ask for guidelines from the release engineering team.
Click to expand...

It’s the general sense of shared vision I’m interested in, and how it comes about.

Here’s an example: it‘s possible to write FreeBSD kernel modules with Rust. I suspect that at least one developer has brought up the question of whether to permit Rust in src. Such a discussion has not taken place on freebsd-hackers@. I don’t know if such a discussion has taken place at all, but from what I am learning, if it has taken place then it probably would have been on the closed developers@ list.

That’s the sort of “behind the scenes” stuff that I’m interested in, and don’t have any visibility in to.

I am learning how to use reviews.freebsd.org to track ongoing development though!
 
patmaddox said:
It’s the general sense of shared vision I’m interested in, and how it comes about.
Click to expand...
I guess that's a bit like chicken and egg. FreeBSD is like it is. It has some coding guidelines (that also touch the field of software design), and some principles that are generally followed (like e.g. POLA). This attracts especially developers with a pretty similar view about how a well-designed OS should be.

patmaddox said:
Here’s an example: it‘s possible to write FreeBSD kernel modules with Rust. I suspect that at least one developer has brought up the question of whether to permit Rust in src. Such a discussion has not taken place on freebsd-hackers@. I don’t know if such a discussion has taken place at all, but from what I am learning, if it has taken place then it probably would have been on the closed developers@ list.
Click to expand...
I can't say for sure nobody ever wanted that. But I'm pretty sure a majority would be against that right now. Not just because there's a strong sense of traditionalism in all BSDs (and while this might sound negative, it also means don't reinvent the wheel "just because", which often leads to better software design in the long run) – I see lots of devs very open to change and new ideas, all in all it's IMHO a good mix. No, I think there are pretty relevant concerns about such an idea. Still, I'll give my personal view now (cause I can only guess what others would think):

C and C++ are specified by standardization organizations with participants from all over the industry and projects. There are many different implemetations of these languages, all following the standard documents (and where they don't, it's a bug in their implementation). For Rust, only a single implementation exists, and that implementation "defines" the language. Why is this a problem? Well, especially for operating system code, you want potentially long-lived implementations. Code from the 90s could gradually evolve. There's great confidence C (and C++) will still exist tomorrow, and breaking changes in the language will be done carefully and long-term. We can't know at the moment that Rust would have a similar level of "stability". On a side note, building userspace applications using the only existing Rust implementation creates binaries that rely on an old FreeBSD ABI, which doesn't really add to confidence....

A more directly practical concern would be: You would need yet another huge compiler package in base. We already have LLVM (supporting C and C++), it's already the one thing causing most of the build time. You probably don't want that twice. Pure speculation, but if Rust continues to become more widespread, maybe there will be some independent language standard some day, maybe there will be more than just one implementation some day, and just maybe, LLVM will support Rust out of the box some day. If that happens, I'd be all open discussing writing parts of FreeBSD in Rust.
 
patmaddox said:
Here’s an example: it‘s possible to write FreeBSD kernel modules with Rust. I suspect that at least one developer has brought up the question of whether to permit Rust in src. Such a discussion has not taken place on freebsd-hackers@. I don’t know if such a discussion has taken place at all, but from what I am learning, if it has taken place then it probably would have been on the closed developers@ list.

That’s the sort of “behind the scenes” stuff that I’m interested in, and don’t have any visibility in to.
Click to expand...

Nah, such discussions would always happen on a public mailing list, not the closed one. The community is more than whoever holds a commit bit right now.

For example, a discussion on Rust in base would happen not as a "planning" thing. It would happen when somebody brings code that can be discussed. Theorizing about code that should be written is not something that is considered productive.
 
cracauer@ said:
Nah, such discussions would always happen on a public mailing list, not the closed one. The community is more than whoever holds a commit bit right now.
Click to expand...
Agreed this would most likely happen in public. Internal lists are mostly used for organizational announcements and such, plus (rarely) discussing commits that already happened.

cracauer@ said:
For example, a discussion on Rust in base would happen not as a "planning" thing. It would happen when somebody brings code that can be discussed. Theorizing about code that should be written is not something that is considered productive.
Click to expand...
While I don't think someone who wanted to do that would present some "monster patch" integrating Rust into base, they would most likely present some smaller "proof of concept" indeed, like e.g. a rewrite of some module in Rust, and then the discussion could start. I guess it wouldn't go too far right now, but that's just my opinion on that specific topic.
 
Thanks for that insight, zirias@ and cracauer@, on your perspective on Rust in src and especially the development / communication process. I’m glad for this thread, because now I have a better understanding of some stuff I’ve wondered about for a while!
 
zirias@ said:
building userspace applications using the only existing Rust implementation creates binaries that rely on an old FreeBSD ABI, which doesn't really add to confidence....
Click to expand...

If I understand this correctly, this means that any ports that use Rust will build on 13.x but fail to build on current? Or Rust uses an even older ABI than that? If so, how does that work? Is FreeBSD backwards-compatible with older version ABIs? Which I think may be what allows for old versions to be run as jails.
 
patmaddox Just have to reiterate: It's only my assumption others would think similar concerning Rust in base. I don't know. But yes, the takeaway is: such things are very unlikely to be discussed on internal lists. So if you don't find any discussion, it most likely never took place ;)
 
patmaddox said:
If I understand this correctly, this means that any ports that use Rust will build on 13.x but fail to build on current? Or Rust uses an even older ABI than that? If so, how does that work? Is FreeBSD backwards-compatible with older version ABIs? Which I think may be what allows for old versions to be run as jails.
Click to expand...
IIRC, Rust userspace binaries require the FreeBSD-11 kernel(!) ABI. Yes, FreeBSD is backwards-compatible, the GENERIC kernel config lists these:
Code: 
options     COMPAT_FREEBSD4     # Compatible with FreeBSD4
options     COMPAT_FREEBSD5     # Compatible with FreeBSD5
options     COMPAT_FREEBSD6     # Compatible with FreeBSD6
options     COMPAT_FREEBSD7     # Compatible with FreeBSD7
options     COMPAT_FREEBSD9     # Compatible with FreeBSD9
options     COMPAT_FREEBSD10    # Compatible with FreeBSD10
options     COMPAT_FREEBSD11    # Compatible with FreeBSD11
options     COMPAT_FREEBSD12    # Compatible with FreeBSD12

For userspace-ABI (shared libs), there are ports available, but Rust doesn't need these at least.
 
ralphbsz, thanks for your detailed response!

One can think of a build system as consisting of a set of tasks: 1) keeping track of dependencies 2) triggering specific target builds based on what dependencies change, 3) scheduling the affected target builds, 4) invoking a command to do the actual build of a target (as well as discovering dependencies), and 5) processing specification of all this in some build specific language (BSL).

Seems that most of the build system related complaints are usually about its BSL providing insufficient flexibility or being too complicated, which is often made worse by a bad BSL design. There is inherent complexity when interdependent programs in more than one language have to be built + when you have to support more than one processor arch. or OS or compiler or OS, compiler & library versions. On top of that we don't want to explicitly specify everything for each of the source files so we need a way to specify more generic rules. And ideally we want a 100% reproducible builds.

As an individual developer you just want to build a small number of programs or libraries (if not just one) and you don't care about multiple platforms or compilers but if your s/w is part of a larger set, someone has to worry about these things. Just in FreeBSD, /usr/share/mk contains over 12K lines of *.mk files, not counting the 2200+ lines of *.mk in /sys/conf! So there is considerable complexity behind a simple Makefile which can be as small as
PROG=foo
.include <bsd.prog.mk>

Back to bazel. I think it does a pretty good job of all the tasks listed above but falls short on BSL in terms of usability. IMHO of co
one can do considerably better in place of its "skylark" BSL in terms of usability & making things less magic. Search for "bazel tutorial" on youtube and you will find many long videos with titles like "Introduction to Bazel to build ..."!

I suspect one can define a simpler, less magic BSL that can do bulk of the work that bazel does.... But really when the problem is inherently complex, all you can do is push around this complexity.

Interested people may want to check out Gilad Bracha's talk on ShapeRank -- around 5:30 he talks about how one can treat the build process as a reactive stream!
 
patmaddox said:
[...] Here’s an example: it‘s possible to write FreeBSD kernel modules with Rust. I suspect that at least one developer has brought up the question of whether to permit Rust in src. Such a discussion has not taken place on freebsd-hackers@. I don’t know if such a discussion has taken place at all, but from what I am learning, if it has taken place then it probably would have been on the closed developers@ list.
Click to expand...
cracauer@ said:
[...] For example, a discussion on Rust in base would happen not as a "planning" thing. It would happen when somebody brings code that can be discussed. Theorizing about code that should be written is not something that is considered productive.
Click to expand...
It might also because of new userland software (with a relatively minor addition of kernel support code) entering FreeBSD, linked to an essential part of the OS, such as ZFS:
Some quotes:
"... So you may be asking yourself why did we choose userland? Why didn't we just implement this in the kernel ... And in addition to that, we wanted to take advantage of newer richer languages that were available, that aren't available in the kernel and so we chose Rust. ... Why Rust?" (starting at ca. 15:30 min, slide 20 & 21)
 
Looks interesting! BTW, I didn't say anything about the language "rust". Although I didn't use it myself so far, what I read about its design and principles looks pretty sane ;). Therefore, I just gave a few arguments why I think it isn't the time to integrate it in base.

And so, any approach to use it for "out of tree" things is probably a good way to move forward. My hope would be for a definitive language standard document with good governance by some organization plus a few competing implementations of the language (probably, it would first need more than one implementation to motivate that standard document). Which is more likely to happen one day if the language is used a lot ;)
 
This is not a call to be royally trolled. :)
I don't understand why FreeBSD does not offer support for DRM content like Netflix. Netflix is served on a FreeBSD appliance. I'm guessing this is a Netflix proprietary issue?
 
The DRM itself widevine by Google is proprietary and aslong they dont decide to create a binary for FreeBSD you have to use other ways.
 
hitest said:
This is not a call to be royally trolled. :)
I don't understand why FreeBSD does not offer support for DRM content like Netflix. Netflix is served on a FreeBSD appliance. I'm guessing this is a Netflix proprietary issue?
Click to expand...

Netflix and others play back through an industry-standard closed source browser plugin. While Netflix might be FreeBSD-friendly they don't control that plugin.
 
You must log in or register to reply here.
Back
Top