Opinion: FreeBSD ports is broken by design

[Mjölnir]

[ Split off from http://forums.freebsd.org/showthread.php?t=40818 -- Mod. ]

I had the same issue when I tried to install x11/kde4-runtime. Installing misc/help2man directly succeeded. I.e. the success of the very same task is pure luck.

Unfortunately, I take your answer:

Greetings,
Just choose i here. This is a harmless error. It means you have not created the directory for backup packages, or the previous version(s) were not where this version expects them. Won't interfere with the new version. But won't permit the ability to revert to previous version, using the portmaster backup package. That's all.
--chris

as a confirmation of my opinion: "FreeBSD ports is broken by design".

• if portmaster needs a folder for backups, it should create it
• If the package is installed the first time, there is no old version

If there's no recursive make, as someone said, then why do we need tools like (so-called) portmaster to correct what make should be able to do on its own? Why do people constantly have trouble compiling ports? Did you ever try to update portmaster with itself? There are numerous bugs like this.

 

[DutchDaemon]

Neither ports, nor portmaster, ever give me problems, occasional (and usually quickly fixed) hiccups with a port's Makefile aside. The broken by design comment is ludicrous.

 

[Mjölnir]

Well, if you call the random success of a task a hiccup, everything is fine

But I'm getting suspicious in such a case. Can you clearly confirm that the ports Makefiles are not recursive?

I agree that it's always possible to run into human-made bugs in the port's configuration, and I do appreciate the effort of all the volunteers who help to make FreeBSD a valuable system.

 

[DutchDaemon]

A Makefile is recursive in so far as it will check whether dependencies are installed and up-to-date, and if not, it will switch to the task of performing and updating them. In that sense, portmaster is not needed to install a single port. Once it comes to upgrading several ports in conjunction, portmaster takes the hassle out of figuring out the order, to prevent double work or clobbering/race conditions. With all of its advanced options and flags for things ranging from automatic backups of previous versions, keeping old libraries around, force-reinstalling up- or downstream dependencies, it certainly takes all of the headaches out of figuring out the proper make targets in the proper order. But: you should upgrade ports at least once a week, and not let things deteriorate, and certainly not update selectively, unless there is an overriding reason not to keep up. But that's where you will start encountering problems, which will grow deeper and worse over time. The ports tree is supposed to be used in conjunction, and to be kept up-to-date and fresh. All installed ports need to be of the most recent version. Then you will have no problems, especially if you give new versions a day or two to incorporate last-minute bug fixes before actually installing them. I don't, but I'm more than capable of backing out of changes that go awry, thanks to the advanced nature of portmaster. But to be honest, with the 1000+ ports I have installed on various systems, with a daily update routine, that happens maybe two or three times a year.

 

[Mjölnir]

It's just that some other not-so-advanced member said the ports Makefiles are not recursive, and honourable Mr. Peter Miller clearly points out that to call make recursively is not the right way to use it. Thus I'd say my provocative comment is not ludicrous.

In fact, as you say yourself, we need

portmaster

as a wrapper to fix the deliberately broken dependency graphs, and it is surprisingly good in doing that. I guess the reason to do so was limited RAM of machines in the past, and maybe easy of managing 1000's of ports, too.

But since using ports instead of pre-packaged packages is such a common thing, I'd be happy to see a solution that is theoretically proven sound. The current solution is not.

 

[DutchDaemon]

The ports system is not broken by design. It is complex, yes.

 

[Mjölnir]

Amen, and the world is flat, because I want it to be.

Sorry, but I think your opinion is based on the fact that the portmaster/make combination does work somehow right now, and it would be a big thing to change it.

Nevertheless, you are saying that it uses recursive make, that we get into trouble if we do not keep up with current versions, and that portmaster does what make should be able to do (finding the order of dependencies, i.e. to construct the tree(s)).

I'd like to understand why the current solution was chosen, and I make the assertion: if the ports collection would use non-recursive make, we'd need no portmaster; all portmaster does could be done from included Makefiles à la bsd.port.mk.

 

[DutchDaemon]

It is time to take your discussion to the freebsd-ports mailing list (join). Nobody here will address or fix your concerns.

 

[SirDice]

As @DutchDaemon noted, you do not need to use portmaster. You can do everything by hand. Seriously not recommended though. Even after using the ports system for the past 15 years I still get a headache thinking about doing it by hand.

But I think there's some misconception about the term "recursive". I haven't dug into it too much but I don't think the ports system actually runs [cmd=]make[/cmd] by itself.

 

[Mjölnir]

As @DutchDaemon noted, you do not need to use portmaster. You can do everything by hand.

That's not true: you'll run into deep deep trouble, because sooner or later you'll encounter broken dependencies (because the Makefiles are recursive).On a dual-core machine, try make -j 3 in a ports subdirectory. Weird things will happen... not for all ports, but for many. Try, say, x11/kde4. Only for small ports, everything is fine.

Seriously not recommended though. Even after using the ports system for the past 15 years I still get a headache thinking about doing it by hand.

Right! This is exactly what I'm saying: you need portmaster to fix the deliberately broken dependency trees. Among other tasks (like keeping backups), that's basically what it does. Likewise does portinstall, but I'm not talking about that because it relies on libraries and thus makes life even harder. It should be vanished, first from the book, later from the ports.

But I think there's some misconception about the term "recursive". I haven't dug into it too much but I don't think the ports system actually runs [cmd=]make[/cmd] by itself.

That's a matter of definition? The ports system is a collection of data to drive the toolchain to build the software. See above, when you want to install KDE4 without a wrapper you do
# cd /usr/ports/x11/kde4
make package
and then you're doing a recursive make, and currently it does not what you want. It can not, because, I'm sorry I have to repeat this: the ports system is broken by design because it uses recursive make. If you do the same task with portmaster[/fike], you have a good chance to succeed if you're able to fix some 'hickups'. Some of these may be due to human-made mistakes in the configuration files; we'll always have such, ok. But I suspect many caused by the incomplete design.
I'll assign myself to the mailing list and see what the gurus tell me about their decisions to do it the way it is.

 

[kpa]

You have repeated this "recursive make is the problem" mantra for quite a while now but still you haven't actually provided any concrete details why recursive make breaks the dependency tracking. Start delivering finally or take back your claims.

If you're going to the mailing lists with this "it's broken, please fix it" attitude and you can not provide a minimal enough test case to showcase the essence of the problem you're going to get ignored, and quite rightly so.

 

[wblock@]

The "recursive make" article seemed to be about the standard way make(1) is used in projects. A top-level Makefile, and Makefiles in subdirectories. Build the project, and it recursively builds the subdirectories. Yes, the ports system is like this. There is a Makefile in /usr/ports/, another in /usr/ports/sysutils, another in each of the port directories in there.

But the article was about large project management, and did not, in my opinion, directly relate to FreeBSD ports. Or rather, not to a typical ports user. Those who make -C /usr/ports may find valid complaints in that article.

And to address a question:

Why do people constantly have trouble compiling ports?

That's an assertion which is not supported by evidence.

Did you ever try to update portmaster with itself?

Yes, and successfully. Probably at least half a dozen times in the last couple of years. If you're fishing for an example of brokenness, this is not it.

Look, we appreciate that you may have had a hard time with ports. As I've said before, sometimes this is due to a bug in expectations, where it works correctly but does not produce what the user expects. If you want help, start a new thread and we'll try to help you. Trolling with this "deliberately broken" line is not productive for any of us.

 

[ShelLuser]

I agree with @kpa that so far we haven't really seen any solid example.

Though personally I'd be much more interested in your ideas on fixing that which you consider to be broken. It's one thing to call something broken, anyone can do that. But to come up with a well set up alternative, that's the harder part.

 

[Mjölnir]

You have repeated this "recursive make is the problem" mantra for quite a while now but still you haven't actually provided any concrete details why recursive make breaks the dependency tracking. Start delivering finally or take back your claims.

You can read about this in the excellent article Recursive Make Considered Harmful

If you're going to the mailing lists with this "it's broken, please fix it" attitude and you can not provide a minimal enough test case to showcase the essence of the problem you're going to get ignored, and quite rightly so.

I do not have this attitude. I want to know the reasons why the ports building process is like it is, and I do not have a better solution at hand because I do not have enough background knowledge.
Testcase: start a parallel build of a big ports collection directly, like
# cd /usr/ports/x11/kde4
# make -j 3
Confirm all the default options. If the dependencies were ok, you should succeed.

 

[Mjölnir]

Though personally I'd be much more interested in your ideas on fixing that which you consider to be broken. Its one thing to call something broken, anyone can do that. But to come up with a well set up alternative, that's the harder part.

IMHO you don't have to have an alternative right now when you suspect s/th to be wrong.
If s/o came up with a well thought out solution to such a complex task within a day, I'd be even more sceptical than with the current portmaster/make combination

I'd like to make clear that my current status is to learn to use the available tools, e.g. I'm building my next system in a jail, and that I don't want my questions to be misinterpreted as assaults. That's not my intention. If someone could tell me why it's not possible to use non-recursive make to build the ports, and thus there is no sound solution but portmaster is the best approximation, I'd be happy.

 

[kpa]

Your test case isn't valid because you're forcefully overriding the number of jobs (-j 3) for all dependent ports. The proper way is to let the ports(7) system decide if a port is safe to build with multiple simultanious jobs and set MAKE_JOBS_NUMBER in make.conf(5) if you need to override the default. Take a look at /usr/ports/Mk/bsd.port.mk for details.

Edit: The default was changed very recently to assume that ports are safe to build with multiple jobs and there's a per port knob MAKE_JOBS_UNSAFE that can be set in the port Makefile to say that this port is not safe to build with multiple jobs. There is no need to use the -j flag of make(1) directly, the ports system will automatically use the optimal number of jobs for each port.

 

[ShelLuser]

You can read about this in the excellent article Recursive Make Considered Harmful

That is an interesting read in my opinion.

However, I think there is one flaw in your reasoning; but strictly speaking within the context of the Ports collection. Your suggested solution (Page 8) basically consists of "Use one Makefile" and throughout the paper it also focusses on the situation as a single project. Maybe a very complex and large project, but still a single project.

Yet that doesn't apply, not in the same extends anyway, to the Ports collection I think. Because, as the name somewhat implicates, its not merely a single project but more a whole collection of projects. Individual projects at that, which may indeed share a certain relationship (dependency). However, they leave it up to the upper structure to actually satisfy those dependencies. In a way it is fragmented by design.

Another issue I wonder about, but I have to admit that my knowledge of make isn't too extensive, is if this solution wouldn't result in a management disaster for something like the ports collection. For example I could imagine that if a certain committer make a small mistake it could potentially break the entire ports collection (when using one single Makefile it most certainly would I'd say), instead of simply breaking one single port in the collection.

Of course people aim not to make mistakes in the first place, but accidents do happen from time to time.

So within that perspective I'd rather have a system which may not be as optimal as it theoretically could be than a system which could be technically superior yet leaves much to desire where management is concerned.

 

[Mjölnir]

Your test case isn't valid because you're forcefully overriding the number of jobs (-j 3) for all dependent ports. The proper way is to let the ports(7) system decide if a port is safe to build with multiple simultanious jobs and set MAKE_JOBS_NUMBER in make.conf(5) if you need to override the default. Take a look at /usr/ports/Mk/bsd.port.mk for details.

Thanks for the hint! I did not know that, and applied it immediately in my build jail.

Anyhow, do it with your correction and it will surely fail directly, but with the portmaster wrapper, it will succeed (except a few hiccups due to human mistakes, that's normal).

 

[wblock@]

Testcase: start a parallel build of a big ports collection directly, like
# cd /usr/ports/x11/kde4
# make -j 3

No, that will not work. Not all ports are safe to build in parallel. Ports that are safe to build in parallel are handled by the ports system. Do not use -j with ports.

This is a good example of that "expectations bug" I was talking about.

 

[Mjölnir]

That is an interesting read in my opinion.

However, I think there is one flaw in your reasoning; but strictly speaking within the context of the Ports collection. Your suggested solution (Page 8) basically consists of "Use one Makefile" and throughout the paper it also focusses on the situation as a single project. Maybe a very complex and large project, but still a single project.

Yet that doesn't apply, not in the same extends anyway, to the Ports collection I think. Because, as the name somewhat implicates, its not merely a single project but more a whole collection of projects.

Thus the thought arises that it could be possible to construct the Makefile for each collection I want to build on demand (based on the current status of the system), and meanwhile lock the building of others. I guess that's what portmaster does somehow, but it's doing it outside of make's awareness. Maybe that's exactly the right thing to do; maybe not. I already stated that portmaster works surprisingly well; nevertheless I suspect it's working on a non-sound foundation and that it applies many quirks to fix what is inheritably broken. Disclaimer: this is an assumption, not an assault.

Individual projects at that, which may indeed share a certain relationship (dependency). However, they leave it up to the upper structure to actually satisfy those dependencies. In a way it is fragmented by design.

This fragmentation is dangerous, because it can break dependencies. The traditional tool for this task is make, or the advanced CMake, and they rely on proper input. I.e., non-fragmented.

Another issue I wonder about, but I have to admit that my knowledge of make isn't too extensive, is if this solution wouldn't result in a management disaster for something like the ports collection. For example I could imagine that if a certain committer make a small mistake it could potentially break the entire ports collection (when using one single Makefile it most certainly would I'd say), instead of simply breaking one single port in the collection.

If the Makefile gets constructed on demand, it would only break builds of dependent ports, like now.

So within that perspective I'd rather have a system which may not be as optimal as it theoretically could be than a system which could be technically superior yet leaves much to desire where management is concerned.

I do not say there's an easy solution to a very complex problem Taking all the options, there are billions of billions of possible different FreeBSD systems...

 

[Mjölnir]

No, that will not work. Not all ports are safe to build in parallel. Ports that are safe to build in parallel are handled by the ports system. Do not use -j with ports.

This is a good example of that "expectations bug" I was talking about.

And, BTW, it's an example that proves we're using make in a wrong way... If the Makefiles were ok, parallel build would succeed because make can handle parallel builds for decades now.

 

[wblock@]

And, BTW, it's an example that proves we're using make in a wrong way... If the Makefiles were ok, parallel build would succeed because make can handle parallel builds for decades now.

Parallel port builds are now the default, and have actually been in use for quite a while. "Does not do it the way I think it should" does not equal "broken". But you know this.

 

[Mjölnir]

I did not see any single argument from you why it's ok to build a house on an unstable ground, or why the ground I consider unsound, is sound.

If (the Makefiles were ok) then parallel make w/o a wrapper would succeed
That's simple logic => the Makefiles are not ok

If (the Makefiles were ok) then it would not build ports that are already installed and do not need to be updated over and over again
But it does => The Makefiles are not ok

And we need portmaster to fix that. Still, as long you do not give it the right options, it builds too much. All I do is asking why and I am sceptical that this is the right solution.

 

[DutchDaemon]

One more time.

It is time to take your discussion to the freebsd-ports mailing list (join).

Closed.