What's wrong with "Recursive Make Considered Harmful"?

[Mjölnir]

From my limited understanding, Peter Miller's thoughts are perfectly right. Does the FreeBSD ports collection use recursive make?

Why am I havíng more trouble than fun with it? Is all the trouble caused by non-perfect Makefile entries (that's human) or is it caused by recursive make? (Why is 'portdumb' misspelled as portmaster? OK, that's provocative). Why is ccache not in the base system? It should be, because the ports are compiled over and over again although they are already installed. (ccache hit ratio went up to >70%). Why can't I use parallel make on a dual core machine? (' make -j 3' fails). Why doesn't make deinstall remove the files that were installed by a port? (e.g. iso8879, have to manually remove the SGML catalogs folder to update).

Is there a secret book about how to jingle with ports that I missed to read?

As it looks for me, it's easier to beadm to a fresh base system, run freebsd-update, and then install the ports I want (with some configuration changed from what failed before), than to update an existing system.
--
=|o)

 

[ShelLuser]

Now, before I spout off keep in mind that although I've became quite a fan of the Ports collection, I most certainly do not consider myself to be a ports collection veteran (not yet anyway

). I've been working with FreeBSD for approximately 5 - 6 months now (give or take) and although I've gone pretty deep at times there's still plenty for me to learn.

Does the FreeBSD ports collection use recursive make?

Not to my knowledge. That is; if you mean this by the definition of calling make directly from within the Makefile itself.

That's not the right way to deal with dependencies, as you can read in the Porters handbook right here. I haven't examined those specific build targets just yet (right now there are other aspects of the port building process I'm concentrating on), but although make is most likely eventually called, it's not directly from a ports own Makefile.

Why am I havíng more trouble than fun with it? Is all the trouble caused by non-perfect Makefile entries (that's human) or is it caused by recursive make?
(Why is 'portdumb' misspelled as 'portmaster'? OK, that's provocative)

Maybe the answer to those questions have the same background: perhaps you're more used to the way Linux deals with compiling source code and source packages and are now reflecting those experiences and ideas onto the ports collection.

Because when I go over your post I see quite some similarities in posts from Linux users who are now "doing FreeBSD" yet treat it as if it was Linux. It's not.

And although there's nothing wrong with that approach perse, it can come to haunt you. And that's usually not much fun.

"When using a BSD(-like) operating system, do as the 'BSD people' do"

Why is 'ccache' not in the base system? It should be, because the ports are compiled over and over again although they are already installed... (ccache hit ratio went up to >70%)

I disagree.

ccache isn't as great as you make it. Sure; it has advantages, but with that come quite some disadvantages as well. It wouldn't be the first time where a build on Linux failed miserably, only to be fixed by removing the cache. And that's even ignoring the issue that a cache for your ports collection can become quite huge (as can be read in that thread).

Another reason why I personally don't believe ccache has any place in the base system is the same reason why tools such as ports-mgmt/portmaster and sys/bsdadminscripts have no place in the base system: the base system is just that. While most people use ports on their FreeBSD environment there are also plenty who don't. So basically it should be as basic, yet complete, as possible.

Quite frankly, if ccache would ever become a standard aspect of the ports collection then it would be the first thing I'd turn off.

Speed is nice, but isn't everything. For my servers I'll take reliability over speed any day of the week.

Why can't I use parallel make on a dual core machine? ('make -j 3' fails)

You mean building 2 ports at the same time? That should be obvious: dependencies. What would happen if one port is building a dependency and another port started doing the same thing? Such a scenario could lead up to quite some horror scenario's.

Offtopic: Its the same reason why a package system (RPM or DPKG) gets locked the very moment you use it, even if you're not doing anything: start dselect or aptitude on Debian, look at the software overview and then try to install a package on another console; it won't work. Even while you're not doing anything.

Perfectly normal behaviour I'd say.

Why doesn't 'make deinstall' remove the files that were installed by a port? (e.g. iso8879, have to manually remove the SGML catalogs folder to update)

I'd imagine for the same reason why many Debian and RedHat packages don't remove certain files after de-installing.

I don't have experience with the port you're addressing here, but the actual reason for that could be anything. From a file which is marked as a configuration file (and therefore left alone) to a directory which contains files installed by another port (so removing it would cause issues).

Even so, it's not very uncommon and something which happens on other platforms as well.

Is there a secret book about how to jingle with ports that I missed to read?

Quite possibly, have you read the Porters handbook? It's a must read for anyone who wants to know more specific details about the ports collection.

As it looks for me, it's easier to beadm to a fresh base system, run freebsd-update, and then install the ports I want (with some config changed from what failed before), than to update an existing system.

Then I wonder if you ever read up on this procedure in the first place?

Because updating an existing ports collection takes only 3, optionally 4, simple tasks:

1. Update the ports collection using # portsnap fetch update.
2. Read /usr/ports/UPDATING for specific instructions.
3. (Optional) Follow the specific instructions in UPDATING. *
4. Update the installed ports using # portmaster -a.

I call that easy.

* You only need to follow the instructions if they apply to a port you have installed.

 

[SirDice]

Why doesn't make deinstall remove the files that were installed by a port? (e.g. iso8879, have to manually remove the SGML catalogs folder to update)

It should actually. So perhaps you ran into a bug. Ports are maintained by volunteers and people make mistakes.

If I look at the port's pkg-plist it all seems to be in order. But if the files are modified locally they won't be removed (to prevent a port or package from removing user edited configurations).

 

[wblock@]

From my limited understanding, Peter Miller's thoughts are perfectly right.
Does the FreeBSD ports collection use recursive make?

From a quick skim of that article, yes.

Why am I havíng more trouble than fun with it? Is all the trouble caused by non-perfect Makefile entries (that's human) or is it caused by recursive make?

Without knowing the specific problem, I'd say that ongoing problems with ports are usually due to a bug in expectations. Sometimes that's due to a misconception, like being familiar with other package systems that are superficially similar. Sometimes it's from following bad advice, like setting custom CFLAGS or BATCH or other "go-fast" options that are misunderstood.

Recursive make(1) in the ports tree is the last thing I would pick to be a problem.

Why is 'ccache' not in the base system? It should be, because the ports are compiled over and over again although they are already installed... (ccache hit ratio went up to >70%)

Sometimes portmaster is dumb about recompiling ports that have already been recompiled. That has generally been fixed in later versions. Note that I use portmaster only as an upgrade tool, not for installing new ports. You don't say what version of FreeBSD or portmaster you use.

I use ccache, but not for ports because the hit rate is just too low to be worthwhile.

Why can't I use parallel make on a dual core machine? ('make -j 3' fails)

The ports system already builds ports in parallel. The ones that are safe to build in parallel, anyway. This is transparent to the user. See Building Ports in Parallel. This is an expectation bug.

Why doesn't 'make deinstall' remove the files that were installed by a port? (e.g. iso8879, have to manually remove the SGML catalogs folder to update)

More specifics on that would be appreciated. It could be a bug with that port, but I suspect it's due to the catalog files having been modified by other ports. In general, a port does remove files it installed, but not if those files have been user-modified.

Is there a secret book about how to jingle with ports that I missed to read?

Well, there's the Porter's Handbook mentioned above. It is aimed at port writers, not port users.

Upgrading FreeBSD Ports is a short article showing my recommended way.

As it looks for me, it's easier to beadm to a fresh base system, run freebsd-update, and then install the ports I want (with some config changed from what failed before), than to update an existing system.

Definitely a problem, then. The next time you have problems with an upgrade, start a thread with specifics and we can figure out the problems.

 

[Chris_H]

Greetings.

Why doesn't 'make deinstall' remove the files that were installed by a port? (e.g. iso8879, have to manually remove the SGML catalogs folder to update)

Because that's the UNIX way -- UNIX is comprised of many tools that do one thing, and do them very well.

In a broader scope; you can get it to de-install the port recursively. But, you need to issue the correct command. As memory serves: make deinstall recursive. But beware, the results might not be what you intended. Which is INMNSHO, why it isn't the default course of action. Is it becoming clearer?

Best wishes.

--chris