Heads-Up: Deprecation of mergemaster

Zirias

Daemon

Reaction score: 1,318
Messages: 2,344

So far, I was using mergemaster(8) when upgrading from source; it worked fine, and it's also described in the handbook.

Now I learned it's deprecated and will be removed soon. It might be dangerous after 13.0-RELEASE, because it relies on SVN source tags that don't exist any more with git!

For details, see PR 252417.

The recommended tool is etcupdate(8).

I immediately ran into a problem with that, trying to "bootstrap" it with etcupdate extract. The reason was: it doesn't play nice with /usr/obj mounted read-only. But this is probably not uncommon if you build FreeBSD on one machine and install it on several others that mount /usr/src and /usr/obj via NFS.

The solution is: for each build you do on your building machine, also build a tarball for etcupdate(8) like this:
etcupdate build -B /usr/obj/etcupdate.tar.bz2.

This tarball can then be used on any other machine, e.g. for "bootstrapping":
etcupdate extract -t /usr/obj/etcupdate.tar.bz2.
 

jb_fvwm2

Daemon

Reaction score: 204
Messages: 1,795

Will someone do a writeup with screenshots so a novice user of etcupdate won't, for example, be locked out from an unwanted change in the password files during an installworld?
 

PMc

Daemon

Reaction score: 576
Messages: 1,196

So far, I was using mergemaster(8) when upgrading from source; it worked fine, and it's also described in the handbook.

Now I learned it's deprecated and will be removed soon. It might be dangerous after 13.0-RELEASE, because it relies on SVN source tags that don't exist any more with git!
That is strange. Up to Rel. 14 clearly requires to use mergemaster:

Code:
$ git branch --show-current
main
$ cat UPDATING
[...]
        To rebuild everything and install it on the current system.
        -----------------------------------------------------------
        # Note: sometimes if you are running current you gotta do more than
        # is listed here if you are upgrading from a really old current.

        <make sure you have good level 0 dumps>
        make buildworld
        make buildkernel KERNCONF=YOUR_KERNEL_HERE
        make installkernel KERNCONF=YOUR_KERNEL_HERE
                                                        [1]
        <reboot in single user>                         [3]
        mergemaster -Fp                                 [5]
        make installworld
        mergemaster -Fi                                 [4]
        make delete-old                                 [6]
        <reboot>
P.S: Got it: there is a bug-report and it is not closed yet - the docs also will (have to) change at some point in the future...
Probably no need to hurry, as somebody knowledgable of the new tool will hopefully provide drop-in replacements for the procedures in UPDTING.
 
OP
Zirias

Zirias

Daemon

Reaction score: 1,318
Messages: 2,344

That is strange. Rel. 14 clearly requires to use mergemaster:
Probably worth a PR then. See PR in my first posting, and also mergemaster's manpage, it does rely on these source tags, so bad things will happen when using it after the first upgrade to 13 or newer.
 

dvl@

Aspiring Daemon
Developer

Reaction score: 89
Messages: 597

The reason was: it doesn't play nice with /usr/obj mounted read-only. But this is probably not uncommon if you build FreeBSD on one machine and install it on several others that mount /usr/src and /usr/obj via NFS.
This is common and I've been using this feature for over 20 years. It is not an edge-case by any means.
 
OP
Zirias

Zirias

Daemon

Reaction score: 1,318
Messages: 2,344

This is common and I've been using this feature for over 20 years. It is not an edge-case by any means.
Well, maybe etcupdate should better support this scenario then? ;) Still, the "workaround" I described in my posting is fine for me!
 

Alain De Vos

Daemon

Reaction score: 347
Messages: 1,290

You can run also "etcupdate -p" and i have totally no idea when i should put the "-p" flag or not.
Code:
-p                 Enable “pre-world” mode.  Only merge changes to files
                        that are necessary to successfully run ‘make
                        installworld’ or ‘make installkernel’.  When this flag
                        is enabled, the existing “current” and “previous”
                        trees are left alone.  Instead, a temporary tree is
                        populated with the necessary files.  This temporary
                        tree is compared against the “current” tree.  This
                        allows a normal update to be run after ‘make
                        installworld’ has completed.  Any conflicts generated
                        during a “pre-world” update should be resolved by a
                        “pre-world” resolve.
This is clear. But what does it mean in practice?. When do you use it or not ?
 
OP
Zirias

Zirias

Daemon

Reaction score: 1,318
Messages: 2,344

This is clear. But what does it mean in practice?. When do you use it or not ?
For me, this is just guesswork: There might be situations where an update to something in /etc is required to successfully execute any install target. At least, that's what I interpret there. I've been following FreeBSD since 11 (back when it was -CURRENT) now and never had such a situation. 🤷‍♂️

edit: the handbook tells how it's meant to be used.
 

dvl@

Aspiring Daemon
Developer

Reaction score: 89
Messages: 597

Zirias we are trying to duplicate your steps here, on a 12.2 host, upgrading to 13.0

There are 11 steps listed in Updating FreeBSD from Source

  • After which step do you do etcupdate extract -t /usr/obj/etcupdate.tar.bz2?
  • What steps, if any, does this replace?
 
OP
Zirias

Zirias

Daemon

Reaction score: 1,318
Messages: 2,344

dvl@ I can't directly answer to your questions as I was just trying to "bootstrap" etcupdate, because I used mergemaster before. The situation was having releng/13.0 in /usr/src and a build of it in /usr/obj, both mounted read-only via NFS. This build was already installed to the target machine (with installkernel and installworld) and running, /etc was updated one last time with mergemaster. Then, executing just etcupdate extract failed, IIRC because this tried to write something to /usr/obj.
 
OP
Zirias

Zirias

Daemon

Reaction score: 1,318
Messages: 2,344

To be clear, you are doing the bootstrap on 12.2 src and obj?
Nope, that was an error in the first version of the post, I quickly fixed it. I tried it on 13.0. Updating /etc from 12.2 to 13.0 was still done with mergemaster before.

I'm not entirely sure the bootstrapping problem was etcupdate's fault, I didn't further analyze it. Since 13.0, I have similar problems with r/o /usr/obj from time to time, with make installworld stopping with an error trying to write something to /usr/obj, IIRC in the context of (lib)magic… Repeating make buildworld on the building host fixes it. :rolleyes:
 
OP
Zirias

Zirias

Daemon

Reaction score: 1,318
Messages: 2,344

This is ambiguous to me and I do not know how to interpret that.
Uhm. Steps to reproduce (or, maybe not, cause I didn't try to reproduce it myself):
  • Checkout releng/13.0 on a build machine running 12.2
  • Build it ( make buildworld; make buildkernel)
  • Mount /usr/src and /usr/obj via NFS r/o on a target machine, also running 12.2
  • Install 13.0 there ( make installkernel; shutdown -r now; make installworld)
  • Update /etc on the target machine with mergemaster
  • Try to bootstrap etcupdate there with etcupdate extract
 

scottro

Daemon

Reaction score: 797
Messages: 1,924

Does /etc/update give the same presentation of a file? For example, mergemaster shows original on left and new one on right and you hit l or r to get it to work. (And I just tried to find out where I learned that, doesn't seem to be in the man page. Oh well, probably web searched it somewhere.

While, as Zirias says, screen shots might be an odd way to do it, some basic examples of its usage ought to be somewhere. These days, I only use freebsd-update, so it's not been an issue for me, but it would be nice to have a more detailed explanation
 

dvl@

Aspiring Daemon
Developer

Reaction score: 89
Messages: 597

BTW, I think you're doing it wrong based on this extract from the PR

"If you have done several worlds with 'mergemaster' instead, then etcupdate's database in /var/db/etcupdate is stale. You can overwrite the database by doing 'etcupdate extract' _before_ you do an 'svn up' or 'git pull'. To check to see what local diffs etcupdate thinks you have, run 'etcupdate diff' (it outputs a patch)."

To me, that says the bootstrap must be done against the src you are upgrading, not the src of what you are upgrading to. In my case, that would be the 12.2 src.

We're going to test this now.
 

dvl@

Aspiring Daemon
Developer

Reaction score: 89
Messages: 597

Does /etc/update give the same presentation of a file? For example, mergemaster shows original on left and new one on right and you hit l or r to get it to work. (And I just tried to find out where I learned that, doesn't seem to be in the man page. Oh well, probably web searched it somewhere.
No, you don't go through that process at all, from what I've seen, but we haven't gotten to that step yet. From the PR:

"If the changes conflict, it saves a version with conflict markers for the user to manually resolve. At the end of 'etcupdate', it lists any conflicts you manually need to resolve. For those, you run 'etcupdate resolve' which gives you options to either keep your version, install the new version only, or invoke an editor to resolve the changes."
 
OP
Zirias

Zirias

Daemon

Reaction score: 1,318
Messages: 2,344

No, you don't go through that process at all, from what I've seen, but we haven't gotten to that step yet. From the PR:

"If the changes conflict, it saves a version with conflict markers for the user to manually resolve. At the end of 'etcupdate', it lists any conflicts you manually need to resolve. For those, you run 'etcupdate resolve' which gives you options to either keep your version, install the new version only, or invoke an editor to resolve the changes."
I'm not sure everyone will like that immediately, but for me, having to resolve conflicts "on the go" was always a hassle. I prefer this new approach with conflict markers. 👍
 
  • Like
Reactions: PMc

Terry_Kennedy

Aspiring Daemon

Reaction score: 329
Messages: 961

This may be an unpopular opinion, but I really wish that "STABLE" meant that as well as the ABI / KBI not changing, the tools needed to maintain a system from source also did not change during the life of a major release. I realize that this is an issue that affects comparatively few people - most users either track binaries with freebsd-update(8) or just install a release and don't change anything afterwards, but for those people it does affect, the confusion and effort seems way out of line with the hoped-for benefit.

For example, the PR states:
PR 252417 said:
mergemaster(8) has been deprecated for several releases and its replacement--etcupdate(8)--has been present in production systems since at least 9.x/10.x.
So, it missed being officially introduced / changed / documented in 11.x, 12.x, and presumably 13.x but now there's a push to (potentially) introduce a major change to 3 production releases? I just checked the 11.4, 12.2 and 13.0 release notes and it isn't mentioned in any of them.

There has always been a fair amount of this going on in FreeBSD. As a recent example, the ports tree changed from svn to git (also not mentioned in the release notes for 11.4, 12.2 or 13.0) with the last entry in /usr/ports/UPDATING essentially saying "Moved, left no forwarding address" with users having to trawl through the forum and mailing lists to find out what the replacement was, only to find the answer was "Nope, not ready yet. Fixed soon, hopefully.", along with a variety of different unofficial solutions. I don't think that the conversion procedure has been officially described anywhere - I just checked the FreeBSD handbook (PDF edition of 1/23/2021, from the official freebsd.org link). That is far from the only case.

This isn't an argument against change - it is an argument against unannounced change when work has been going on "behind the scenes" but that doesn't get communicated to users until it is presented as a fait accompli. If the replacement of mergemaster(8) wirh etc-update(8) has been "in the works" since FreeBSD 10.4 (taking the PR's "since 9.x/10/x" at its latest possible date), it has been over 3.5 years since this decision was made. If one assumes it has been since FreeBSD 10.0, that has been over 7 years. And in the same version of the handbook I mentioned above, there are zero mentions of etcupdate(8). Looking at /usr/src/UPDATING (svn r369959) there are 4 mentions of etcupdate(8), all of which are of the form "mergemaster or etcupdate".

It is a reasonable thing to ask me "If you're so concerned about it, why don't you write up an article and post it to the wiki?" A reasonable question, except that I usually find out about this the same time everybody else it affects does - when the sky falls. Sure, I could write something up, proofread it, ask for some reviews and when everyone is happy, post it to the wiki. But by then most people who care will have likely either figured it out on their own or given up. Not to mention that there would be no pointers to the wiki, which has a bunch of outdated and incomplete articles and thus isn't a particularly trustworthy source of information. As an example (yes, I know my example is ports-related, I'm making a point here), look at the pages for Python3Default (last updated 2020-10-10) and Python2EOL (last updated 2021-04-25), neither of which has any information useful to end users. The first article is downright misleading for developers (there is lots of out-of-date information there) and the sole nugget of useful information for developers in the second article can be summed up as "Please use USES=python:3.6+ from now on".
 

bakul

Member

Reaction score: 28
Messages: 64

It is a reasonable thing to ask me "If you're so concerned about it, why don't you write up an article and post it to the wiki?" A reasonable question, except that I usually find out about this the same time everybody else it affects does - when the sky falls.
If the core group (or some review group) does a weekly/biweekly/monthly review of commits for the purpose of preparing release notes, that can also provide an early warning system. I think by the time release notes come out, it is too late; changes have been baked in. And warning what features might get deprecated in the next release may be too early as people keep using what works for them until all of a sudden it doesn’t.
 

Terry_Kennedy

Aspiring Daemon

Reaction score: 329
Messages: 961

If the core group (or some review group) does a weekly/biweekly/monthly review of commits for the purpose of preparing release notes, that can also provide an early warning system. I think by the time release notes come out, it is too late; changes have been baked in. And warning what features might get deprecated in the next release may be too early as people keep using what works for them until all of a sudden it doesn’t.
I'm pretty sure that Bugzilla has a "Relnotes: Y/N" flag. If it wasn't Bugzilla, it was Phabricator.

If there's a heads-up in the release notes for a release where the old way still works, that'd be great. I'd be happy with it showing up in the release notes for the next major version where it takes effect. People who don't read the release notes before upgrading to a new major release are likely to encounter other surprises along the way. Not that I said "major release" - I don't think low-urgency changes like this are appropriate to happen in a point release - in this case, if it waited 3.5 years it can wait a while longer.

Committing a change to mergemaster(8) (in this case) saying "This utility will be deprecated in FreeBSD 14.0 - refer to https://www.freebsd.org/someURL for more information" with a pause (similar to the way portupgrade(1) warns about an unsupported FreeBSD version) would be great, too - once the timeframe and target release are known.
 
OP
Zirias

Zirias

Daemon

Reaction score: 1,318
Messages: 2,344

Not that I said "major release" - I don't think low-urgency changes like this are appropriate to happen in a point release
There's no way to keep mergemaster(8) working, it relies on source tags GIT just won't generate.

Also note PR 252417 starts like this:
mergemaster(8) has been deprecated for several releases and its replacement--etcupdate(8)--has been present in production systems since at least 9.x/10.x.

So, there probably was a deprecation notice. Maybe many didn't read it. Also, the handbook used etcupdate(8) for a while, but there's been this ongoing problem of finding an old and outdated version of the handbook with Google…
 
Top