Heads-Up: Deprecation of mergemaster

Terry_Kennedy

Aspiring Daemon

Reaction score: 329
Messages: 961

There's no way to keep mergemaster(8) working, it relies on source tags GIT just won't generate.
I believe there was agreement to continue to provide svn updates through the life of 12.x.
Also note PR 252417 starts like this:
Exactly. Some people have known about this for a minimum of 3.5 years (assuming the PR meant 10.4). Expecting users to read every bug report to find which happen to mention changes that break existing procedures isn't reasonable. This reminds me of a Douglas Adams quote.
So, there probably was a deprecation notice. Maybe many didn't read it. Also, the handbook used etcupdate(8) for a while, but there's been this ongoing problem of finding an old and outdated version of the handbook with Google…
I specifically checked the official master FreeBSD copy of the Handbook from: https://download.freebsd.org/ftp/doc/en/books/handbook/book.pdf (no hotlink title so the URL is visible in the post). Word search in Acrobat: etcupdate, 0 matches; mergemaster, 25 matches.

I always do read the release notes and errata, and I don't remember seeing it. That doesn't mean there never was one, of course. Even if there was, this is the sort of thing that should remain in the release notes until the change happens (and then the release notes should say that as of the version the release notes pertain to, things have changed).
 
OP
Zirias

Zirias

Daemon

Reaction score: 1,318
Messages: 2,344

I believe there was agreement to continue to provide svn updates through the life of 12.x.
Yes, 12.x is not affected. Branches for 13.x exist only in GIT though.

I specifically checked the official master FreeBSD copy of the Handbook from: https://download.freebsd.org/ftp/doc/en/books/handbook/book.pdf (no hotlink title so the URL is visible in the post). Word search in Acrobat: etcupdate, 0 matches; mergemaster, 25 matches.
This is the old, outdated content indeed. So you probably discovered a bug here, maybe with building the PDF version…

The HTML version (which is linked in the top menu as well as on top of https://docs.freebsd.org/en/ ) is correct.

I always do read the release notes and errata, and I don't remember seeing it. That doesn't mean there never was one, of course. Even if there was, this is the sort of thing that should remain in the release notes until the change happens (and then the release notes should say that as of the version the release notes pertain to, things have changed).
You won't repeat a deprecation note in every release. I think doing it once and updating the handbook really is good enough, but the main problem here seems to be too many editions of the handbook with outdated content floating around. The PDF you found really is a bummer. And yes, I was surprised to find mergemaster(8) deprecated as well.
 

Duffyx

Active Member

Reaction score: 43
Messages: 119

You can run also "etcupdate -p" and i have totally no idea when i should put the "-p" flag or not.
<snip>
This is clear. But what does it mean in practice?. When do you use it or not ?

As I rewrote that part of the handbook to use etcupdate over mergemaster I can elaborate a bit more on this. One recent change where etcupdate -p was needed was the addition of the system ntpd user. If you would update from source with that commit including, the ntpd user would not be present yet on the running system. This would cause error messages when running installworld because it references that user in the install() command to install files as that user, installworld would fail saying something like "ntpd user not existing, or UID 123 not existing". Honestly, you can safely run etcupdate -p every time you update from source, if it doesn't need to do anything it wont do anything.

That is strange. Up to Rel. 14 clearly requires to use mergemaster:
<snip>
P.S: Got it: there is a bug-report and it is not closed yet - the docs also will (have to) change at some point in the future...
Probably no need to hurry, as somebody knowledgable of the new tool will hopefully provide drop-in replacements for the procedures in UPDTING.

Changes to /usr/src/UPDATING have been under review for way to long causing confusing.
I have added a reminder so that both the handbook and the instructions shipped with the source are the same.
 

Terry_Kennedy

Aspiring Daemon

Reaction score: 329
Messages: 961

Changes to /usr/src/UPDATING have been under review for way to long causing confusing.
I have added a reminder so that both the handbook and the instructions shipped with the source are the same.
I'm not singling you out here, honest. But the Phabricator review shows the exact problem I'm talking about. If you are coming to FreeBSD via a new install, your instructions are perfect. However, for people who have been using / updating FreeBSD for ages, it is incomplete information in the wrong place. In addition to what you have way at the bottom in "COMMON ITEMS", there should be something like this with the current date, near the top:
Code:
20210610:
        The default method of updating system configuration files has changed from
        mergemaster to etcupdate. If you have been using mergemaster, please perform
        the following steps to convert to using etcupdate:
          Before updating your source tree, buildworld, installworld, etc:
            1) ...
            2) ...
            3) ...
         After running buildworld, installworld, etc:
            1) ...
            2) ...
        After you have converted to etcupdate, only the second set of steps will be
        needed.
While someone is in there, converting from svn to git should also be covered (preferably recommending the lightweight gitup client, similar to the way svnlite is used instead of full svn).

Again, I'd like to strongly suggest that changes like this not happen during the lifetime of a release, but only when a new x.0 is promoted to STABLE. As I am unlikely to win that one, it is even more important that things like this be noted in /usr/src/UPDATING.
 

Alain De Vos

Daemon

Reaction score: 347
Messages: 1,290

People always do their effort. This time it is possible to formulate the complexity in a simple way.
Order:
1.Run "etcupdate -p" as pre-installworld etc-update.
2.Run installworld
3.Run "etcupdate" as post-installworld etc-update.

We live in a non-perfect world of non-perfect persons. And my name is non-perfect.
But if things are clarified in timely matter, which they are, I can live with it.
(And mergemaster was ready for improvement.)
 
OP
Zirias

Zirias

Daemon

Reaction score: 1,318
Messages: 2,344

Again, I'd like to strongly suggest that changes like this not happen during the lifetime of a release, but only when a new x.0 is promoted to STABLE. As I am unlikely to win that one, it is even more important that things like this be noted in /usr/src/UPDATING.
Well, in fact, the reason mergemaster(8) won't work reliably any more is directly tied to the switch to GIT, which was made for a new major release. Still being able to use mergemaster(8) for the upgrade from 12.2 to 13.0 is a side effect of how it works and not really intended.

But I guess you do have a point here that communication could have been better. It hit me by surprise as well ;)
 

Terry_Kennedy

Aspiring Daemon

Reaction score: 329
Messages: 961

People always do their effort. This time it is possible to formulate the complexity in a simple way.
Order:
1.Run "etcupdate -p" as pre-installworld etc-update.
2.Run installworld
3.Run "etcupdate" as post-installworld etc-update.

We live in a non-perfect world of non-perfect persons. And my name is non-perfect.
But if things are clarified in timely matter, which they are, I can live with it.
(And mergemaster was ready for improvement.)
From what I've read in this topic and in the PR, it seems to be necessary to "pre-load" etcupdate with "the way things are" if an install has been using mergemaster up to the present time. What you describe appears fine (I've never run it so I can't say for sure) for systems that have always used etcupdate. The issue is "how do we get there from here?" on existing systems that use mergemaster, and communicating that clearly near the top of UPDATING.
 
OP
Zirias

Zirias

Daemon

Reaction score: 1,318
Messages: 2,344

From what I've read in this topic and in the PR, it seems to be necessary to "pre-load" etcupdate with "the way things are" if an install has been using mergemaster up to the present time. What you describe appears fine (I've never run it so I can't say for sure) for systems that have always used etcupdate. The issue is "how do we get there from here?" on existing systems that use mergemaster, and communicating that clearly near the top of UPDATING.
It's by the way the very reason I decided to start this thread. Yes, if you used mergemaster before, you have to bootstrap etcupdate (while installing a binary release will come with a bootstrapped etcupdate) and I wasn't aware before. And it's suprirising how many people weren't aware, even including some committers it seems :-/ (my motivation to start the thread was to increase awareness before people start shooting their feet…)

So I'm with you that there is some problem. But I still don't think deprecating a tool should be mentioned more than once, my guess is that the change to the hanbook might have been too late (not sure about that), and there are too many outdated versions still available. Seeing that the PDF version still doesn't have the relevant change is really bad…

Additionally, yes, an UPDATING entry for RELEASE-13.0 should have been there.
 

grahamperrin

Aspiring Daemon

Reaction score: 189
Messages: 643

… Order:
1.Run "etcupdate -p" as pre-installworld etc-update.
2.Run installworld
3.Run "etcupdate" as post-installworld etc-update.

Early this morning, in the absence of notes, I mistakenly ran things in the wrong order:
  1. etcupdate -B before installworld
  2. make installworld
  3. etcupdate -p post installworld.
After rebooting (and regaining access to notes) I realised my mistake. Things seem to work, touch wood, but should I do anything in particular to put right what I did wrong?

14.0-CURRENT, updated from n247020-e0fa04e257c to n247266-5e5ca4c8fc5.

Code:
% bectl list -c creation
BE                    Active Mountpoint Space Created
n246499-097e8701c9f-d -      -          11.6G 2021-05-12 16:33
n247020-e0fa04e257c-d -      -          4.33G 2021-06-05 07:53
n247020-e0fa04e257c-e -      -          61.2M 2021-06-06 13:45
n247266-5e5ca4c8fc5-a NR     /          57.3G 2021-06-10 00:43
%
 

Alain De Vos

Daemon

Reaction score: 347
Messages: 1,290

I can tell what I had done.
Backup of /etc directory
etcupdate extract
cp -axfvR the newly extracted /etc in /var over the existing /etc
Perform manual changes.
This procedure is not for the faint of hart but gave me in my case a "virgin" system.
 

Terry_Kennedy

Aspiring Daemon

Reaction score: 329
Messages: 961

Well, in fact, the reason mergemaster(8) won't work reliably any more is directly tied to the switch to GIT, which was made for a new major release. Still being able to use mergemaster(8) for the upgrade from 12.2 to 13.0 is a side effect of how it works and not really intended.
With 10.4 being released October 3rd, 2017 and 13.0 being released April 13, 2021 there was still more than 3 years to provide a heads-up, switch 13.0-RELEASE to etcupdate, deal with any questions, and so on. That didn't happen. If it wasn't time-critical enough that it didn't matter whether or not it made it into 13.0 after 3 years, then it should wait until 14.0. If that isn't possible due to the tags that mergemaster depends on "going away" in 13, then the way in which it was decided to proceed with the changeover to git for 13 should be looked at, to discover why an important piece of infrastructure (mergemaster / etcupdate) was not considered.

We had similar things happen with the conversion from CVS to Subversion. That time the justification was that there may have been a compromise in one of the CVS repositories. Of course, a number of us felt at the time that it was a "the dog ate my homework" excuse.
 
OP
Zirias

Zirias

Daemon

Reaction score: 1,318
Messages: 2,344

There's still time to fix this. For users of -RELEASE versions, nothing will go wrong with mergemaster(8) up until now. Things would start to go wrong when updating to 13.1. Users of development branches are (rightfully) expected to follow mailing lists etc to know things they need to know.

So, the course of action now should be:
  • Make sure every published edition of the handbook is up-to-date, delete outdated content (people use Google)
  • Add something to UPDATING for former users of mergemaster(8) to guide them
It should have happened before 13.0-RELEASE, sure, but you can't fix that now, and it doesn't help to get furious about that…
 

Terry_Kennedy

Aspiring Daemon

Reaction score: 329
Messages: 961

There's still time to fix this. For users of -RELEASE versions, nothing will go wrong with mergemaster(8) up until now. Things would start to go wrong when updating to 13.1. Users of development branches are (rightfully) expected to follow mailing lists etc to know things they need to know.

So, the course of action now should be:
  • Make sure every published edition of the handbook is up-to-date, delete outdated content (people use Google)
  • Add something to UPDATING for former users of mergemaster(8) to guide them
And add a warning message to mergemaster saying that is is being deprecated, with a link to some place it is documented.
It should have happened before 13.0-RELEASE, sure, but you can't fix that now, and it doesn't help to get furious about that…
I'm not furious, just disheartened that the Project keeps shooting itself in the foot over and over again. With collateral damage to other feet that happen to be standing nearby.
 
OP
Zirias

Zirias

Daemon

Reaction score: 1,318
Messages: 2,344

And add a warning message to mergemaster saying that is is being deprecated, with a link to some place it is documented.
That's a great idea, and again, should have been done much earlier… 😐

I'm not furious, just disheartened that the Project keeps shooting itself in the foot over and over again. With collateral damage to other feet that happen to be standing nearby.
Just keep things in perspective. FreeBSD is still an open-source project with a lot of people working on it in their spare time. Sure that's not an excuse for failing, but often an explanation. If you compare to other FOSS projects, FreeBSD doesn't do bad, the overall documentation quality is great (IMHO).

So what to do with that? At the best, learn a lesson, somehow…
 

Terry_Kennedy

Aspiring Daemon

Reaction score: 329
Messages: 961

Just keep things in perspective. FreeBSD is still an open-source project with a lot of people working on it in their spare time. Sure that's not an excuse for failing, but often an explanation. If you compare to other FOSS projects, FreeBSD doesn't do bad, the overall documentation quality is great (IMHO).
Lots of developers work on things that they enjoy. Some developers work on things their employers pay them to do. And some developers dig through the PRs and other stuff that nobody really wants to look at, because it needs to be done. That last one is a thankless job, but may very well be the most important of the bunch.

It is because we have such talented developers in all 3 categories that I don't understand why stuff like this drops through the cracks. For example, if code goes through a Phabricator review and (for example) "docs" gets added as a reviewer, it seems to me that some time in the last 3.5 years this could have progressed to being more complete than the state it currently is in.

So what to do with that? At the best, learn a lesson, somehow…
Agreed. The problem with repeatedly shooting yourself in the foot is that you eventually run out of feet, though. :rolleyes:
 

scottro

Daemon

Reaction score: 797
Messages: 1,925

I just tried a source update on a test machine, and found that etcupdate worked without problem. I followed the handbook's instructions of running etcupdate -p and, after installing world, etcupdate -B. This seemed to correctly update /etc files. For example, with mergemaster, I would have to manually merge /etc/passwd, but etcupdate did it correctly without manual intervention. (This is a simple install, a single disk, using ZFS).
 

dvl@

Aspiring Daemon
Developer

Reaction score: 89
Messages: 597

We've managed to upgrade two hosts from 12.2 to 13.0:

  • one required a etcupdate bootstrap because it has been updated numerous times using mergemaster
  • the second was originally installed as 12.2 and did not require this bootstrap
I hope to have a blog post outlining the steps soon. We have some follow-up issues on the second host and we will work on resolving those today.
 
Top