freebsd-update upgrade conflicts and mergemaster

gladiola

gladiola

I usually install my upgrades by just rebuilding the system. This time, I tried the freebsd-update upgrade method. Similar to what's outlined here: https://docs.freebsd.org/en_US.ISO8859-1/books/handbook/updating-upgrading-freebsdupdate.html . I had some conflicts with an unusual notation, that appear to be related to using a program called mergemaster.

Is there a notation explaining these conflicts and their resolution in vi in the directions somewhere?

I was able to get through it by doing my usual "100 Mistakes a Day." However, I thought those notations with the <<<< and the ==== marks might be confusing. How is a newbie to understand this if this is not covered in the handbook? Is everyone else finding out the hard way, too?

My conflicts were in ssh-related programs. Failure to resolve the conflicts by editing the files was creating a situation where I was locked out of the computer. I got around it by using a backup. It seems like this might be an area where a lack of easy-to-find documentation might cause problems for someone new. Like, when I just accepted the files as-is, without editing them, sshd would not work. Since a restart is a part of the upgrade process, that could be bad luck for somebody.

Can we get the handbook edited to mention this use of mergemaster with freebsd-update upgrade?
 
gladiola said:
Is there a notation explaining these conflicts and their resolution in vi in the directions somewhere?
Click to expand...
Conflicts shouldn't happen. They only happen if you modified those files initially yourself, which then conflicts with the changes that need to be made for the update. So there's no clear cut explanation or resolution of the possible conflicts because they depend on the changes you originally made. Only you can answer the question what should remain, which changes should be allowed and how your settings are to be merged with the changes of the update.
 
Read the similar thread
forums.freebsd.org

Upgrade to 13.0 -BETA2

Going to test freeBSD 13.0-BETA2 I do: freebsd-update upgrade -r 13.0-BETA2 , but it gives me : The following file could not be merged automatically: /etc/login.conf Press Enter to edit this file in vi and resolve the conflicts manually... Press Enter and go to /etc/login.conf > 13.0-BETA2...
forums.freebsd.org forums.freebsd.org

If you not familiar with 'vi' - just change a default editor to 'ee' or 'mcedit' before freebsd-update running.
The other editors instead vi may save your time.
In case of using csh as shell: run the command setenv EDITOR ee. It will set 'ee' as default editor.

The notation like '<<' '>>' is used by popular software named diff().
diff shows the differences between a couple of files.
'<<' means 'old' or 'removed lines'
'>>' means 'new lines'.

While using mergemaster or freebsd-update you may receive the output like:
<<<
existing file, old line
existing file, old line
====
new file line
new file line
>>>>

You have to choose a correct combination from old and new ones.
Every file may contain a few conflickted blocks '<<==>>'. You must resolve any of them.
Marks '<< == >>' should be removed too, while solving conflicts.
 
  • Thanks
Reactions: rwp
SirDice said:
Conflicts shouldn't happen. They only happen if you modified those files initially yourself, which then conflicts with the changes that need to be made for the update. So there's no clear cut explanation or resolution of the possible conflicts because they depend on the changes you originally made. Only you can answer the question what should remain, which changes should be allowed and how your settings are to be merged with the changes of the update.
Click to expand...
I meant, shouldn't it be noted that mergemaster was being used? We can see the diff notation, but we didn't put it there. The upgrade process is intruding into the config files and not telling operators how or what it put in. If there had been a note in there that the syntax was from mergemaster, it would be easier to look up an explanation.

In the reply above, we can see an example answer. In my case, I had to look up this one: https://forums.freebsd.org/threads/upgrading-to-9-1-resolving-conflicts-using-vi.37308/

If our programs or files are being altered, it seems like we deserve an explanation that would let us use the man pages to look up what to do. That's absent in these cases.
 
gladiola said:
I meant, shouldn't it be noted that mergemaster was being used?
Click to expand...
freebsd-update(8) doesn't use mergemaster(8). Just read it, /usr/sbin/freebsd-update is a giant shell script. mergemaster(8) actually uses a very different way of merging files (by picking left or right).

How do you think changes in configuration files are done? Sometimes updates/upgrades involve different configuration options or at least some changes to those files. Normally these changes are merged automatically (you will find a list of modified files at the end of the update/upgrade). But because, at some point in time, those files have been modified in such a way the merging can't be done automatically anymore. So it asks you what to do. You, as the administrator of that system, should know which of these changes are to be applied because you as a systems administrator have likely modified those files in the first place. And you probably had a good reason to do so.
 
Well, it's apparently doing the same job, producing the same output, and lacking any direction for remedy, as I outlined earlier. I will upgrade the next box, witness the same fault, and try to provide some photos. Perhaps then we can see that there's limited guidance.

This type of problem will trip up anybody who modifies that sshd_config. Notice that to install 2FA on that ssh, that file will need to be modified. We should have some kind of warning or direction about the edits that are being made to our config files automatically.
 
gladiola said:
it's apparently doing the same job, producing the same output
Click to expand...
You've never used mergemaster(8) because its output and conflict resolution is very different.

gladiola said:
This type of problem will trip up anybody who modifies that sshd_config.
Click to expand...
Nope. Only if you modified it in such a way the merge can't be done automatically any more. I have several systems with modified sshd_config and none of them had merge conflicts.

gladiola said:
We should have some kind of warning or direction about the edits that are being made to our config files automatically.
Click to expand...
That's exactly what the screen is telling you. You want a warning for the warning? And at the end of the freebsd-update upgrade you are presented with a list of all the files that are going to be modified (automatically or not). Note that these changes haven't been made yet. They're going to be applied with the freebsd-update install step, not during the freebsd-update upgrade steps.
 
For most cases you can restore from backup your own sshd_config, after freebsd-update had changed it.
 
SirDice said:
You've never used mergemaster(8) because its output and conflict resolution is very different.
Click to expand...
OK, it's like this. That thread I linked to earler: it holds an example that looks exactly like what I saw. The output was exactly the same.

I churned through this process several times. It's like this: freebsd-update upgrade alters our config files, and then doesn't explain the annotations it makes in the files. If you don't undo what it does, the programs it affects will fail. We need a process for upgrading that doesn't damage the configuration of a working system by rendering it broken. We don't have that right now.

Arguing with me about what you didn't witness is not being helpful or constructive. Let me get through an upgrade far enough to get a picture of the failure, and then you can see for yourself. I'll be sure to include an example of what I told you about.
 
im said:
For most cases you can restore from backup your own sshd_config, after freebsd-update had changed it.
Click to expand...
The entire system had to be restored from backup because there was no way to get into the host once ssh was broken by the upgrade. This also meant that the upgrade process had to be started over, because the backups were from a while before the upgrade began.
 
gladiola said:
OK, it's like this. That thread I linked to earler: it holds an example that looks exactly like what I saw. The output was exactly the same.
Click to expand...
Yes, the output is the same. The explanation that it is caused by mergemaster(8) made by wblock@ however was not. His explanation of how to deal with it is correct though.

gladiola said:
Arguing with me about what you didn't witness is not being helpful or constructive.
Click to expand...
I know exactly what you see. I've been using freebsd-update(8) for as long as it's been available (6.2-RELEASE if I'm not mistaken). I've upgraded thousands of systems over the years that way.
 
OK, I stand corrected. Still, I will try to get a photo of what I saw. I felt that these directions should be somehow better covered by the upgrade process described in the manual.
 
im gave a nice explanation already in post #3.

But what exactly it has a conflict with is going to depend severely on your situation. Your conflicts just aren't going to be the same as my conflicts in any of those configuration files. That's why nobody will be able to tell you which of those changes are good and which are not. Because it all depends on your specific configuration. Only you will be able to decide which of these changes are good to apply. The markers just give you an idea of which part of the current configuration file it has a merge problem. So you edit that file until the configuration suits your situation.

gladiola said:
The entire system had to be restored from backup because there was no way to get into the host once ssh was broken by the upgrade.
Click to expand...
With any update/upgrade there's a certain risk involved. This risk gets bigger when you're doing this remotely. That's why it's important to have console access, through IPMI, a networked KVM, serial console, remote hands, anything. I've had this happen too, more than once over the years. Even without doing updates/upgrades, just a dumb typo in /etc/fstab causing the system to get stuck in single user mode will do it. These things happen. Make sure you have some sort of access in case of emergencies like this.
 
Yeah, that's apparent ... now. Look at the directions in the handbook. Do they tell you anything about this? No. It's like: surprise! We edited your files. Now you have to fix them so that they will work at all. BTW, if you just accept our changes and continue on, then you won't be able to use your machine at all.

There's no preparation for this in the manual. There's no emphasis on the significance of the changes that are made. And, the configuration changes are destructive to the system.

When you get to the end, right before you have to commit to a restart, there's also no way to easily identify what they changed so that you can go back and check it again. Screw up one line, and your box is hosed. It needs warnings and directions and an opportunity to identify risk before commitment.
 
Here are the pictures.
1. fetching.png : Fetching the files that are about to be merged
2. resolveConflicts.png: Direction to manually edit files that were merged automatically
3. changes.png: diff - like notation that breaks the configuration and prevents the program from working.

This process does not ask to edit the files. It makes its changes on its own, damages the configuration so that ssh will no longer work, and then asks us to "manually edit" the files. Notice how there's no warning about how the upgrade process just broke a program it can't fix. It doesn't offer to revert to the previous configuration on its own. It can't undo its own failures. And it doesn't warn users that their system is about to become inaccessible through ssh. This is a fatal error in the upgrade process, and it carries no warning.

I feel that this needs warnings in the documentation and a process improvement. Want to get locked out of a box during an upgrade? Just accept what the freebsd-update upgrade does and don't change anything. This is not an acceptable way to handle exceptions or errors during the program run. We need an improvement.
 

Attachments

  • fetching.png
    fetching.png
    53.6 KB · Views: 138
  • resolveConflicts.png
    resolveConflicts.png
    256.2 KB · Views: 163
  • changes.png
    changes.png
    680.4 KB · Views: 143
gladiola said:
This process does not ask to edit the files.
Click to expand...
Those files have been automatically merged without issues. It's not going to bother you with asking about every single change it makes, that will become tedious really quick. Especially if many of these changes are quite mundane (like updating the RCS string or a copyright notice).
gladiola said:
It makes its changes on its own, damages the configuration so that ssh will no longer work, and then asks us to "manually edit" the files.
Click to expand...
No, that's not what's happening. It tried to merge it automatically, failed to do so, then presented you a file that includes the changes with fairly obvious markers, for you to edit and review. This process doesn't look at which file it has a problem with. You get the exact same notice and are asked to edit and review on any file it happens to fail to merge automatically. Exactly this process prevents freebsd-update(8) from damaging your configuration automatically.

I'm fairly certain you didn't pay attention at the end of the freebsd-update upgrade process. Because it presents you with a list of all the files it's going to remove or merge. Nothing has been changed at this point, you have the opportunity to back out here. If you have a problem with any of the files being listed, don't run freebsd-update install. By running freebsd-update install you are acknowledging those changes are going to be made. Once the changes have been been made you have another opportunity to double check those files before you reboot. Don't reboot if you're uncertain about specific files, like your sshd_config for example, because that file is important to your specific configuration. For me it doesn't matter, because I have IPMI or KVM access to all remote servers. I don't bother double-checking it. If it fails it fails and I will deal with it.

gladiola said:
And it doesn't warn users that their system is about to become inaccessible through ssh.
Click to expand...
If you edit sshd_config by hand, make a mistake (a typo in the IP address for example) and reboot, does it warn you've locked yourself out? If you modify firewall rules incorrectly, does it warn you you might not be able to get back into the system? Is it going to cry foul if you run rm -f /etc/passwd? The system will not prevent you from shooting yourself in the foot. And believe me, I've shot myself in the foot countless times. Which is why I always make sure I have some way of getting back into the system, no matter how big of a mistake I make. You're going to make mistakes. Everyone does. Some more often than others but we've all been in a situation where we screwed up and locked ourselves out or managed to make the system fail to boot at all.
 
SirDice
I just have to reply because the head in the sand is infuriating. The option requested is that when there is a conflict there is a way to ignore the merge so that there is no change, no RCS or diff markers added to the file, so that a user can just go through all the files the updater requests for user review and manually ignore changes instead of having to worry about the program actually modifying the text files. This is especially a problem when a user (like me) blindly thought :q on the editor would mean changes including the diff markers would be gone when at the end of the process it actually corrupted all of them. It wasn't obvious that those diff markers needed to be changed because it was strange that an updater would put the actual change markers in the file rather than giving a preview of the changes and giving an option to discard or manually edit. It also didn't seem right because that was quite a lot of tedious work if there was a lot of files, especially in my scenario, and a combination of not being able to go back if I made a mistake merging conflict without restarting the whole process again and wishful thinking led me to believe that those markers would be automatically removed in the event the actual file was modified. I didn't have a backup and so I made a script to comment out all the offending lines so I was able to boot again. Problem now is that when I go and update to a new version such as 12.3 -> 12.4 freebsd-update is getting stuck on all those files I commented on and now have to manually edit/make a script to remove those and it's tedious. It would be very handy if I could tell the program to skip those files when it comes to them like an updater in majority of Linux distros and other OSs. You're here talking like it's the user's problem when this is clearly the software being obtuse.
 
Just ignoring conflicting files would be a horrible band-aid, because more conflicts would pile on the next upgrades, up to a point where resolving them becomes a close-to-impossible task, plus software might not work any more because it needs some newer configurations that are now missing. Sure, some "Linux tools" do such things, still it's stupid.

About the request to provide an interactive way for resolving conflicts: Please don't!. I've never seen a tool doing so that's efficient to use. Just using your preferred editor is way better, so these conflict markers are the most user-friendly way, once you understand how they work, and it's really not difficult.

So, yes, this is a "user problem", it's documented how these things work on FreeBSD, and it's the most efficient way I've ever seen to resolve conflicts.

Side note, this thread is very old, don't use mergemaster any more, it's deprecated. For updates from source, the tool to use now is etcupdate.
 
zirias@
This thread is about freebsd-update and I've come to learn that it doesn't use either mergemaster or etcupdate, but diff3. Why do the markers need to always be inserted into the file? Why can't the user have an option to just to see them without them being inserted (such as right above the enter to resolve merge prompt)? Why should the user be forced to remove these diff markers even if the only change is the RCS/version marker over a lot of files? If mergemaster or etcupdate were to be implemented into freebsd-update would that be the worst thing in the world? Sure, if the user blindly ignores new config software may stop working, but at least it would still be bootable and not require a reinstall or restore from backup assuming they made the latter especially for a once in a while hobby test system.
 
  1. Mergemaster is in the thread title, so I think the hint that it's deprecated is needed, just in case.
  2. Etcupdate (for use with updates from source) will do the same thing on conflicts: Insert conflict markers, so you can resolve the conflict with your favorite text editor (and, of course, list conflicted files). I literally just explained why this is the most efficient (and safe) way to handle these things.
  3. A single line removed will not lead to a conflict unless you modified exactly that line locally. So, something else was already wrong on your system previously.
  4. Even if you missed conflicts, there's *no* reason at all to reinstall/restore. You can easily find conflict markers below /etc e.g. with grep(1), and the markers contain all information to resolve the conflict.
  5. Of course, once you just removed the markers, you're doomed ... you will end up with files containing different versions and no markers to understand the situation.
 
2. The difference with etcupdate is that it has resolve mode which accounts for the scenario I'm talking about and allows the user to see changes and/or disregard changes without the file being modified. From man page (https://www.freebsd.org/cgi/man.cgi?etcupdate):
Resolve Mode
The resolve mode is used to resolve any conflicts encountered during a
merge. In this mode, etcupdate iterates over any existing conflicts
prompting the user for actions to take on each conflicted file. For each
file, the following actions are available:

(p) postpone Ignore this conflict for now.
(df) diff-full Show all changes made to the merged file as a unified
diff.
(e) edit Change the merged file in an editor.
(r) resolved Install the merged version of the file into the desti-
nation directory.
(mf) mine-full Use the version of the file in the destination direc-
tory and ignore any changes made to the file in the
"current" tree.
(tf) theirs-full Use the version of the file from the "current" tree and
discard any local changes made to the file.
(h) help Display the list of commands.
Click to expand...
3 - 4. Yes, there was a mess up because I was following 24.6.1. Quick Start before the warning at https://docs.freebsd.org/en/books/handbook/cutting-edge/#updating-src-quick-start was moved up and made more prominent and a large part of my configuration needed fixing. As said I made a script to quickly just comment out the offending conflict markers as I did not have time to implement the logic to remove everything from separator to new or old, but other people may not have any of that kind of time. Additionally, commenting or removing those conflicting lines without the aforementioned logic will yield configuration files that have both old and new changes in the same file causing issues. There may be other future scenarios where an experimental system script or other goes awry and causes havoc and the freebsd-update is frustratingly not fit to account for that.
 
You must log in or register to reply here.
Back
Top