etcupdate is not working

HL1234 · Nov 10, 2022

Hello,
I have updated to FreeBSD 13.1 with buildkernel installkernel, buildworld and installworld.
Now I need to do etcupdate
I have done etcupdate -n to see what happens. The output was:

Code:

U /etc/defaults/devfs.rules
      U /etc/defaults/periodic.conf
      U /etc/defaults/rc.conf
      U /etc/devd.conf
      C /etc/freebsd-update.conf
      U /etc/mtree/BSD.debug.dist
      U /etc/mtree/BSD.include.dist
      U /etc/mtree/BSD.root.dist
      U /etc/mtree/BSD.tests.dist
      U /etc/mtree/BSD.usr.dist
      U /etc/network.subr
      U /etc/rc.d/auditd
      U /etc/rc.d/ctld
      U /etc/rc.d/devmatch
      U /etc/rc.d/growfs
      U /etc/rc.d/gssd
      U /etc/rc.d/ipfw
      U /etc/rc.d/jail
      U /etc/rc.d/linux
      U /etc/rc.d/mountd
      U /etc/rc.d/nfsd
      U /etc/rc.d/ntpd
      U /etc/rc.d/os-release
      U /etc/rc.d/pf
      U /etc/rc.d/rctl
      U /etc/rc.d/securelevel
      U /etc/rc.d/zfs
      U /etc/security/audit_event
      U /etc/ssh/moduli
      C /etc/ssh/ssh_config
      C /etc/ssh/sshd_config
      M /etc/ttys
      C /root/.profile
      A /etc/rc.d/sysctl_lastload
      A /etc/rc.d/tlsclntd
      A /etc/rc.d/tlsservd
      A /etc/rc.d/zfskeys
    Warnings:
      Removed file changed: /etc/regdomain.xml

    "For each file that is updated a line will be output with a leading
    character to indicate the action taken.  The possible actions follow:"
         A  Added
         C  Conflict
         D  Deleted
         M  Merged
         U  Updated

Then I done etcupdate. It runs a while but give no output.
In /var/db/etcupdate/log are a lot of lines.
Then I checked the system and see that no files has changed, like /etc/rc.d/jail. It is still of 2021.

I have also tried etcupdate extract and try again etcupdate, but with the same result.
"The extract mode generates a new "current" tree."

Also I try etcupdate resolve "to resolve any conflicts encountered during a merge"
no output

It is the second time I try to use etcupdate. The first time I was been told that I must use mergemaster the last time, because etcupdate could not have a reference of etc-files to compare.

I don't know how to fix this. Can anyone help me?

SirDice · Nov 10, 2022

What does etcupdate -B do?

HL1234 · Nov 10, 2022

It shows me this (after some seconds) :

[/usr/src] => etcupdate -B
[/usr/src] =>

ll /etc/rc.d | grep jail
-r-xr-xr-x 1 root wheel 14208 Juni 3 2021 jail

HL1234 · Nov 11, 2022

SirDice there is no more idea what I can do?
I have had a similar problem 2021 https://forums.FreeBSD.org/threads/etcupdate-has-nothing-updated.80714/post-515344 For solving hat I took mergemaster - the last time. Now I switching from 13.0 to 13.1 but again it doesn't work.

/var/db/etcupdate

Code:

 root wheel      0 Nov. 10 22:12 added.files
 
-rw-r--r--   1 root  wheel  -     0B 10 Nov. 22:12:49 2022 added.files
-rw-r--r--   1 root  wheel  -   7,5K 10 Nov. 22:12:49 2022 both.files
drwxr-xr-x   2 root  wheel  -   512B 10 Nov. 22:12:49 2022 conflicts
drwxr-xr-x   7 root  wheel  -   512B 10 Nov. 22:12:49 2022 current
-rw-r--r--   1 root  wheel  -   126B  1 Juni 17:22:51 2021 etcupdate.log
-rw-r--r--   1 root  wheel  -   198K 10 Nov. 22:12:51 2022 log
-rw-r--r--   1 root  wheel  -   7,5K 10 Nov. 22:12:49 2022 new.files
drwxr-xr-x   7 root  wheel  -   512B 10 Nov. 20:37:04 2022 old
-rw-r--r--   1 root  wheel  -   7,5K 10 Nov. 22:12:49 2022 old.files
drwxr-xr-x   3 root  wheel  -   512B 10 Nov. 12:29:38 2022 preworld
-rw-r--r--   1 root  wheel  -     0B 10 Nov. 22:12:49 2022 removed.files

/conflicts is empty

Code:

ll current
total 44K
-rw-r--r--  2 root wheel 1023 Nov. 10 22:12 .cshrc
-rw-r--r--  2 root wheel  507 Nov. 10 22:12 .profile
drwxr-xr-x  2 root wheel  512 Nov. 10 22:12 boot
-r--r--r--  1 root wheel 6109 Nov. 10 22:12 COPYRIGHT
drwxr-xr-x 21 root wheel 2048 Nov. 10 22:12 etc
drwxr-xr-x  2 root wheel  512 Nov. 10 22:12 root
drwxr-xr-x  3 root wheel  512 Nov. 10 22:12 usr
drwxr-xr-x  3 root wheel  512 Nov. 10 22:12 var

Code:

 ll old

-rw-r--r--  2 root wheel 1023 Nov. 10 20:36 .cshrc
-rw-r--r--  2 root wheel  507 Nov. 10 20:36 .profile
drwxr-xr-x  2 root wheel  512 Nov. 10 20:37 boot
-r--r--r--  1 root wheel 6109 Nov. 10 20:36 COPYRIGHT
drwxr-xr-x 21 root wheel 2048 Nov. 10 20:37 etc
drwxr-xr-x  2 root wheel  512 Nov. 10 20:36 root
drwxr-xr-x  3 root wheel  512 Nov. 10 20:37 usr
drwxr-xr-x  3 root wheel  512 Nov. 10 20:37 var

The logfile log tells me nothing. There is no time or date and a directory "etcupdate-O5iDPDx" isn't also there. Maybe the developer know what he is logging. Here are the last lines:

Code:

3878 install -N /usr/src/etc  -d -m 0755 -o root  -g wheel  /var/db/etcupdate/etcupdate-O5iDPDx/etc
   3879 install -N /usr/src/etc  -C -o root  -g wheel -m 644  /usr/src/usr.sbin/ntp/ntpd/ntp.conf /var/db/etcupdate/etcupdate-O5iDPDx/e
   3879 tc/ntp.conf
   3880 ===> usr.sbin/ntp/ntpdc (installconfig)
   3881 ===> usr.sbin/ntp/ntpq (installconfig)
   3882 ===> usr.sbin/ntp/ntpdate (installconfig)
   3883 ===> usr.sbin/ntp/ntptime (installconfig)
   3884 ===> usr.sbin/ntp/ntp-keygen (installconfig)
   3885 ===> usr.sbin/ntp/sntp (installconfig)
   3886 ===> usr.sbin/ntp/doc (installconfig)
   3887 ===> usr.sbin/ntp/doc/drivers (installconfig)
   3888 ===> usr.sbin/ntp/doc/drivers/icons (installconfig)
   3889 ===> usr.sbin/ntp/doc/drivers/scripts (installconfig)
   3890 ===> usr.sbin/ntp/doc/hints (installconfig)
   3891 ===> usr.sbin/ntp/doc/icons (installconfig)
   3892 ===> usr.sbin/ntp/doc/pic (installconfig)
   3893 ===> usr.sbin/ntp/doc/scripts (installconfig)
   3894 ===> usr.sbin/keyserv (installconfig)
   3895 ===> usr.sbin/rpc.tlsclntd (installconfig)
   3896 ===> usr.sbin/rpc.tlsservd (installconfig)
   3897 ===> usr.sbin/ftp-proxy (installconfig)
   3898 ===> usr.sbin/pkg (installconfig)
   3899 installing DIRS CONFSDIR
   3900 install -N /usr/src/etc  -d -m 0755 -o root  -g wheel  /var/db/etcupdate/etcupdate-O5iDPDx/etc/pkg
   3901 install -N /usr/src/etc  -C -o root  -g wheel -m 644  /usr/src/usr.sbin/pkg/FreeBSD.conf.quarterly /var/db/etcupdate/etcupdate-[svman.de /var/db/etcupdate] => less log

How can I etcupdate bring to work?

Addition:
It seems that in current/* and old/* are the same files. I compared one file in ../etc/* newsyslog.conf of both directories in current/* and old/*. They are the same. I am sure that I have changed the file /etc/newsyslog.conf.
For what stand the current/* and old/* directories?

SirDice · Nov 11, 2022

etcupdate -B probably already fixed it for you.

How I update my -STABLE systems:

Code:

cd /usr/src
git pull
make -DWITH_META_MODE buildworld buildkernel
make installkernel
make installworld
etcupdate -B
make -DBATCH_DELETE_OLD_FILES delete-old
make -DBATCH_DELETE_OLD_FILES delete-old-libs
shutdown -r now

HL1234 · Nov 18, 2022

I do it this way:

Code:

cd /usr/src    #empty
git init .
git clone --branch releng/13.1 --depth 1 https://git.FreeBSD.org/src.git /usr/src
git branch -r
-->    origin/releng/13.1
git status
-->    On branch releng/13.1
    Your branch is up to date with 'origin/releng/13.1'.
less /usr/src/UPDATING  # reading the last parts
ln -s /root/kernelconfig/VSERVER_KERNEL /usr/src/sys/amd64/conf/VSERVER_KERNEL
make cleandir && make cleandir
tmux
cd /usr/src
date > _finished.txt
make -j3 buildkernel > buildkernel.log && echo "finished" >> _finished.txt && date >> _finished.txt &
j
-->[1]  +  35826 Running (....)
tail -f buildkernel.log
-->with ^C closing buildkernel.log
#controlling finished.txt and buildkernel.log
script installkernel.log
cd /usr/src
make installkernel   
shutdown -r +1

#next:
uname -a #ok its new

tmux
cd /usr/src
date > _finished.txt
make -j3 buildworld > buildworld.log && echo "Fertig" >> _Fertig.txt && date >> _Fertig.txt &
#after 5:44 Std
etcupdate -p
shutdown -p +1 "We stop now!"
# booting in singel user mode
console> adjkerntz -i       
console> script installworld.log
console> cd /usr/src
console> make installworld
console> ^D
console> make delete-old   
console> make delete-old-libs
reboot

#solving problem with merging files

cd /usr/obj
rm -rf *
cd /usr/src
rm -rf *
rm -rf .arcconfig .arclint .cirrus.yml .clang-format .git .gitattributes .github .gitignore

SirDice · Nov 18, 2022

HL1234 said:
Code:

cd /usr/src #empty git init .

Why are you running git init . here? This is to initialize a directory for a new git repository. There's no 'new' git repository, you're cloning an existing git repository. Which has already been initialized or else you wouldn't be able to clone or pull it in the first place.

You're supposed to do a buildworld before buildkernel.

If you do a buildkernel first then your toolchain may not be set up properly. There's actually a specific target to use when doing buildkernel without (or before) buildworld

Code:

     kernel-toolchain      Rebuild the tools needed for kernel compilation.
                           Use this if you did not do a buildworld first.

HL1234 · Nov 18, 2022

SirDice /usr/src is empty when I start. I do the commands how I have learned.
In real I do a bit more:

Code:

git init .
--> standard initialisation: some Infos

>git remote add freebsd-src https://git.FreeBSD.org/src.git

#some tests
>less .git/config
--> there are written some, and this url = https://git.FreeBSD.org/src.git

#what is, or is there, the latest version?
>git ls-remote --heads freebsd-src
--> long list, including but not limited to with
    c3c13035ef270dcf0d24d2d847dd590edc535ed0        refs/heads/releng/13.1
--> this is the newest version

#this is now not working
>git clone --branch releng/13.1 --depth 1 https://git.FreeBSD.org/src.git /usr/src
--> fatal: destination path '/usr/src' already exists and is not an empty directory.
>rm -rf /usr/src/.git

>git clone --branch releng/13.1 --depth 1 https://git.FreeBSD.org/src.git /usr/src
-->Cloning into '/usr/src'...
=>OK!

#some tests
>less .git/config
--> now again: url = https://git.FreeBSD.org/src.git
    and ther is: fetch = +refs/heads/releng/13.1:refs/remotes/origin/releng/13.1
    
>git branch -r
-->    origin/releng/13.1
    
>git remote --v
-->    origin  https://git.FreeBSD.org/src.git (fetch)
    origin  https://git.FreeBSD.org/src.git (push)

In my documentation I found this piece, I haven't written:

Code:

Problems?
    Make sure you start "fresh".
    #make clean
    #make kernel-toolchain # This builds all the tools needed to build a kernel.
                           # Do this before
    #make buildkernel   
>tmux
>cd /usr/src
>date > _finish.txt
>make -j3 kernel-toolchain > kernel-toolchain.log && echo "finish" >> _finish.txt && date >> _finished.txt &

I believe the commands are from you

For long time I had have that problem.

SirDice · Nov 18, 2022

HL1234 said:
@SirDice /usr/src is empty when I start.

Yes, and that's fine. When you git clone a repository the target directory must be empty (or not even exist). You use git init when you want to create a new git repository yourself. You don't need to 'init' a git repository before you clone it.

Cloning is done only once. If you want to update the sources you just run git pull in that directory.

HL1234 said:

You don't need to do all this. Just git clone https://git.freebsd.org/src.git /usr/src. You can then switch branches; git checkout releng/13.1 or git checkout stable/13 for example. Or do git clone --branch releng/13.1 https://git.freebsd.org/src.git /usr/src and be checked out in the releng/13.1 branch immediately. Once you've cloned the repository you can checkout any branch you want, the --branch option just changes the branch after cloning. Without the --branch option you land in the default main branch aka -CURRENT.

HL1234 said:
>rm -rf /usr/src/.git

This actually removes all the git init . and git remote add ... you did before.

HL1234 said:
I believe the commands are from you

I'm not surprised

Best order is to simply do buildworld, buildkernel, installkernel, installworld. And I rarely, if ever, reboot after installkernel, I just follow it with installworld, delete-old{-libs} and etcupdate(8) directly after. Then I'll reboot.

HL1234 · Nov 18, 2022

I have a serious question about the etcupdate directories and what are in the manual, that's confusing me:
In the manual is written: "

Code:

To    perform    a three-way merge, etcupdate keeps copies of the current and
previous versions of files that it manages.  These copies are stored in two trees known 
as the "current" and "previous" trees.

and:
The default mode merges changes from the source tree to the destination
directory.  First, it updates the "current" and "previous" trees.   
...
If a "current" tree already exists, then that tree is saved as the "previous" tree.
An older "previous" tree is    removed    if it exists. 
By default the new "current" tree is built from a source tree.

But in /var/db/etcupdate there is no "previous" folder. There is a folder named "current" an a folder named "old".
1) Is the "old" folder the "previous" folder?
2) Does "destination directory" mean the /etc /boot (and some more) folder in the host?
And
"If a "current" tree already exists, then that tree is saved as the "previous" tree." If I do this mistake, so both "current" and "previous" are now have the same content.
3) how can I rebuild the "previous" tree form the "destination directories", suppose an update and merge has yet not be done?

SirDice · Nov 18, 2022

I'm honestly a little fuzzy on the details of etcupdate(8) myself. I've used mergemaster(8) for almost two decades, but that tool has become somewhat useless since the switch to git. So I had to switch too. I can remember mucking around a lot on a system that once started as 9-STABLE and is now running 13-STABLE. That system has been progressively upgraded over the years, every time a new major -STABLE was released ( 9 -> 10 -> 11 -> 12 -> 13 ).

The handbook now mentions doing etcupdate extract and etcupdate diff before updating the sources. Never did this of course, that information was read after I mucked everything up

What does etcupdate diff now show?

HL1234 · Nov 18, 2022

etcupdate diff | less shows a lot of output. I know this files. I have something changed in them form /etc.

Code:

"The diff mode compares the versions of files in the destination directory
to the "current" tree and generates a unified format diff of the changes.
This can be used to determine which files have been locally modified and how.""

but this does not fill the "old/" folder. It is just to been prepared what could be merged.

Code:

"The extract mode generates    a new "current"    tree. 
Unlike the default mode, it does not save any existing "current" tree."

But this will not fill the "old/" folder, what I think is the "previous" folder mentioned in the man page.

So I wish someone can tell me: "Yes the "previous" files are in the old/ folder" or if not, where do they go?

and

3) how can I rebuild the "previous" tree form the "destination directories", suppose an update and merge has yet not be done and in old/ are the same files like in current/?

Yet - i like to close for that moment and will be back in the weekend. Thanks for answering

PMc · Nov 18, 2022

My experience with etcupdate:

one must initialize it before the first use. So, before attempting an upgrade, make sure that the stuff in /usr/src still matches the current installation. Then run etcupdate extract. This is only needed once.

(The default location is /var/db. Make sure that this will be mounted in singleuser, or place it somewhere else with a configfile in /etc/etcupdate.conf)
Then do the normal upgrade cycle. Update the src tree, then buildworld, buildkernel, installkernel.
Reboot into singleuser and run etcupdate -p && etcupdate resolve
Now do the installworld.
Then run etcupdate && etcupdate resolve
Reboot into operation.

PMc · Nov 18, 2022

HL1234 said:
I have also tried etcupdate extract and try again etcupdate, but with the same result.
"The extract mode generates a new "current" tree."

Okay, so you have done etcupdate extract after the upgrade. So etcupdate has stored the rc-files from the new /usr/src tree as the original files of your current (old) installation, and the old rc-files in /etc as deliberate local changes. etcupdate does now assume that the reverse-diff from the new rc-files to the old rc-files (plusminus your local changes) are alltogether your local changes.
So there are two problems:

etcupdate has no reason to update anything, because it recorded the new /usr/src tree as the current (old) installation, and so there is no visible diff in the new /usr/src tree.
etcupdate has recorded all the diffs between the old installation and the new installation as local changes of yours, and in reverse form. This in addition to the actual local changes.

The most straightforward approach to fix this, without sorting it all out, and given you still have your old non-updated rc-files in /etc, would be:

kill all the stuff that is in /var/db/etcupdate
switch your gitrepo in /usr/src to the release or version that was installed before (i.e. the version from which the files in /etc are currently still from)
now run etcupdate extract
switch the gitrepo back to the release/version that the OS now has been freshly upgraded to
now run etcupdate and etcupdate resolve. This should then do the normal thing.

HL1234 · Nov 23, 2022

PMc said:
The most straightforward approach to fix this, without sorting it all out, and given you still have your old non-updated rc-files in /etc, would be:

kill all the stuff that is in /var/db/etcupdate

switch your gitrepo in /usr/src to the release or version that was installed before (i.e. the version from which the files in /etc are currently still from)

now run etcupdate extract

switch the gitrepo back to the release/version that the OS now has been freshly upgraded to

now run etcupdate and etcupdate resolve. This should then do the normal thing.

Yes, thanks. After I studied /usr/sbin/etcupdate bash script this weekend, I get also this idea. Because what I want isn't possible: etcupdate should grab all the files form /etc what is needed and put it in "prevent" and then make a diff with "current" is build from the new /usr/src. There was a lot confusing me.
Lets tell something about I has found:
"current" for my English understanding means present - that is the state what actually is installed. But after running etcupdate then in the ../current folder are the new files from /usr/src. That was wrong misunderstood by me.
In the 27 page long script I counted 41 subroutines. The most interested me is the update_cmd() which is the default funktion when no other mode was selected and extract_cmd() which is called extract_tree() and that build_tree() as "build_tree $1;" $1 parameter is $dir with "dir=`mktemp -d $WORKDIR/etcupdate-XXXXXXX". He uses not the ../current folder, but a temporary directory which is at the end deleted.
There is no ../present folder. The "present" files are in the folder ../old.

Code:

in short: (Update: 08.12.2022 some of the description was wrong)
WORKDIR=/var/db/etcupdate # unless specifically stated otherwise
# The default    work directory is _destdir_/var/db/etcupdate

OLDTREE=$WORKDIR/old
NEWTREE=$WORKDIR/current

update_cmd() #if etcupdate is called without any parameter
    local     dir new old

    # Save tree names to use for rotation later.
    old=$OLDTREE    # old = $WORKDIR/old
    new=$NEWTREE    # new = $WORKDIR/current

Temporary directory
    dir=`mktemp -d $WORKDIR/etcupdate-XXXXXXX`
and
    extract_tree $dir    #Function extract_tree() # Function extract_tree()
    
    # Compare the new tree against the previous tree.  For
    # the preworld case OLDTREE already points to the
    # current stock tree.
    #
    # -z <string> - True if the length of string is zero.
    #  that means is -p "preworld" not set then its changing
    #    OLDTREE with NEWTREE
    if [ -z "$preworld" ]; then
        OLDTREE=$NEWTREE
    fi
    
    NEWTREE=$dir    # is now the temporary directory
                    # OLDTREE is now $WORKDIR/current
...    
    for file in `cat $WORKDIR/both.files`; do
        handle_modified_file $file
    done


...
when done
    remove_tree $old;    # löschen von $WORKDIR/old

then
    mv $OLDTREE $old    # mv $WORKDIR/current to $WORKDIR/old
and
    remove_tree $new;   # deleting of $WORKDIR/current

At the end moving the  temporary directory to ../current and with thi it is deletet
    mv $NEWTREE $new    # mv $WORKDIR//etcupdate-XXXXXXX to $WORKDIR/current


#---------------------------------    
handle_modified_file()
{    
    local cmp dest file new newdestcmp old

    compare $OLDTREE/$file $NEWTREE/$file
    #HL: compare ../current/$file    ../etcupdate-XXXXXXX/$file

    update_unmodified $file;    # Function update_unmodified()
    # Update a file that has no local modifications.
...
}

These three files:
removed.files
added.files
both.files

Goes line by line into the subroutines:
handle_removed_file()
handle_removed_directory()
handle_modified_file()   
handle_added_file()       

This is not all about, what happens in the code but now I understood that in /current are now the actual files from /usr/src. In ../old is that what was in ../current.

[I]If something is wrong, what I has written, please correct me[/I] 🤔

HL1234 · Nov 23, 2022

PMc I found a tarball form /var/db/etcupdate/current of the latest 13.0 installation.
It was build with
etcupdate build -L FreeBSD-13_0-configuration-files_210605.log FreeBSD-13_0-configuration-files_210605.tar.bz2

I have done what you told me. I cleared all the stuff that is in /var/db/etcupdate and delete all files in ../old and ../current.

Then I extracted the *.tar.gz file in /var/db/etcupdate/old.
Then I do a etcupdate extract for the 13.1 version. After that I do

etcupdate -B

I have a lot of files "ignored" with /etc/etcupdate.conf, but for the rest of the files nothing is changed in the system

The files new.files and old.files have now the same content.
In ../old and ../current are now the same content of files.
The directory ../conflicts is empty
The file added.files is empty | 0.

I.E. in /etc/rc.d are no new scripts. In /etc/* are no new or changed files.

What has going wrong?

PMc · Nov 23, 2022

HL1234 said:
PMc I found a tarball
[...]
Then I extracted the *.tar.gz file in /var/db/etcupdate/old.

HL1234 said:
I have done what you told me.

I didn't say anything like that.

HL1234 · Nov 24, 2022

PMc said:
I didn't say anything like that.

Yes, but I ~~think~~ thought that in the script it would compare ../old with the new files from an temporary directory. I read your advice again and see the tarball from 13.0 should go to the ../current directory. I will try it with this again.

Update-221124: with putting the old tarball in ../current it seems that it hat worked. 34 files now updated or added.
I created the tarball last year in /var/db/etcupdate with:
etcupdate build -L FreeBSD-13_0-configuration-files_210605.log FreeBSD-13_0-configuration-files_210605.tar.bz2

In the manual is written:

Code:

Default Mode
If    the -r option is not specified,    then the first step taken is to    update
     the "current" and "previous" trees. If a "current" tree already exists,
     then that tree is saved as    the "previous" tree.  An older "previous" tree
     is    removed    if it exists.

The "previous" tree is in the ../old folder. But in the /old folder nothing has changed? Maybe because this time the /old and /current has had the same content as I started the etcupdate -B.
With the next FreeBSD version I will see what will happen.

PMc · Nov 24, 2022

Do what You like. I'm out.

etcupdate is not working

HL1234

SirDice

Administrator

HL1234

HL1234

SirDice

Administrator

HL1234

SirDice

Administrator

HL1234

SirDice

Administrator

HL1234

SirDice

Administrator

HL1234

PMc

PMc

HL1234

HL1234

PMc

HL1234

PMc