Solved Upgrade to 12.4-RELEASE Syntax error stops boot

R

russelld

Hi Vererated FreeBSD Brains Trust,

After unpdating FreeBSD nas box from 12.3 to 12.4-RELEASE the boot halts with this:
Code: 
/etc/rc.subr: 2: Syntax error: redirection unexpected (expecting word)
Enter full pathname of shell or RETURN for /bin/sh

Entering RETURN and arriving at command line, asked for uname -a
Code: 
# uname -a
FreeBSD 12.4-RELEASE FreeBSD 12.4-RELEASE r372781 GENERIC amd64

A photo of the output is attached as I can't ssh into the box.

This is a very bog standard FreeBSD, nothing clever with software mods, installed from packages, no ports.
With a jail for the zfs which holds the files.

Thanks in advance for help with how to proceed with this.
 

Attachments

  • 20221215_152548.jpg
    20221215_152548.jpg
    126.5 KB · Views: 162
Hi SirDice and cracauer, thankyou for the reply!

re: /etc/login.conf
This appears to have merge conflicts.
See attached photo

re: grep ==== /etc/rc*
Git: 
# grep ==== /etc/rc*
/etc/rc.subr:=======
Opening /etc/rc.subr shows merge errors as well.
Photo attached.

Thank you for your help with this,
 

Attachments

  • 20221216_102831.jpg
    20221216_102831.jpg
    216.5 KB · Views: 156
  • 20221216_104109.jpg
    20221216_104109.jpg
    199.7 KB · Views: 139
Delete the lines between <<<<<< current version and ===========. Then remove those <<<<, ==== and >>>>> marker lines. Check the rest of the file for similar markers and fix it in the same way.
 
Hi SirDice,
Thank you for the instructions.
After editing as you suggest, I cannot save changes as editor (vim) tell me the file is read-only.

Checking the file permissions:
Code: 
# ls -la /etc/login.conf
-rw-r--r--   1  root  wheel   6961 Dec 14 22:46 /etc/login.conf

Being logged in as root user, I expected to be able to edit system files, so this is unusual.

How to tackle this new problem?
 
When you boot to single user mode only the root filesystem is mounted and is mounted read-only.

For UFS, mount -a should mount everything and should remount the root filesystem read/write.
For ZFS, zfs set readonly=off zroot/ROOT/default (assuming the default pool name and BE; adjust if needed).

After you modified /etc/login.conf you may need to run cap_mkdb /etc/login.conf to update the datebase. You don't need to do this for the other /etc/rc* files that had merge issues. You only need to fix the files themselves.
 
Hi SirDice and Alain, thank you for taking the time to help.

the output from the various commands:
"kenv | egrep ..." is
Code: 
# kenv | egrep "currdev|mountfrom|kernel_path|kernelname"
currdev="zfs/:zroot/ROOT/default:"
kernel_path="/boot/kernel"
kernelname="/boot/kernel/kernel"
vfs.root.mountfrom="zfs:zroot/ROOT/default"
from "service zpool .."
Code: 
# service zpool onestart
/etc/rc.subr: 2: Syntax error: redirection unexpected (expecting word)
from "service zfs .."
Code: 
# service zfs onestart
/etc/rc.subr: Syntax error: redirection unexpected (expecting word)

Ran the ZFS commands to enable writing to the disk, edited /etc/login.conf and /etc/rc.subr.
Updated database cap_mkdb /etc/login.conf
Rebooted to similar looking prompt as in above previous photos.

While at prompt, I checked zpools with "zpool status"
zroot resilvered itself yesterday morning (1008K with 0 errors)
Pool that holds data is on separate disks and is error free
Also I checked zfs with "zfs list" and found zsnapshot routinely taking snapshots up until just before the upgrade.
So possible, if needed, to roll back to previous snapshot

Thought that there could be other config files that are still unmerged.
Ran "grep -r ==== /etc/*" which found 2 screens of configs that need editing!
I get busy and report back.

Thanks again for the elegant solutions!
 

Attachments

  • 20221217_132730.jpg
    20221217_132730.jpg
    143.6 KB · Views: 134
  • 20221217_132738.jpg
    20221217_132738.jpg
    167.1 KB · Views: 140
After editing above files to remove merge conflicts, rebooted into multiuser mode, normal prompt, network came up.

Couldn't have done it without your help, thanks again!
 
Would that many config files have changed between 12.3 and 12.4?

Main thing is that it’s fixed for you but seems like a lot of churn and manual intervention needed for a point release on a bog standard install so is something else going on?

I don’t know, just curious!
 
Hi Richard,
Thanks for asking.
Yes there was something else going on.

This box (HP-N40L microserver) has been siting unpowered on a worshop desk for about 9 months after a hasty move from office about to be flooded in Feb this year.
I could see on the boot up the boot disks having some trouble so reseated all the sata cables and pci card. After reseating the disks all came back on line.
I suspect the hasty move and long period of idleness had disturbed the harddrive sata cables.
I don't think this is to do with FreeBSD system, its to do with fragile hardware.

As far as I can tell, the one of the zfs disks on the pool for ROOT became detacted from the pool midway through the upgrade.
How I know from a "zpool status" says the disk was resilvered at around the time of the upgrade.
There are stil CKSUM errors, which I see to over next few days.
Code: 
$ zpool status zroot
  pool: zroot
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: resilvered 1008K in 0 days 00:00:04 with 0 errors on Fri Dec 16 01:48:48 2022
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0p3  ONLINE       0     0     0
            ada1p3  ONLINE       0     0     6

errors: No known data errors

If this was any other OS and file system, it would have been a lot more difficult to recover, if at all

Moral of story: make sure the disks and file system is all good before attempting an upgrade.
And the OS is FreeBSD with ZFS it can recover from hardware failure mid upgrade!
 
You must log in or register to reply here.
Back
Top