unexpected soft update inconsistency

dr. bob · Oct 12, 2022

Terry_Kennedy said:
If you can mount the filesystem from a newer version of FreeBSD (11.1, 13.1, whatever), you can start networking (even in single-user mode) and use whatever method you'd like (rdump(8), NFS export, etc.) to get your data from that system to a new VM. HOWEVER just because a particular FreeBSD version can mount a corrupted filesystem doesn't mean that it won't panic when trying to access files on that filesystem. Mounting doesn't do the full set of consistency checks that fsck(8) does - it just checks to make sure the filesystem is of a known type and that the superblock is properly formed.

you might want to take a look here to see if there's a consultant with FreeBSD experience that can assist you.

Terry, thanks. I like the challenge of DIY, but it's definitely slow going. Do the helpful people here ever take work as consultants?

I'm not sure I actually can start networking. I tried ping and had 0% success.

As far as fsck, now I'm wondering if "unexpected soft update inconsistency" is really "corruption". Would the journal (whatever that is) being out of sync with the filesystem (maybe because I told fsck once not to use the journal) be considered a soft update inconsistency? Could the journal be out of sync and the filesystem still be ok? Can I just delete the journal? Would it restart from scratch? Then would there no longer be any inconsistency?

Also, does "file system was modified" necessarily mean fsck found problems and fixed them? I see here and elsewhere that updating the badblock and / or free inode counts, even if unchanged, can trigger that message.

I guess my wishful thinking is that the file system is (now) ok and the reason it can't boot is a new Hyper-V / old FreeBSD incompatibility.

dr. bob · Oct 12, 2022

_martin said:
Still it would be interesting to know what it fails now.

Martin, here you go (part 1, bumping into the character limit here):

Code:

 +============Welcome to FreeBSD===========+ +o   .--`         /y:`      +.
 |                                         |  yo`:.            :o      `+-
 |  1. Boot Multi User [Enter]             |   y/               -/`   -o/
 |  2. Boot [S]ingle User                  |  .-                  ::/sy+:.
 |  3. [Esc]ape to loader prompt           |  /                     `--  /
 |  4. Reboot                              | `:                          :`
 |                                         | `:                          :`
 |  Options:                               |  /                          /
 |  5. [K]ernel: kernel (1 of 2)           |  .-                        -.
 |  6. Configure Boot [O]ptions...         |   --                      -.
 |                                         |    `:`                  `:`
 |                                         |      .--             `--.
 |                                         |         .---.....----.
 +=========================================+
                                          

/boot/kernel/kernel text=0xfe2de8 data=0x129430+0x207fa0 syms=[0x8+0x146f88+0x8+0x1613ae]
/boot/kernel/accf_data.ko size 0x1598 at 0x19bd000
/boot/kernel/accf_http.ko size 0x26a8 at 0x19bf000
Booting...
Table 'FACP' at 0x3ffe16a3
Table 'APIC' at 0x3ffe1717
APIC: Found table at 0x3ffe1717
APIC: Using the MADT enumerator.
MADT: Found CPU APIC ID 0 ACPI ID 0: enabled
SMP: Added CPU 0 (AP)
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.3-RELEASE-p11 #0: Mon Oct 24 18:49:24 UTC 2016
    root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
Hyper-V Version: 6.1.7100 [SP0]
  Features: 0x222<TMREFCNT,HYERCALL,REFTSC>
  Recommends: 00000000 ffffffff
  Limits: Vcpu:-1 Lcpu:64 Int:0
Timecounter "Hyper-V" frequency 10000000 Hz quality 10000000
Preloaded elf kernel "/boot/kernel/kernel" at 0xffffffff819c2000.
Preloaded elf obj module "/boot/kernel/accf_data.ko" at 0xffffffff819c2c70.
Preloaded elf obj module "/boot/kernel/accf_http.ko" at 0xffffffff819c3160.
Calibrating TSC clock ... TSC clock: 2200008884 Hz
CPU: Intel Xeon E312xx (Sandy Bridge, IBRS update) (2200.01-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206a1  Family=0x6  Model=0x2a  Stepping=1
  Features=0x783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
  Features2=0x9fb82203<SSE3,PCLMULQDQ,SSSE3,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,HV>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  XSAVE Features=0x1<XSAVEOPT>
3rd-level cache: 16MByte, 16-way set associative, 64 byte line size
2nd-level cache: 2-MB, 8-way set associative, 64-byte line size
1st-level instruction cache: 32 KB, 8-way set associative, 64 byte line size
1st-level data cache: 32 KB, 8-way set associative, 64 byte line size
L2 cache: 512 kbytes, 16-way associative, 64 bytes/line
Hypervisor: Origin = "Microsoft Hv"
real memory  = 1073741824 (1024 MB)
Physical memory chunk(s):
0x0000000000001000 - 0x000000000009bfff, 634880 bytes (155 pages)
0x0000000000100000 - 0x00000000001fffff, 1048576 bytes (256 pages)
0x0000000001a04000 - 0x000000003e573fff, 1018626048 bytes (248688 pages)
avail memory = 1009868800 (963 MB)
Event timer "LAPIC" quality 600
LAPIC: ipi_wait() us multiplier 2 (r 786515145 tsc 2200008884)
ACPI APIC Table: <BOCHS  BXPC    >
APIC: CPU 0 has ACPI ID 0
x86bios:  IVT 0x000000-0x0004ff at 0xfffff80000000000
x86bios: SSEG 0x001000-0x001fff at 0xfffffe003d12b000
x86bios: EBDA 0x09f000-0x09ffff at 0xfffff8000009f000
x86bios:  ROM 0x0a0000-0x0fefff at 0xfffff800000a0000
XEN: CPU 0 has VCPU ID 0
Hardware, Intel Secure Key RNG: RDRAND is not present
Hardware, VIA Nehemiah Padlock RNG: VIA Padlock RNG not present
Falling back to <Software, Yarrow> random adaptor
random: <Software, Yarrow> initialized
ULE: setup cpu 0
ACPI: RSDP 0x00000000000F5870 000014 (v00 BOCHS )
ACPI: RSDT 0x000000003FFE17B7 000030 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
ACPI: FACP 0x000000003FFE16A3 000074 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
ACPI: DSDT 0x000000003FFE0040 001663 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
ACPI: FACS 0x000000003FFE0000 000040
ACPI: APIC 0x000000003FFE1717 000078 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
ACPI: WAET 0x000000003FFE178F 000028 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
MADT: Found IO APIC ID 0, Interrupt 0 at 0xfec00000
ioapic0: Routing external 8259A's -> intpin 0
MADT: Interrupt override: source 0, irq 2
ioapic0: Routing IRQ 0 -> intpin 2
MADT: Interrupt override: source 5, irq 5
ioapic0: intpin 5 trigger: level
MADT: Interrupt override: source 9, irq 9
ioapic0: intpin 9 trigger: level
MADT: Interrupt override: source 10, irq 10
ioapic0: intpin 10 trigger: level
MADT: Interrupt override: source 11, irq 11
ioapic0: intpin 11 trigger: level
lapic: Routing NMI -> LINT1
lapic: LINT1 trigger: edge
lapic: LINT1 polarity: high
ioapic0 <Version 1.1> irqs 0-23 on motherboard
cpu0 BSP:
     ID: 0x00000000   VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff
  lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
  timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
wlan: <802.11 Link Layer>
snd_unit_init() u=0x00ff8000 [512] d=0x00007c00 [32] c=0x000003ff [1024]
feeder_register: snd_unit=-1 snd_maxautovchans=16 latency=5 feeder_rate_min=1 feeder_rate_max=2016000 feeder_rate_round=25
null: <null device, zero device>
nfslock: pseudo-device
VESA: INT 0x10 vector 0xc000:0x5262
VESA: information block
0000   56 45 53 41 00 03 d9 54 00 c0 00 00 00 00 22 00
0010   00 02 40 00 00 00 ed 54 00 c0 00 55 00 c0 14 55
0020   00 c0 01 01 11 01 10 01 12 01 03 01 14 01 13 01
0030   05 01 17 01 16 01 15 01 18 01 07 01 19 01 1a 01
0040   00 00 01 00 02 00 03 00 04 00 05 00 06 00 07 00
0050   0d 00 0e 00 0f 00 10 00 11 00 12 00 13 00 6a 00
0060   ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0070   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0080   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0090   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00a0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00b0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00c0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00d0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00e0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00f0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0100   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0110   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0120   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0130   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0140   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0150   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0160   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0170   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0180   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0190   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01a0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01b0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01c0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01d0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01e0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01f0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
VESA: 31 mode(s) found
VESA: v3.0, 4096k memory, flags:0x0, mode table:0xfffffe003d15e022 (2000022)
VESA: SeaBIOS VBE(C) 2011
VESA: SeaBIOS Developers SeaBIOS VBE Adapter Rev. 1
io: <I/O>
VMBUS: load
kbd: new array size 4
kbd1 at kbdmux0
mem: <memory>
hptnr: R750/DC7280 controller driver v1.1.4
hpt27xx: RocketRAID 27xx controller driver v1.2.7
hptrr: RocketRAID 17xx/2xxx SATA controller driver v1.2
acpi0: <BOCHS BXPC> on motherboard
ACPI: All ACPI Tables successfully acquired
ioapic0: routing intpin 9 (ISA IRQ 9) to lapic 0 vector 48
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu0: switching to generic Cx mode
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: registered as a time-of-day clock (resolution 1000000us, adjustment 0.500000000s)
ioapic0: routing intpin 8 (ISA IRQ 8) to lapic 0 vector 49
Event timer "RTC" frequency 32768 Hz quality 0
ACPI timer: 1/53 1/58 1/51 1/50 1/60 1/28 1/27 1/30 1/29 1/35 -> 10
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x608-0x60b on acpi0
pci_link0:        Index  IRQ  Rtd  Ref  IRQs
  Initial Probe       0   10   N     0  5 10 11
  Validation          0   10   N     0  5 10 11
  After Disable       0  255   N     0  5 10 11
pci_link1:        Index  IRQ  Rtd  Ref  IRQs
  Initial Probe       0   10   N     0  5 10 11
  Validation          0   10   N     0  5 10 11
  After Disable       0  255   N     0  5 10 11
pci_link2:        Index  IRQ  Rtd  Ref  IRQs
  Initial Probe       0   11   N     0  5 10 11
  Validation          0   11   N     0  5 10 11
  After Disable       0  255   N     0  5 10 11
pci_link3:        Index  IRQ  Rtd  Ref  IRQs
  Initial Probe       0   11   N     0  5 10 11
  Validation          0   11   N     0  5 10 11
  After Disable       0  255   N     0  5 10 11
pci_link4:        Index  IRQ  Rtd  Ref  IRQs
  Initial Probe       0    9   N     0  9
  Validation          0    9   N     0  9
  After Disable       0  255   N     0  9
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: decoding 5 range 0-0xff
pcib0: decoding 4 range 0-0xcf7
pcib0: decoding 4 range 0xd00-0xffff
pcib0: decoding 3 range 0xa0000-0xbffff
pcib0: decoding 3 range 0x40000000-0xfebfffff
pcib0: decoding 3 range 0x100000000-0x17fffffff
ACPI: Found matching pin for 0.1.INTA at func 3: 9
ACPI: Found matching pin for 0.1.INTD at func 2: 11
ACPI: Found matching pin for 0.3.INTA at func 0: 11
ACPI: Found matching pin for 0.4.INTA at func 0: 11
ACPI: Found matching pin for 0.5.INTA at func 0: 10
ACPI: Found matching pin for 0.6.INTA at func 0: 10
ACPI: Found matching pin for 0.7.INTA at func 0: 11
pci0: <ACPI PCI bus> on pcib0
pci0: domain=0, physical bus=0
found->    vendor=0x8086, dev=0x1237, revid=0x02
    domain=0, bus=0, slot=0, func=0
    class=06-00-00, hdrtype=0x00, mfdev=0
    cmdreg=0x0103, statreg=0x0000, cachelnsz=0 (dwords)
    lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
found->    vendor=0x8086, dev=0x7000, revid=0x00
    domain=0, bus=0, slot=1, func=0
    class=06-01-00, hdrtype=0x00, mfdev=1
    cmdreg=0x0103, statreg=0x0200, cachelnsz=0 (dwords)
    lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
found->    vendor=0x8086, dev=0x7010, revid=0x00
    domain=0, bus=0, slot=1, func=1
    class=01-01-80, hdrtype=0x00, mfdev=0
    cmdreg=0x0103, statreg=0x0280, cachelnsz=0 (dwords)
    lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
pcib0: allocated type 4 (0x1f0-0x1f7) for rid 10 of pci0:0:1:1
pcib0: allocated type 4 (0x3f6-0x3f6) for rid 14 of pci0:0:1:1
pcib0: allocated type 4 (0x170-0x177) for rid 18 of pci0:0:1:1
pcib0: allocated type 4 (0x376-0x376) for rid 1c of pci0:0:1:1
    map[20]: type I/O Port, range 32, base 0xc140, size  4, enabled
pcib0: allocated type 4 (0xc140-0xc14f) for rid 20 of pci0:0:1:1
found->    vendor=0x8086, dev=0x7020, revid=0x01
    domain=0, bus=0, slot=1, func=2
    class=0c-03-00, hdrtype=0x00, mfdev=0
    cmdreg=0x0107, statreg=0x0000, cachelnsz=0 (dwords)
    lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
    intpin=d, irq=11
    map[20]: type I/O Port, range 32, base 0xc0c0, size  5, enabled
pcib0: allocated type 4 (0xc0c0-0xc0df) for rid 20 of pci0:0:1:2
pcib0: matched entry for 0.1.INTD (src \_SB_.LNKD:0)
pcib0: slot 1 INTD routed to irq 11 via \_SB_.LNKD
found->    vendor=0x8086, dev=0x7113, revid=0x03
    domain=0, bus=0, slot=1, func=3
    class=06-80-00, hdrtype=0x00, mfdev=0
    cmdreg=0x0103, statreg=0x0280, cachelnsz=0 (dwords)
    lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
    intpin=a, irq=9
pcib0: matched entry for 0.1.INTA (src \_SB_.LNKS:0)
pcib0: slot 1 INTA routed to irq 9 via \_SB_.LNKS
found->    vendor=0x1013, dev=0x00b8, revid=0x00
    domain=0, bus=0, slot=2, func=0
    class=03-00-00, hdrtype=0x00, mfdev=0
    cmdreg=0x0103, statreg=0x0000, cachelnsz=0 (dwords)
    lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
    map[10]: type Prefetchable Memory, range 32, base 0xfc000000, size 25, enabled
pcib0: allocated type 3 (0xfc000000-0xfdffffff) for rid 10 of pci0:0:2:0
    map[14]: type Memory, range 32, base 0xfebf0000, size 12, enabled
pcib0: allocated type 3 (0xfebf0000-0xfebf0fff) for rid 14 of pci0:0:2:0
found->    vendor=0x1af4, dev=0x1000, revid=0x00
    domain=0, bus=0, slot=3, func=0
    class=02-00-00, hdrtype=0x00, mfdev=0
    cmdreg=0x0103, statreg=0x0010, cachelnsz=0 (dwords)
    lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
    intpin=a, irq=11
    MSI-X supports 3 messages in map 0x14
    map[10]: type I/O Port, range 32, base 0xc0e0, size  5, enabled
pcib0: allocated type 4 (0xc0e0-0xc0ff) for rid 10 of pci0:0:3:0
    map[14]: type Memory, range 32, base 0xfebf1000, size 12, enabled
pcib0: allocated type 3 (0xfebf1000-0xfebf1fff) for rid 14 of pci0:0:3:0
    map[20]: type Prefetchable Memory, range 64, base 0xfe000000, size 14, enabled
pcib0: allocated type 3 (0xfe000000-0xfe003fff) for rid 20 of pci0:0:3:0
pcib0: matched entry for 0.3.INTA (src \_SB_.LNKC:0)
pcib0: slot 3 INTA routed to irq 11 via \_SB_.LNKC
found->    vendor=0x1af4, dev=0x1003, revid=0x00
    domain=0, bus=0, slot=4, func=0
    class=07-80-00, hdrtype=0x00, mfdev=0
    cmdreg=0x0103, statreg=0x0010, cachelnsz=0 (dwords)
    lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
    intpin=a, irq=11
    MSI-X supports 2 messages in map 0x14
    map[10]: type I/O Port, range 32, base 0xc080, size  6, enabled
pcib0: allocated type 4 (0xc080-0xc0bf) for rid 10 of pci0:0:4:0
    map[14]: type Memory, range 32, base 0xfebf2000, size 12, enabled
pcib0: allocated type 3 (0xfebf2000-0xfebf2fff) for rid 14 of pci0:0:4:0
    map[20]: type Prefetchable Memory, range 64, base 0xfe004000, size 14, enabled
pcib0: allocated type 3 (0xfe004000-0xfe007fff) for rid 20 of pci0:0:4:0
pcib0: matched entry for 0.4.INTA (src \_SB_.LNKD:0)
pcib0: slot 4 INTA routed to irq 11 via \_SB_.LNKD
found->    vendor=0x1af4, dev=0x1001, revid=0x00
    domain=0, bus=0, slot=5, func=0
    class=01-00-00, hdrtype=0x00, mfdev=0
    cmdreg=0x0107, statreg=0x0010, cachelnsz=0 (dwords)
    lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
    intpin=a, irq=10
    MSI-X supports 2 messages in map 0x14
    map[10]: type I/O Port, range 32, base 0xc000, size  7, enabled
pcib0: allocated type 4 (0xc000-0xc07f) for rid 10 of pci0:0:5:0
    map[14]: type Memory, range 32, base 0xfebf3000, size 12, enabled
pcib0: allocated type 3 (0xfebf3000-0xfebf3fff) for rid 14 of pci0:0:5:0
    map[20]: type Prefetchable Memory, range 64, base 0xfe008000, size 14, enabled
pcib0: allocated type 3 (0xfe008000-0xfe00bfff) for rid 20 of pci0:0:5:0
pcib0: matched entry for 0.5.INTA (src \_SB_.LNKA:0)
pcib0: slot 5 INTA routed to irq 10 via \_SB_.LNKA
found->    vendor=0x1af4, dev=0x1002, revid=0x00
    domain=0, bus=0, slot=6, func=0
    class=00-ff-00, hdrtype=0x00, mfdev=0
    cmdreg=0x0103, statreg=0x0010, cachelnsz=0 (dwords)
    lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
    intpin=a, irq=10
    map[10]: type I/O Port, range 32, base 0xc100, size  5, enabled
pcib0: allocated type 4 (0xc100-0xc11f) for rid 10 of pci0:0:6:0
    map[20]: type Prefetchable Memory, range 64, base 0xfe00c000, size 14, enabled
pcib0: allocated type 3 (0xfe00c000-0xfe00ffff) for rid 20 of pci0:0:6:0
pcib0: matched entry for 0.6.INTA (src \_SB_.LNKB:0)
pcib0: slot 6 INTA routed to irq 10 via \_SB_.LNKB
found->    vendor=0x1af4, dev=0x1005, revid=0x00
    domain=0, bus=0, slot=7, func=0
    class=00-ff-00, hdrtype=0x00, mfdev=0
    cmdreg=0x0103, statreg=0x0010, cachelnsz=0 (dwords)
    lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
    intpin=a, irq=11
    map[10]: type I/O Port, range 32, base 0xc120, size  5, enabled
pcib0: allocated type 4 (0xc120-0xc13f) for rid 10 of pci0:0:7:0
    map[20]: type Prefetchable Memory, range 64, base 0xfe010000, size 14, enabled
pcib0: allocated type 3 (0xfe010000-0xfe013fff) for rid 20 of pci0:0:7:0
pcib0: matched entry for 0.7.INTA (src \_SB_.LNKC:0)
pcib0: slot 7 INTA routed to irq 11 via \_SB_.LNKC
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX3 WDMA2 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc140-0xc14f at device 1.1 on pci0
ata0: <Hyper-V ATA storage disengage driver> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
ioapic0: routing intpin 15 (ISA IRQ 15) to lapic 0 vector 50
uhci0: <Intel 82371SB (PIIX3) USB controller> port 0xc0c0-0xc0df irq 11 at device 1.2 on pci0
ioapic0: routing intpin 11 (ISA IRQ 11) to lapic 0 vector 51
usbus0 on uhci0
uhci0: usbpf: Attached
pci0: <bridge> at device 1.3 (no driver attached)
vgapci0: <VGA-compatible display> mem 0xfc000000-0xfdffffff,0xfebf0000-0xfebf0fff at device 2.0 on pci0
vgapci0: Boot video device
virtio_pci0: <VirtIO PCI Network adapter> port 0xc0e0-0xc0ff mem 0xfebf1000-0xfebf1fff,0xfe000000-0xfe003fff irq 11 at device 3.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci0
virtio_pci0: host features: 0x79bfffe7 <EventIdx,RingIndirect,0x8000000,NotifyOnEmpty,SetMacAddress,GuestAnnounce,RxModeExtra,VLanFilter,RxMode,ControlVq,Status,MrgRxBuf,TxUFO,TxTSOECN,TxTSOv6,TxTSOv4,RxUFO,RxECN,RxTSOv6,RxTSOv4,TxAllGSO,MacAddress,0x4,RxChecksum,TxChecksum>
virtio_pci0: negotiated features: 0x308fbbe3 <EventIdx,RingIndirect,SetMacAddress,VLanFilter,RxMode,ControlVq,Status,MrgRxBuf,TxTSOECN,TxTSOv6,TxTSOv4,RxECN,RxTSOv6,RxTSOv4,TxAllGSO,MacAddress,RxChecksum,TxChecksum>
vtnet0: bpf attached
vtnet0: Ethernet address: 00:16:3e:01:33:14
virtio_pci0: attempting to allocate 3 MSI-X vectors (3 supported)
msi: routing MSI-X IRQ 256 to local APIC 0 vector 52
msi: routing MSI-X IRQ 257 to local APIC 0 vector 53
msi: routing MSI-X IRQ 258 to local APIC 0 vector 54
virtio_pci0: using IRQs 256-258 for MSI-X
virtio_pci0: using per VQ MSIX interrupts
virtio_pci1: <VirtIO PCI Console adapter> port 0xc080-0xc0bf mem 0xfebf2000-0xfebf2fff,0xfe004000-0xfe007fff irq 11 at device 4.0 on pci0
virtio_pci2: <VirtIO PCI Block adapter> port 0xc000-0xc07f mem 0xfebf3000-0xfebf3fff,0xfe008000-0xfe00bfff irq 10 at device 5.0 on pci0
vtblk0: <VirtIO Block Adapter> on virtio_pci2
virtio_pci2: host features: 0x79006e54 <EventIdx,RingIndirect,0x8000000,NotifyOnEmpty,0x4000,0x2000,ConfigWCE,Topology,WriteCache,BlockSize,DiskGeometry,MaxNumSegs>
virtio_pci2: negotiated features: 0x10000e54 <RingIndirect,ConfigWCE,Topology,WriteCache,BlockSize,DiskGeometry,MaxNumSegs>
virtio_pci2: attempting to allocate 2 MSI-X vectors (2 supported)
msi: routing MSI-X IRQ 259 to local APIC 0 vector 55
msi: routing MSI-X IRQ 260 to local APIC 0 vector 56
virtio_pci2: using IRQs 259-260 for MSI-X
virtio_pci2: using per VQ MSIX interrupts
vtblk0: 40960MB (83886081 512 byte sectors)
virtio_pci3: <VirtIO PCI Balloon adapter> port 0xc100-0xc11f mem 0xfe00c000-0xfe00ffff irq 10 at device 6.0 on pci0
vtballoon0: <VirtIO Balloon Adapter> on virtio_pci3
virtio_pci3: host features: 0x79000002 <EventIdx,RingIndirect,0x8000000,NotifyOnEmpty,StatsVq>
virtio_pci3: negotiated features: 0
ioapic0: routing intpin 10 (ISA IRQ 10) to lapic 0 vector 57
virtio_pci3: using legacy interrupt
virtio_pci4: <VirtIO PCI Entropy adapter> port 0xc120-0xc13f mem 0xfe010000-0xfe013fff irq 11 at device 7.0 on pci0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
ioapic0: routing intpin 4 (ISA IRQ 4) to lapic 0 vector 58
uart0: fast interrupt
uart0: PPS capture mode: DCDinvalid
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
atkbd: the current kbd controller command byte 0061
atkbd: keyboard ID 0x41ab (2)
kbdc: RESET_KBD return code:00fa
kbdc: RESET_KBD status:00aa
kbd0 at atkbd0
kbd0: atkbd0, AT 101/102 (2), config:0x0, flags:0x3d0000
ioapic0: routing intpin 1 (ISA IRQ 1) to lapic 0 vector 59
atkbd0: [GIANT-LOCKED]
psm0: unable to allocate IRQ
psmcpnp0: <PS/2 mouse port> irq 12 on acpi0
psm0: current command byte:0061
kbdc: TEST_AUX_PORT status:0000
kbdc: RESET_AUX return code:00fa
kbdc: RESET_AUX status:00aa
kbdc: RESET_AUX ID:0000
kbdc: RESET_AUX return code:00fa
kbdc: RESET_AUX status:00aa
kbdc: RESET_AUX ID:0000
psm: status 00 02 64
psm: status 00 00 64
psm: status 00 03 64
psm: status 00 03 64
psm: data 08 00 00
psm: status 00 02 64
psm0: <PS/2 Mouse> irq 12 on atkbdc0
ioapic0: routing intpin 12 (ISA IRQ 12) to lapic 0 vector 60
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4-00, 5 buttons
psm0: config:00000000, flags:00000008, packet size:4
psm0: syncmask:08, syncbits:00
fdc0: <floppy drive controller (FDE)> port 0x3f2-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
ACPI: Enabled 2 GPEs in block 00 to 0F
acpi0: wakeup code va 0xfffffe004e675000 pa 0x4000
ex_isa_identify()
pcib0: allocated type 3 (0xa0000-0xa07ff) for rid 0 of orm0
pcib0: allocated type 3 (0xa0800-0xa0fff) for rid 0 of orm0
pcib0: allocated type 3 (0xa1000-0xa17ff) for rid 0 of orm0

dr. bob · Oct 12, 2022

Part 2:

Code:

pcib0: allocated type 3 (0xa1800-0xa1fff) for rid 0 of orm0
pcib0: allocated type 3 (0xa2000-0xa27ff) for rid 0 of orm0
pcib0: allocated type 3 (0xa2800-0xa2fff) for rid 0 of orm0
pcib0: allocated type 3 (0xa3000-0xa37ff) for rid 0 of orm0
pcib0: allocated type 3 (0xa3800-0xa3fff) for rid 0 of orm0
pcib0: allocated type 3 (0xa4000-0xa47ff) for rid 0 of orm0
pcib0: allocated type 3 (0xa4800-0xa4fff) for rid 0 of orm0
pcib0: allocated type 3 (0xa5000-0xa57ff) for rid 0 of orm0
pcib0: allocated type 3 (0xa5800-0xa5fff) for rid 0 of orm0
pcib0: allocated type 3 (0xa6000-0xa67ff) for rid 0 of orm0
pcib0: allocated type 3 (0xa6800-0xa6fff) for rid 0 of orm0
pcib0: allocated type 3 (0xa7000-0xa77ff) for rid 0 of orm0
pcib0: allocated type 3 (0xa7800-0xa7fff) for rid 0 of orm0
pcib0: allocated type 3 (0xa8000-0xa87ff) for rid 0 of orm0
pcib0: allocated type 3 (0xa8800-0xa8fff) for rid 0 of orm0
pcib0: allocated type 3 (0xa9000-0xa97ff) for rid 0 of orm0
pcib0: allocated type 3 (0xa9800-0xa9fff) for rid 0 of orm0
pcib0: allocated type 3 (0xaa000-0xaa7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xaa800-0xaafff) for rid 0 of orm0
pcib0: allocated type 3 (0xab000-0xab7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xab800-0xabfff) for rid 0 of orm0
pcib0: allocated type 3 (0xac000-0xac7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xac800-0xacfff) for rid 0 of orm0
pcib0: allocated type 3 (0xad000-0xad7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xad800-0xadfff) for rid 0 of orm0
pcib0: allocated type 3 (0xae000-0xae7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xae800-0xaefff) for rid 0 of orm0
pcib0: allocated type 3 (0xaf000-0xaf7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xaf800-0xaffff) for rid 0 of orm0
pcib0: allocated type 3 (0xb0000-0xb07ff) for rid 0 of orm0
pcib0: allocated type 3 (0xb0800-0xb0fff) for rid 0 of orm0
pcib0: allocated type 3 (0xb1000-0xb17ff) for rid 0 of orm0
pcib0: allocated type 3 (0xb1800-0xb1fff) for rid 0 of orm0
pcib0: allocated type 3 (0xb2000-0xb27ff) for rid 0 of orm0
pcib0: allocated type 3 (0xb2800-0xb2fff) for rid 0 of orm0
pcib0: allocated type 3 (0xb3000-0xb37ff) for rid 0 of orm0
pcib0: allocated type 3 (0xb3800-0xb3fff) for rid 0 of orm0
pcib0: allocated type 3 (0xb4000-0xb47ff) for rid 0 of orm0
pcib0: allocated type 3 (0xb4800-0xb4fff) for rid 0 of orm0
pcib0: allocated type 3 (0xb5000-0xb57ff) for rid 0 of orm0
pcib0: allocated type 3 (0xb5800-0xb5fff) for rid 0 of orm0
pcib0: allocated type 3 (0xb6000-0xb67ff) for rid 0 of orm0
pcib0: allocated type 3 (0xb6800-0xb6fff) for rid 0 of orm0
pcib0: allocated type 3 (0xb7000-0xb77ff) for rid 0 of orm0
pcib0: allocated type 3 (0xb7800-0xb7fff) for rid 0 of orm0
pcib0: allocated type 3 (0xb8000-0xb87ff) for rid 0 of orm0
pcib0: allocated type 3 (0xb8800-0xb8fff) for rid 0 of orm0
pcib0: allocated type 3 (0xb9000-0xb97ff) for rid 0 of orm0
pcib0: allocated type 3 (0xb9800-0xb9fff) for rid 0 of orm0
pcib0: allocated type 3 (0xba000-0xba7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xba800-0xbafff) for rid 0 of orm0
pcib0: allocated type 3 (0xbb000-0xbb7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xbb800-0xbbfff) for rid 0 of orm0
pcib0: allocated type 3 (0xbc000-0xbc7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xbc800-0xbcfff) for rid 0 of orm0
pcib0: allocated type 3 (0xbd000-0xbd7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xbd800-0xbdfff) for rid 0 of orm0
pcib0: allocated type 3 (0xbe000-0xbe7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xbe800-0xbefff) for rid 0 of orm0
pcib0: allocated type 3 (0xbf000-0xbf7ff) for rid 0 of orm0
pcib0: allocated type 3 (0xbf800-0xbffff) for rid 0 of orm0
ahc_isa_identify 0: ioport 0xc00 alloc failed
ahc_isa_identify 1: ioport 0x1c00 alloc failed
ahc_isa_identify 2: ioport 0x2c00 alloc failed
ahc_isa_identify 3: ioport 0x3c00 alloc failed
ahc_isa_identify 4: ioport 0x4c00 alloc failed
ahc_isa_identify 5: ioport 0x5c00 alloc failed
ahc_isa_identify 6: ioport 0x6c00 alloc failed
ahc_isa_identify 7: ioport 0x7c00 alloc failed
ahc_isa_identify 8: ioport 0x8c00 alloc failed
ahc_isa_identify 9: ioport 0x9c00 alloc failed
ahc_isa_identify 10: ioport 0xac00 alloc failed
ahc_isa_identify 11: ioport 0xbc00 alloc failed
ahc_isa_identify 12: ioport 0xcc00 alloc failed
ahc_isa_identify 13: ioport 0xdc00 alloc failed
ahc_isa_identify 14: ioport 0xec00 alloc failed
isa_probe_children: disabling PnP devices
atkbdc: atkbdc0 already exists; skipping it
atrtc: atrtc0 already exists; skipping it
sc: sc0 already exists; skipping it
uart: uart0 already exists; skipping it
isa_probe_children: probing non-PnP devices
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc97ff,0xe9000-0xeffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sc0: fb0, kbd1, terminal emulator: scteken (teken terminal)
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
pcib0: allocated type 4 (0x3c0-0x3df) for rid 0 of vga0
pcib0: allocated type 3 (0xa0000-0xbffff) for rid 0 of vga0
attimer0: <AT timer> at port 0x40 on isa0
pcib0: allocated type 4 (0x40-0x43) for rid 0 of attimer0
Timecounter "i8254" frequency 1193182 Hz quality 0
ioapic0: routing intpin 2 (ISA IRQ 0) to lapic 0 vector 61
Event timer "i8254" frequency 1193182 Hz quality 100
fdc0: No FDOUT register!
fdc0 failed to probe at port 0x3f0 irq 6 drq 2 on isa0
ppc0: cannot reserve I/O port range
ppc0 failed to probe at irq 7 on isa0
pcib0: allocated type 4 (0x2f8-0x2ff) for rid 0 of uart1
uart1 failed to probe at port 0x2f8-0x2ff irq 3 on isa0
wbwd0 failed to probe on isa0
isa_probe_children: probing PnP devices
Device configuration finished.
procfs registered
lapic: Divisor 2, Frequency 500021880 Hz
Timecounters tick every 10.000 msec
vlan: initialized, using hash tables with chaining
tcp_init: net.inet.tcp.tcbhashsize auto tuned to 8192
lo0: bpf attached
hptnr: no controller detected.
hpt27xx: no controller detected.
hptrr: no controller detected.
GEOM: new disk vtbd0
usbus0: 12Mbps Full Speed USB v1.0
ata1: reset tp1 mask=03 ostat0=41 ostat1=50
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ata1: stat0=0x50 err=0x01 lsb=0xff msb=0xff
ata1: stat1=0x00 err=0x01 lsb=0x14 msb=0xeb
ata1: reset tp2 stat0=50 stat1=00 devices=0x20000
pass0 at ata1 bus 0 scbus0 target 1 lun 0
pass0: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device
pass0: Serial Number QM00004
pass0: 16.700MB/s transfers (WDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0 at ata1 bus 0 scbus0 target 1 lun 0
cd0: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device
cd0: Serial Number QM00004
cd0: 16.700MB/s transfers (WDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: 3193MB (1634993 2048 byte sectors)
GEOM: new disk cd0
random: unblocking device.
TSC timecounter discards lower 1 bit(s)
Timecounter "TSC-low" frequency 1100004442 Hz quality 800
Event timer "HyperV" frequency 10000000 Hz quality 1000
VMBUS: vmbus callback vector 254
VMBUS: Calling smp_rendezvous, smp_started = 0
kernel trap 9 with interrupts disabled

Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer    = 0x20:0xffffffff80dedb63
stack pointer            = 0x28:0xffffffff819c8a60
frame pointer            = 0x28:0xffffffff819c8a90
code segment        = base 0x0, limit 0xfffff, type 0x1b
            = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags    = resume, IOPL = 0
current process        = 0 (swapper)
trap number        = 9
panic: general protection fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff8098e3e0 at kdb_backtrace+0x60
#1 0xffffffff809510b6 at vpanic+0x126
#2 0xffffffff80950f83 at panic+0x43
#3 0xffffffff80d55f8b at trap_fatal+0x36b
#4 0xffffffff80d55c0d at trap+0x77d
#5 0xffffffff80d3b8d2 at calltrap+0x8
#6 0xffffffff8099d725 at smp_rendezvous_cpus+0xd5
#7 0xffffffff80deed0f at vmbus_bus_init+0x2ef
#8 0xffffffff808f7ac8 at mi_startup+0x108
#9 0xffffffff802e266c at btext+0x2c
Uptime: 1s
Automatic reboot in 15 seconds - press a key on the console to abort
--> Press a key on the console to reboot,
--> or switch off the system now.

I don't know if that changes your thinking or not.

_martin · Oct 12, 2022

I hope you didn't have too much trouble with getting that log in ascii. As I mentioned earlier your photo was enough. However having this information doesn't hurt.

I'm trying to reproduce the bug. This caught my attention:

Code:

Hypervisor: Origin = "Microsoft Hv"
ACPI APIC Table: <BOCHS  BXPC    >

It's an interesting combination. I tried HyperV on my Win10, the ACPI tables have different maker (label). I spawned a VM in Azure to see it running on their infrastructure, same issue.

This got me thinking whether this is not a qemu VM on windows with hyperv accelerator (whpx). Funny enough I can't get FreeBSD 13 to boot under this combination. There's known problem that was fixed in PR 213155 long time ago.
This doesn't answer the question why it suddenly stopped working for you but then we don't know the history of the VM nor the hypervisor it was running on.
Also this doesn't change a fact you should upgrade the FreeBSD, even more so as it hosts internet facing services.

I don't want to write long posts explaining if it makes sense or not so I'd say it this way: it's a long shot but worth trying. If you have serial console attached and if you can influence the boot (which I assume you can as you were able to toggle the verbose boot) as soon as you are presented with FreeBSD boot menu press 3 to escape to boot prompt (OK prompt) and type this

Code:

set hw.use_xsave=0
boot -v

And see if you can boot the machine.

dr. bob · Oct 12, 2022

_martin said:
type this

Code:

set hw.use_xsave=0 boot -v

And see if you can boot the machine.

Martin, thanks. Waiting for it to reboot so I can try that. I feel like you're a wizard giving me a spell to cast!

dr. bob · Oct 12, 2022

alas:

_martin · Oct 12, 2022

It was a long shot.
I was not able to reproduce your setup, I wonder what kind of qemu with hyperv enlightenments is that. But unfortunately without being able to trigger the crash it's practically nothing we can do.

Terri_Kennedy · Oct 13, 2022

dr. bob said:
Terry, thanks. I like the challenge of DIY, but it's definitely slow going. Do the helpful people here ever take work as consultants?

I'm sure some do. You don't even need someone on the same continent as long as the remote console is available.

dr. bob said:
I'm not sure I actually can start networking. I tried ping and had 0% success.

If booting from CD, it doesn't have any knowledge of your network configuration. If you can mount the / filesystem, look in /etc/rc.conf for something like this:

Code:

defaultrouter="a.b.c.d"
ifconfig_vtnet0="e.f.g.h netmask i.j.k.l maybe-options"

where all of the x.x.x.x are IPv4 addresses/masks and maybe-options is any additional stuff for your config (like "mtu 9000"). If there isn't any additional stuff, stop after the "netmask x.x.x.x". You don't type the literal word "maybe-options".
You want to translate those into shell commands so you can issue them interactively, like this:

# ifconfig vtnet0 up (ensure that the interface is initialized and up)
# ifconfig vtnet0 inet e.f.g.h netmask i.j.k.l maybe-options (set the IP address, mask and any options)
# route add default a.b.c.d (tell the system how to get to the outside world)

dr. bob said:
As far as fsck, now I'm wondering if "unexpected soft update inconsistency" is really "corruption". Would the journal (whatever that is) being out of sync with the filesystem (maybe because I told fsck once not to use the journal) be considered a soft update inconsistency? Could the journal be out of sync and the filesystem still be ok? Can I just delete the journal? Would it restart from scratch? Then would there no longer be any inconsistency?

Also, does "file system was modified" necessarily mean fsck found problems and fixed them? I see here and elsewhere that updating the badblock and / or free inode counts, even if unchanged, can trigger that message.

I guess my wishful thinking is that the file system is (now) ok and the reason it can't boot is a new Hyper-V / old FreeBSD incompatibility.

Your system is panicing before it ever tries to mount any filesystems. The following is a very simplified view of the boot process, which has complexities for BIOS vs. UEFI booting, different root partition types and a bunch of other stuff:

The boot loaders have some rudimentary knowledge of filesystems and grab the necessary pieces (kernel, modules, loader config files, etc.) and place them at special memory locations. Then the boot loaders transfer control to the kernel which continues with initialization, etc., mounts the root filesystem (via a hint it got from the boot loader) and eventually ends up by starting the init(8) process in userspace, which then proceeds to bring the system up to the desired state (normally multiuser).

It looks like your system is failing to get even to the point where it starts any additional CPU cores/threads - refer to the "smp_started = 0" in your console output just before the panic.

I think your VM may only have a single non-hyperthreaded CPU core. Using the method _martin mentioned in comment 29, try escaping to the loader prompt as shown in that comment and typing:

Code:

set kern.smp.disabled=1
boot -v

and see if anything changes.

dr. bob · Oct 13, 2022

Terry_Kennedy said:
I'm sure some do. You don't even need someone on the same continent as long as the remote console is available.

Terry, do you? You seem to know what you're talking about. And then this wouldn't be all volunteer.

Or, Martin, how about you?

Terry_Kennedy said:
You want to translate those into shell commands so you can issue them interactively

Super, thanks! ping went from 0% success to 0% packet loss.

Terry_Kennedy said:
Your system is panicing before it ever tries to mount any filesystems.

Right, if I boot from the hard disk. But if I boot from the CD, I can get through to fsck, which if I say to use the journal doesn't seem to find any problems, though does report "file system was modified":

Weirdly (if you ask me), fsck -n, which (as far as I can tell) just says not to use the journal, instead of it deciding to skip the journal, gives me that "unexpected soft update inconsistency":

Terry_Kennedy said:
try escaping to the loader prompt as shown in that comment and typing:

Code:

set kern.smp.disabled=1 boot -v

and see if anything changes.

Alas, it hangs at:

I remember Martin said something about virtio, I don't know if this is related.

dr. bob · Oct 13, 2022

_martin said:
I was not able to reproduce your setup, I wonder what kind of qemu with hyperv enlightenments is that. But unfortunately without being able to trigger the crash it's practically nothing we can do.

Martin, would you like me to ask? I'd be happy to. "What kind of qemu with hyperv enlightenments is this?"

_martin · Oct 13, 2022

dr. bob said:
I guess my wishful thinking is that the file system is (now) ok and the reason it can't boot is a new Hyper-V / old FreeBSD incompatibility.

FS is just a victim of the panic you're getting. That panic is linked to the hyperv, failure in vmbus initialization routine. Every panic your FS gets dirty.

dr. bob said:
Martin, would you like me to ask? I'd be happy to. "What kind of qemu with hyperv enlightenments is this?"

I compiled new qemu from sources on Linux to see if I can get this running I'm right now I'm failing to do so. I would be glad if you ask, I'm really curious how that VM is launched.

dr. bob said:
I remember Martin said something about virtio, I don't know if this is related.

Virtio is not a problem here. It's the acceleration (optimization between host and guest) and initialization procedure.

dr. bob said:
Or, Martin, how about you?

You mean install fresh VM ? I don't have problem with that. But you're trusting with your stuff to somebody from internet. That's always a big risk.

dr. bob · Oct 13, 2022

_martin said:
Every panic your FS gets dirty.

Martin, thanks, I didn't know that, but it makes sense.

_martin said:
I would be glad if you ask, I'm really curious how that VM is launched.

OK, I asked.

_martin said:
You mean install fresh VM ? I don't have problem with that. But you're trusting with your stuff to somebody from internet. That's always a big risk.

Install fresh VM and also get email, web server, etc, running. I've already been trusting you. I've invoked spells I don't understand myself! You seem knowledgeable and have been responsive. What more could I ask for?

dr. bob · Oct 13, 2022

Hi, everyone. I asked for FreeBSD 13.1-RELEASE on a new VM. STABLE would have a later Expected EoL, but would be more experimental, and anyway is n/a?

_martin · Oct 13, 2022

You should stick to the release versions and patch those. 13.1 is the newest right now.

_martin · Oct 13, 2022

I don't know what I was doing last night but this command under Linux qemu-system-x86_64 -cpu host,hv_relaxed,hv_time -accel kvm... is doing what I want. Now it's interesting to see which enlightenments could help here.

_martin · Oct 13, 2022

As I was able to reproduce the bug I was able to attach debugger and debug that boot. I think I can give you a fix. Most likely your provider is using qemu with those enlightenments (I'm not aware of any other hypervisor of being able to do so). If so you need to ask them to add these two: hv_synic,hv-vpindex into the qemu VM specification. They may have others set.

For example part of my testing vm setup:

Code:

qemu-system-x86_64 -cpu host,hv_relaxed,hv_time,hv_synic,hv-vpindex -accel kvm -m 1024 -s \
        -device virtio-blk-pci,id=pci0,drive=hd \
        -drive if=none,id=hd,file=fbsd10.qcow2

dr. bob · Oct 14, 2022

Martin, wow. I asked them to make that change, let's see if they can do that.

In the meantime, they set me up with a new VM with 12.1-RELEASE and a disk image of the old VM. Does it make sense for me to start by upgrading to 13.1-RELEASE?

_martin · Oct 14, 2022

I don't quite understand their reasoning, most likely they don't have available images and there's just not enough demand for them to make them (some sort of local provider I guess). If you can boot CD of your choice (13.1) you can overwrite the 12.1 with the 13.1 fresh installation. I wouldn't bother with rolling upgrades, especially on fresh system.

I'm curious about that fix. I'm not sure if they share how they launch the VM, they may consider that as their know-how. But let's see.

dr. bob · Oct 14, 2022

Martin, thanks. Not sure how I'd boot from a CD of my choice, as opposed to a CD they gave me. Download an ISO image, mount it, and reboot from it??

freebsd-update only updates to the next version? are there so many versions between 12.1 and 13.1?

_martin · Oct 14, 2022

I don't know what options you have with your provider or how you can manage the VM. Yes, idea is to download the iso and mount it. Personally I don't use freebsd-update much ( always used sources), I'd have to google myself to see which versions are available and what upgrade paths are still valid.
But with vanilla image it's easier to install it from rescue mode (new disk layout, extract sources, check fstab, loader.conf, root passwd) and off you go with shiny new 13.1.

Erichans · Oct 14, 2022

dr. bob said:
[...] freebsd-update only updates to the next version? are there so many versions between 12.1 and 13.1?

When going with freebsd-update(8) you can easily go from FreeBSD RELEASE-12.1 to a next major version such as RELEASE-13.1; consult the relevant FreeBSD 13.1-RELEASE Release Notes too.

dr. bob · Oct 16, 2022

OK, getting ready to try updating! Thought it might be a good idea first to see what if any packages were installed? Ran into a version issue:

Code:

# pkg info
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:12:amd64/quarterly, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
Installing pkg-1.18.4...
Newer FreeBSD version for package pkg:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1203000
- running kernel: 1201000
Ignore the mismatch and continue? [y/N]: N

If that pkg assumes 12.3, is it ok to install on 12.1? Thanks!

dr. bob · Oct 16, 2022

took a chance and installed it. didn't seem to be an issue. now running 13.1! ?

dr. bob · Oct 23, 2022

_martin said:
I'm curious about that fix. I'm not sure if they share how they launch the VM, they may consider that as their know-how. But let's see.

Martin, you da bomb! They said "we found the right combination of odd tweaks and settings". I imagine you got them thinking there might actually be a fix + pointed them in the right direction.
How do hv_synic and hv-vpindex make a difference?

_martin · Oct 23, 2022

Oh boy, I should have sent my bill to them then! ;-). Great, then my work was not a waste.

I attached the debugger to live FreeBSD boot and was able to see what's happening. Issue is in a very early boot code when CPU is being identified and set. There are only two reasons why GPF can occur in the case you had. While FreeBSD should not touch that part of the code under VM (hence the PR and fix in FreeBSD 11) qemu is capable of providing many features to guest (has to be on Linux as FreeBSD is missing kvm for it). With debugger I was able to tell what was FreeBSD asking for (even though it should not ask for it at all).

In a bigger picture these are speed and performance optimizations which guest could potentionaly benefit from. Here main reason for them to be enabled is to work with the buggy code of FreeBSD 10.