kernel panic when upgraded to FreeBSD 14.0 releng

B

bvgdas

Hello,
Please help.
I get this picture several times a week.
It was ok when FreeBSD 13.3 had been installed. What is wrong?
custom kernel:


Code: 
cpu             HAMMER
ident           VIDEO

makeoptions     DEBUG=-g                # Build kernel with gdb(1) debug symbols
makeoptions     WITH_CTF=1              # Run ctfconvert(1) for DTrace support

options         SCHED_ULE               # ULE scheduler
options         NUMA                    # Non-Uniform Memory Architecture support
options         PREEMPTION              # Enable kernel thread preemption
options         VIMAGE                  # Subsystem virtualization, e.g. VNET
options         INET                    # InterNETworking
options         INET6                   # IPv6 communications protocols
options     IPSEC           # IP security

options         IPSEC_SUPPORT   # Allow kldload of ipsec and tcpmd5
options         ROUTE_MPATH             # Multipath routing support
options         FIB_ALGO                # Modular fib lookups
options         TCP_OFFLOAD             # TCP offload
options         TCP_BLACKBOX            # Enhanced TCP event logging
options         TCP_HHOOK               # hhook(9) framework for TCP
options         TCP_RFC7413             # TCP Fast Open
options         SCTP_SUPPORT            # Allow kldload of SCTP
options         KERN_TLS                # TLS transmit & receive offload
options         FFS                     # Berkeley Fast Filesystem
options         SOFTUPDATES             # Enable FFS soft updates support
options         UFS_ACL                 # Support for access control lists
options         UFS_DIRHASH             # Improve performance on big directories
options         UFS_GJOURNAL            # Enable gjournal-based UFS journaling
options         QUOTA                   # Enable disk quotas for UFS
options         MD_ROOT                 # MD is a potential root device
options         NFSCL                   # Network Filesystem Client
options         NFSD                    # Network Filesystem Server
options         NFSLOCKD                # Network Lock Manager
options         NFS_ROOT                # NFS usable as /, requires NFSCL
options         MSDOSFS                 # MSDOS Filesystem
options         CD9660                  # ISO 9660 Filesystem
options         PROCFS                  # Process filesystem (requires PSEUDOFS)
options         PSEUDOFS                # Pseudo-filesystem framework
options         TMPFS                   # Efficient memory filesystem
options         GEOM_RAID               # Soft RAID functionality.
options         GEOM_LABEL              # Provides labelization
options         EFIRT                   # EFI Runtime Services support

options         COMPAT_FREEBSD11        # Compatible with FreeBSD11
options         COMPAT_FREEBSD12        # Compatible with FreeBSD12
options         SCSI_DELAY=5000         # Delay (in ms) before probing SCSI
options         KTRACE                  # ktrace(1) support
options         STACK                   # stack(9) support
options         SYSVSHM                 # SYSV-style shared memory
options         SYSVMSG                 # SYSV-style message queues
options         SYSVSEM                 # SYSV-style semaphores
options         _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options         PRINTF_BUFR_SIZE=128    # Prevent printf output being interspersed.
options         KBD_INSTALL_CDEV        # install a CDEV entry in /dev
options         HWPMC_HOOKS             # Necessary kernel hooks for hwpmc(4)
options         AUDIT                   # Security event auditing
options         CAPABILITY_MODE         # Capsicum capability mode
options         CAPABILITIES            # Capsicum capabilities
options         MAC                     # TrustedBSD MAC Framework
options         KDTRACE_FRAME           # Ensure frames are compiled in
options         KDTRACE_HOOKS           # Kernel DTrace hooks
options         DDB_CTF                 # Kernel ELF linker loads CTF data
options         INCLUDE_CONFIG_FILE     # Include this file in kernel
options         RACCT                   # Resource accounting framework
options         RACCT_DEFAULT_TO_DISABLED # Set kern.racct.enable=0 by default
options         RCTL                    # Resource limits

# Debugging support.  Always need this:
options         KDB                     # Enable kernel debugger support.
options         KDB_TRACE               # Print a stack trace for a panic.
# For full debugger support use (turn off in stable branch):
options         BUF_TRACKING            # Track buffer history
options         DDB                     # Support DDB.
options         FULL_BUF_TRACKING       # Track more buffer history
options         GDB                     # Support remote GDB.
options         DEADLKRES               # Enable the deadlock resolver
options         INVARIANTS              # Enable calls of extra sanity checking
options         INVARIANT_SUPPORT       # Extra sanity checks of internal structures, required by INVARIANTS
options         QUEUE_MACRO_DEBUG_TRASH # Trash queue(2) internal pointers on invalidation


options         MALLOC_DEBUG_MAXZONES=8 # Separate malloc(9) zones
options         VERBOSE_SYSINIT=0       # Support debug.verbose_sysinit, off by default

# Kernel dump features.
options         EKCD                    # Support for encrypted kernel dumps
options         GZIO                    # gzip-compressed kernel and user dumps
options         ZSTDIO                  # zstd-compressed kernel and user dumps
options         DEBUGNET                # debugnet networking
options         NETDUMP                 # netdump(4) client support
options         NETGDB                  # netgdb(4) client support

# Make an SMP-capable kernel by default
options         SMP                     # Symmetric MultiProcessor Kernel
options         EARLY_AP_STARTUP

# CPU frequency control
device          cpufreq

# Bus support.
device          acpi
device          smbios
options         IOMMU
device          pci
options         PCI_HP                  # PCI-Express native HotPlug
options         PCI_IOV                 # PCI SR-IOV support

options         COMPAT_LINUXKPI

# Enable support for the kernel PLL to use an external PPS signal,
# under supervision of [x]ntpd(8)
# More info in ntpd documentation: http://www.eecis.udel.edu/~ntp

options         PPS_SYNC

# Floppy drives
#device         fdc

# ATA controllers
device          ahci                    # AHCI-compatible SATA controllers
device          ata                     # Legacy ATA/SATA controllers
device          mvs                     # Marvell 88SX50XX/88SX60XX/88SX70XX/SoC SATA
#device         siis                    # SiliconImage SiI3124/SiI3132/SiI3531 SATA

# SCSI Controllers
#device         ahc                     # AHA2940 and onboard AIC7xxx devices
#device         ahd                     # AHA39320/29320 and onboard AIC79xx devices
#device         hptiop                  # Highpoint RocketRaid 3xxx series
#device         isp                     # Qlogic family
##device                ispfw                   # Firmware for QLogic HBAs- normally a module
#device         mpt                     # LSI-Logic MPT-Fusion
#device         mps                     # LSI-Logic MPT-Fusion 2
#device         mpr                     # LSI-Logic MPT-Fusion 3
#device         sym                     # NCR/Symbios Logic
#device         isci                    # Intel C600 SAS controller
#device         ocs_fc                  # Emulex FC adapters
#device         pvscsi                  # VMware PVSCSI

# ATA/SCSI peripherals
device          scbus                   # SCSI bus (required for ATA/SCSI)
device          ch                      # SCSI media changers
device          da                      # Direct Access (disks)
device          sa                      # Sequential Access (tape etc)
device          cd                      # CD
device          pass                    # Passthrough device (direct ATA/SCSI access)
device          ses                     # Enclosure Services (SES and SAF-TE)
#device         ctl                     # CAM Target Layer

# RAID controllers interfaced to the SCSI subsystem
device          arcmsr                  # Areca SATA II RAID
device          ciss                    # Compaq Smart RAID 5*
device          ips                     # IBM (Adaptec) ServeRAID
device          smartpqi                # Microsemi smartpqi driver
device          tws                     # LSI 3ware 9750 SATA+SAS 6Gb/s RAID controller

# RAID controllers
device          aac                     # Adaptec FSA RAID
device          aacp                    # SCSI passthrough for aac (requires CAM)
device          aacraid                 # Adaptec by PMC RAID

# NVM Express (NVMe) support
device          nvme                    # base NVMe driver
device          nvd                     # expose NVMe namespaces as disks, depends on nvme

# Intel Volume Management Device (VMD) support
device          vmd

# atkbdc0 controls both the keyboard and the PS/2 mouse
device          atkbdc                  # AT keyboard controller
device          atkbd                   # AT keyboard
#device         psm                     # PS/2 mouse

device          kbdmux                  # keyboard multiplexer

# syscons is the legacy console driver, resembling an SCO console
#device         vga                     # VGA video card driver
device          splash                  # Splash screen and screen saver support
#device         sc
#options        SC_PIXEL_MODE           # add support for the raster text mode

# vt is the default video console driver
device          vt
device          vt_vga
device          vt_efifb
device          vt_vbefb

#device         agp                     # support several AGP chipsets

# CardBus bridge support
device          cbb                     # CardBus (yenta) bridge
device          cardbus                 # CardBus (32-bit) bus

# Serial (COM) ports
device          uart                    # Generic UART driver

# Parallel port
device          ppc
device          ppbus                   # Parallel port bus (required)
device          lpt                     # Printer
device          ppi                     # Parallel port interface device
#device         vpo                     # Requires scbus and da

device          puc                     # Multi I/O cards and multi-channel UARTs

# PCI/PCI-X/PCIe Ethernet NICs that use iflib infrastructure
device          iflib
device          em                      # Intel PRO/1000 Gigabit Ethernet Family
device          ix                      # Intel PRO/10GbE PCIE PF Ethernet
device          ixv                     # Intel PRO/10GbE PCIE VF Ethernet
device          ixl                     # Intel 700 Series Physical Function
device          iavf                    # Intel Adaptive Virtual Function
device          ice                     # Intel 800 Series Physical Function

# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device          miibus                  # MII bus support
device          dc                      # DEC/Intel 21143 and various workalikes
device          fxp                     # Intel EtherExpress PRO/100B (82557, 82558)

# ZFS support
# # NB: This depends on crypto, cryptodev and ZSTDIO
options         ZFS

# Pseudo devices.
device          crypto                  # core crypto support
device          cryptodev
device          aesni                   # AES-NI OpenCrypto module
device          loop                    # Network loopback
device          padlock_rng             # VIA Padlock RNG
device          rdrand_rng              # Intel Bull Mountain RNG
device          ether                   # Ethernet support
device          vlan                    # 802.1Q VLAN support
device          tuntap                  # Packet tunnel.
device          md                      # Memory "disks"
device          gif                     # IPv6 and IPv4 tunneling
device          firmware                # firmware assist module
device          xz                      # lzma decompression

# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device          bpf                     # Berkeley packet filter

# USB support
options         USB_DEBUG               # enable debug msgs
device          uhci                    # UHCI PCI->USB interface
device          ohci                    # OHCI PCI->USB interface
device          ehci                    # EHCI PCI->USB interface (USB 2.0)
device          xhci                    # XHCI PCI->USB interface (USB 3.0)
device          usb                     # USB Bus (required)
device          ukbd                    # Keyboard
device          umass                   # Disks/Mass storage - Requires scbus and da

# Sound support
device          sound
device          snd_hda                 # Intel High Definition Audio

# MMC/SD
device          mmc                     # MMC/SD bus
device          mmcsd                   # MMC/SD memory card
device          sdhci                   # Generic PCI SD Host Controller
device          rtsx                    # Realtek SD card reader

# VirtIO support
device          virtio                  # Generic VirtIO bus (required)
device          virtio_pci              # VirtIO PCI device
device          vtnet                   # VirtIO Ethernet device
device          virtio_blk              # VirtIO Block device
device          virtio_scsi             # VirtIO SCSI device
device          virtio_balloon          # VirtIO Memory Balloon device

# HyperV drivers and enhancement support
device          hyperv                  # HyperV drivers

# Netmap provides direct access to TX/RX rings on supported NICs
device          netmap                  # netmap(4) support

# evdev interface
options         EVDEV_SUPPORT           # evdev support in legacy drivers
device          evdev                   # input event device support
device          uinput                  # install /dev/uinput cdev

# HID support
options         HID_DEBUG               # enable debug msgs
device          hid                     # Generic HID support
options         IICHID_SAMPLING         # Workaround missing GPIO INTR support



options    IPFIREWALL                   # enables IPFW
options    IPFIREWALL_VERBOSE           # enables logging for rules with log keyword to syslogd(8)
options    IPFIREWALL_VERBOSE_LIMIT=5   # limits number of logged packets per-entry
options    IPFIREWALL_DEFAULT_TO_ACCEPT # sets default policy to pass what is not explicitly denied
options    IPFIREWALL_NAT               # enables basic in-kernel NAT support
options    LIBALIAS                     # enables full in-kernel NAT support
options    IPFIREWALL_NAT64             # enables in-kernel NAT64 support
options    IPFIREWALL_NPTV6             # enables in-kernel IPv6 NPT support
options    IPFIREWALL_PMOD              # enables protocols modification module support
options    IPDIVERT                     # enables NAT through natd(8)



#
# Temperature sensors:
#
# coretemp: on-die sensor on Intel Core and newer CPUs
# amdtemp: on-die sensor on AMD K8/K10/K11 CPUs
#
device          coretemp
device          amdtemp

#
# CPU control pseudo-device. Provides access to MSRs, CPUID info and
# microcode update feature.
#
device          cpuctl

#
# SuperIO driver.
#
device          superio


Screenshot_66.png

Screenshot_65.png
 
Why do you have a custom kernel in the first place? I would suggest trying the standard GENERIC kernel and see how that works.
 
First of all, kernel config should be better something like /usr/src/sys/amd64/conf/GENERIC-MMCCAM does.
This way, one can focus on differences against included one, and differences between previous and current versions of included one. Your old-style config forces everyone to check line-by-line, thus difficult and easy to overlook.
Note that you can unset already-set options by nooptions and disable already-enabled devices by nodevice.

And are you loading anything via loader.conf? If you are attempting to load too large modules which makes loader's staging area to overflow, thus making undefined behaviour.
See this thread and this post for detail.
 
/boot/loader.conf

Code: 
zfs_load="YES"
zpool_cache_load="YES"
zpool_cache_name="/boot/zfs/zpool.cache"
zpool_cache_type="/boot/zfs/zpool.cache"
vfs.root.mountfrom="zfs:zsys/ROOT/default"
 
SirDice said:
Why do you have a custom kernel in the first place? I would suggest trying the standard GENERIC kernel and see how that works.
Click to expand...
I have custom kernel because I do not need 90% of drivers mentioned in GENERIC and I need IPFW options.
 
bvgdas said:
/boot/loader.conf

Code: 
zfs_load="YES"
zpool_cache_load="YES"
zpool_cache_name="/boot/zfs/zpool.cache"
zpool_cache_type="/boot/zfs/zpool.cache"
vfs.root.mountfrom="zfs:zsys/ROOT/default"
Click to expand...
You don't seem to be using geli(8), so mid 3 lines (zpool_cache_* lines) wouldn't be needed. Not sure it hurts or not without geli(8), I don't have them in my /boot/loader.conf both on stable/14 and main (aka -current).

For your info: (rg is textproc/ripgrep)
% rg -n -A 10 -B 10 "zpool_cache_name" /usr/src/
/usr/src/usr.sbin/bsdinstall/scripts/zfsboot
1546- # Set cachefile for boot pool so it auto-imports at system start
1547- f_dprintf "$funcname: Configuring zpool.cache for boot pool..."
1548- f_eval_catch $funcname zpool "$ZPOOL_SET" \
1549- "cachefile=\"$BSDINSTALL_CHROOT/boot/zfs/zpool.cache\"" \
1550- "$bootpool_name" || return $FAILURE
1551-
1552- # Some additional geli(8) requirements for loader.conf(5)
1553- for option in \
1554- 'zpool_cache_load=\"YES\"' \
1555- 'zpool_cache_type=\"/boot/zfs/zpool.cache\"' \
1556: 'zpool_cache_name=\"/boot/zfs/zpool.cache\"' \
1557- 'geom_eli_passphrase_prompt=\"YES\"' \
1558- ; do
1559- f_eval_catch $funcname echo "$ECHO_APPEND" "$option" \
1560- $BSDINSTALL_TMPBOOT/loader.conf.zfs ||
1561- return $FAILURE
1562- done
1563- return $SUCCESS
1564-}
1565-
1566-# dialog_menu_diskinfo
 
bvgdas said:
I have custom kernel because I do not need 90% of drivers mentioned in GENERIC and I need IPFW options.
Click to expand...
So maybe you would be better compare your kernel conig with MINIMAL.
Then, whichever
  • Add missing lines which MINIMAL has to your config
  • Create new config including MINIMAL and add missing-in-MINIMAL lines
could help.
MINIMAL is the config file which removed known-to-be-safe--to-delete lines from GENERIC.
In other words, ANYTHING IN MINIMAL ARE KNOWN TO BE UNSAFE TO DELETE (or at least not enough tested) FOR SMP SYSTEMS.
For uni-processor systems, 2 more lines are known to be safe to delete.
MINIMALUP is for the configuration.

And your backtrace has ip_mfilter_free() and inp_freemoptions() in it, so not enough sure but possibly something related with network are missing or mis-configured in your kernek config, or using unstable driver options are specified in /etc/rc.conf, or specifying something inappropreate in your IPFW configurations.
 
bvgdas, have you copy&pasted the correct kernel config file?

I did a test build with your kernel config on a 14.0-RELEASE source, the build failed due to missing device sound, but ended successful after adding the directive.

The built 14.0-RELEASE VIDEO kernel is running just fine in a 14.0-RELEASE VM.

bvgdas said:
vfs.root.mountfrom="zfs:zsys/ROOT/default"
Click to expand...
Do you have multiple Root-on-ZFS installations? The above bootstrap configuration is only needed if you want to boot an other pool than the pool where the kernel resides.
 
T-Aoki said:
You don't seem to be using geli(8), so mid 3 lines (zpool_cache_* lines) wouldn't be needed. Not sure it hurts or not without geli(8), I don't have them in my /boot/loader.conf both on stable/14 and main (aka -current).
Click to expand...
thank you for advice. I'll try to remove it from `loader.conf`
 
T-Daemon said:
I did a test build with your kernel config on a 14.0-RELEASE source, the build failed due to missing device sound, but ended successful after adding the directive.
Click to expand...
yes. I accidentally removed this line when was clearing file from comments. sound device is present in the config of course.

T-Daemon said:
Do you have multiple Root-on-ZFS installations? The above bootstrap configuration is only needed if you want to boot an other pool than the pool where the kernel resides.
Click to expand...
thank you, I see and will try to remove it.

I've compared GENERIC and my VIDEO configs with diff.

uncommented these lines:
Code: 
options         COMPAT_FREEBSD13

# syscons is the legacy console driver, resembling an SCO console
device      vga         # VGA video card driver
device      splash          # Splash screen and screen saver support
device      sc
options     SC_PIXEL_MODE       # add support for the raster text mode

Perhaps these changes will help to avoid kernel panic. I'll see tomorrow for server working.

p.s. thank you very much for all for comments. 🙏
 
bvgdas said:
I've compared GENERIC and my VIDEO configs with diff.
Click to expand...
If you create a custom kernel configuration you should take T-Aoki 's advice. Simply include GENERIC (or MINIMAL) and work from there.


An include directive is available for use in configuration files. This allows another configuration file to be included in the current one, making it easy to maintain small changes relative to an existing file. If only a small number of additional options or drivers are required, this allows a delta to be maintained with respect to GENERIC, as seen in this example:
Code: 
include GENERIC
ident MYKERNEL

options         IPFIREWALL
options         DUMMYNET
options         IPFIREWALL_DEFAULT_TO_ACCEPT
options         IPDIVERT
Click to expand...
docs.freebsd.org

Chapter 10. Configuring the FreeBSD Kernel

This chapter covers how to configure the FreeBSD Kernel. When to build a custom kernel, how to take a hardware inventory, how to customize a kernel configuration file, etc
docs.freebsd.org docs.freebsd.org
 
bvgdas said:
ok. I'll rewrite config.
Click to expand...
To make it easier to maintain in the future, you should choose for baseline to include
  • GENERIC, if you have a small amount of options/devices to disable from,
  • MINIMAL, if you have little to add to.
The smaller your config is, the better the maintainability is.
 
created 2 configs. It's too many devices for exclude with included GENERIC. When new useless entries appear in GENERIC I won't exclude them. It's too many options and devices included with included MINIMAL.
There is no point to use inclusion any predefined configurations. It's faster to use coloured diff to compare GENERIC with custom config as I did it before after switching to new version.
 
Kernel panics stopped. I guess it is cause. This line was commented
Code: 
options         COMPAT_FREEBSD13

Thanks for all for help.
 
You must log in or register to reply here.
Back
Top