Kernel upgrade from 12.2-STABLE to 12.4-RELEASE, sshd "could not bind"

E

escape

Compiling new kernel 12.4-RELEASE made most of the network applications to stop in reboot, at least sshd. The error was:

Code: 
Aug 27 15:48:00 my root[1088]: /etc/rc: WARNING: failed to start dhcpd
Aug 27 15:48:01 my sshd[1135]: error: Bind to port 22 on 192.168.yy.y failed: Can't assign requested address.
Aug 27 15:48:01 my sshd[1135]: error: Bind to port 22 on 192.168.xx.x failed: Can't assign requested address.
Aug 27 15:48:01 my sshd[1135]: fatal: Cannot bind any address.

After these lines the kernel announced the network interfaces. Could not bind?

After setting sshd last in rc.conf no improvement happened. What is wrong and how can I fix this?

best, esc
 
Can you provide more details?

1) output of the ifconfig command
2) output of dmesgrelated to network interfaces
3) /etc/rc.conf
4) /etc/ssh/sshd_config - options like "Port", "AddressFamily", "ListenAddress"
 
Do you think it could be the make.conf (last lines)?

/var/log/messages including text witness_debugger:

Code: 
# cat fbsd_f.log
Aug 27 16:50:18 my kernel: ---<<BOOT>>---
Aug 27 16:50:18 my kernel: Copyright (c) 1992-2021 The FreeBSD Project.
Aug 27 16:50:18 my kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Aug 27 16:50:18 my kernel:     The Regents of the University of California. All rights reserved.
Aug 27 16:50:18 my kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
Aug 27 16:50:18 my kernel: FreeBSD 12.4-RELEASE-p4 releng/12.4-595be1fc5 NOTEBOOK amd64
Aug 27 16:50:18 my kernel: FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)
Aug 27 16:50:18 my kernel: WARNING: WITNESS option enabled, expect reduced performance.
Aug 27 16:50:18 my kernel: VT(efifb): resolution 800x600
Aug 27 16:50:18 my kernel: CPU: Intel(R)
...
Aug 27 16:50:18 my kernel: pci1: <network, ethernet> at device 0.0 (no driver attached)
Aug 27 16:50:18 my kernel: pcib2: <ACPI PCI-PCI bridge> at device 19.1 on pci0
Aug 27 16:50:18 my kernel: pci2: <ACPI PCI bus> on pcib2
Aug 27 16:50:18 my kernel: pci2: <network, ethernet> at device 0.0 (no driver attached)
Aug 27 16:50:18 my kernel: pcib3: <ACPI PCI-PCI bridge> at device 19.2 on pci0
Aug 27 16:50:18 my kernel: pci3: <ACPI PCI bus> on pcib3
Aug 27 16:50:18 my kernel: pci3: <network, ethernet> at device 0.0 (no driver attached)
Aug 27 16:50:18 my kernel: pcib4: <ACPI PCI-PCI bridge> at device 19.3 on pci0
Aug 27 16:50:18 my kernel: pci4: <ACPI PCI bus> on pcib4
Aug 27 16:50:18 my kernel: pci4: <network, ethernet> at device 0.0 (no driver attached)
Aug 27 16:50:18 my kernel: xhci0: <Intel ... > mem ...
Aug 27 16:50:18 my kernel: xhci0: 32 bytes context size, 64-bit DMA
Aug 27 16:50:18 my kernel: usbus0 on xhci0
Aug 27 16:50:18 my kernel: usbus0: 5.0Gbps Super Speed USB v3.0
...
Aug 27 16:50:18 my kernel: Security policy loaded: MAC/ntpd (mac_ntpd)
Aug 27 16:50:18 my ntpd[968]: ntpd 4.2.8p15-a (1): Starting
Aug 27 16:50:18 my ntpd[968]: Command line: /usr/sbin/ntpd -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -f /var/db/ntp/ntpd.drift
Aug 27 16:50:18 my ntpd[968]: ----------------------------------------------------
Aug 27 16:50:18 my ntpd[968]: ntp-4 is maintained by Network Time Foundation,
Aug 27 16:50:18 my ntpd[968]: Inc. (NTF), a non-profit 501(c)(3) public-benefit
Aug 27 16:50:18 my ntpd[968]: corporation.  Support and training for ntp-4 are
Aug 27 16:50:18 my ntpd[968]: available at https://www.nwtime.org/support
Aug 27 16:50:18 my ntpd[968]: ----------------------------------------------------
Aug 27 16:50:18 my ntpd[969]: leapsecond file ('/var/db/ntpd.leap-seconds.list'): good hash signature
Aug 27 16:50:18 my ntpd[969]: leapsecond file ('/var/db/ntpd.leap-seconds.list'): loaded, expire=2023-12-28T00:00:00Z last=2017-01-01T00:00:00Z ofs=37
Aug 27 16:50:18 my kernel: lock order reversal:
Aug 27 16:50:18 my kernel:  1st 0xfffff800552f7428 ufs (ufs) @ /usr/src/sys/kern/vfs_mount.c:930
Aug 27 16:50:18 my kernel:  2nd 0xfffff800552f7068 devfs (devfs) @ /usr/src/sys/kern/vfs_mount.c:941
Aug 27 16:50:18 my kernel: stack backtrace:
Aug 27 16:50:18 my kernel: #0 0xffffffff80804861 at witness_debugger+0x71
Aug 27 16:50:18 my kernel: #1 0xffffffff80770c7d at lockmgr_lock_fast_path+0x18d
Aug 27 16:50:18 my kernel: #2 0xffffffff80c8e980 at VOP_LOCK1_APV+0xb0
Aug 27 16:50:18 my kernel: #3 0xffffffff8087fdb5 at _vn_lock+0x65
Aug 27 16:50:18 my kernel: #4 0xffffffff808656a8 at vfs_domount+0xdc8
Aug 27 16:50:18 my kernel: #5 0xffffffff80864317 at vfs_donmount+0x7d7
Aug 27 16:50:18 my kernel: #6 0xffffffff80863b11 at sys_nmount+0x71
Aug 27 16:50:18 my kernel: #7 0xffffffff80bac12e at amd64_syscall+0x2be
Aug 27 16:50:18 my kernel: #8 0xffffffff80b82c3e at fast_syscall_common+0xf8
Aug 27 16:50:18 my dhcpd[1063]: WARNING: Host declarations are global.  They are not limited to the scope you declared them in.
...
Aug 27 16:50:18 my dhcpd[1063]: Not configured to listen on any interfaces!
...
Aug 27 16:50:18 my dhcpd[1063]:
Aug 27 16:50:18 my dhcpd[1063]: If you think you have received this message due to a bug rather
Aug 27 16:50:18 my dhcpd[1063]: than a configuration issue please read the section on submitting
Aug 27 16:50:18 my dhcpd[1063]: bugs on either our web page at www.isc.org or in the README file
Aug 27 16:50:18 my dhcpd[1063]: before submitting a bug.  These pages explain the proper
Aug 27 16:50:18 my dhcpd[1063]: process and the information we find helpful for debugging.
Aug 27 16:50:18 my dhcpd[1063]:
Aug 27 16:50:18 my dhcpd[1063]: exiting.
Aug 27 16:50:18 my root[1064]: /etc/rc: WARNING: failed to start dhcpd
Aug 27 16:50:18 my sshd[1111]: error: Bind to port 22 on <IP of real IF> failed: Can't assign requested address.
Aug 27 16:50:18 my kernel: .
Aug 27 16:50:18 my sshd[1111]: error: Bind to port 22 on <IP of second real IF> failed: Can't assign requested address.
Aug 27 16:50:18 my sshd[1111]: fatal: Cannot bind any address.
Aug 27 16:50:19 my kernel: igb0: <Intel(R) I211 (Copper)> port 0xe000-0xe01f mem 0xc1300000-0xc131ffff,0xc1320000-0xc1323fff at device 0.0 on pci1
Aug 27 16:50:19 my kernel: igb0: NVM V0.4 imgtype1
Aug 27 16:50:19 my kernel: igb0: Using 1024 TX descriptors and 1024 RX descriptors
Aug 27 16:50:19 my kernel: igb0: Using 2 RX queues 2 TX queues
Aug 27 16:50:19 my kernel: igb0: Using MSI-X interrupts with 3 vectors
...
Aug 27 16:50:19 my kernel: igb1: <Intel(R) I211 (Copper)> port 0xd000-0xd01f mem 0xc1200000-0xc121ffff,0xc1220000-0xc1223fff at device 0.0 on pci2
Aug 27 16:50:19 my kernel: igb1: NVM V0.4 imgtype1
Aug 27 16:50:19 my kernel: igb1: Using 1024 TX descriptors and 1024 RX descriptors
Aug 27 16:50:19 my kernel: igb1: Using 2 RX queues 2 TX queues
Aug 27 16:50:19 my kernel: igb1: Using MSI-X interrupts with 3 vectors
...
Aug 27 16:50:19 my root[1203]: /etc/rc: WARNING: failed to start apache24
Aug 27 16:50:19 my kernel: igb2: <Intel(R) I211 (Copper)> port 0xc000-0xc01f mem 0xc1100000-0xc111ffff,0xc1120000-0xc1123fff at device 0.0 on pci3
Aug 27 16:50:19 my kernel: igb2: NVM V0.4 imgtype1
Aug 27 16:50:19 my kernel: igb2: Using 1024 TX descriptors and 1024 RX descriptors
Aug 27 16:50:19 my kernel: igb2: Using 2 RX queues 2 TX queues
Aug 27 16:50:19 my kernel: igb2: Using MSI-X interrupts with 3 vectors
...
Aug 27 16:50:19 my kernel: igb3: <Intel(R) I211 (Copper)> port 0xb000-0xb01f mem 0xc1000000-0xc101ffff,0xc1020000-0xc1023fff at device 0.0 on pci4
Aug 27 16:50:19 my kernel: igb3: NVM V0.4 imgtype1
Aug 27 16:50:19 my kernel: igb3: Using 1024 TX descriptors and 1024 RX descriptors
Aug 27 16:50:19 my kernel: igb3: Using 2 RX queues 2 TX queues
Aug 27 16:50:19 my kernel: igb3: Using MSI-X interrupts with 3 vectors
...
Aug 27 16:50:22 my kernel: vlan0:
...

There are multiple VLAN:s and multiple real interfaces. In boot rc.conf should load them like:
Code: 
ifconfig_igb0="inet <ip> netmask <mask> descr MYNET"
...
vlans_igb2="NAME"
create_args_NAME="vlan 14"
...
dhcpd_enable="YES"
...
sshd_enable="YES"

Later found that the kernel was compiled using
Code: 
$ cat /etc/make.conf
EMUL_32BIT="1"
 
Last edited by a moderator:
escape said:
error: Bind to port 22 on <IP of real IF> failed: Can't assign requested address.
Click to expand...
Your sshd configuration has configured to listen some specified IP(s).
Check the "ListenAddress" in the sshd_config
and check the output of ifconfig
You should have configured IPs specified in the sshd_config
 
escape said:
Aug 27 16:50:18 my kernel: 1st 0xfffff800552f7428 ufs (ufs) @ /usr/src/sys/kern/vfs_mount.c:930 Aug 27 16:50:18 my kernel: 2nd 0xfffff800552f7068 devfs (devfs) @ /usr/src/sys/kern/vfs_mount.c:941 Aug 27 16:50:18 my kernel: stack backtrace: Aug 27 16:50:18 my kernel: #0 0xffffffff80804861 at witness_debugger+0x71 Aug 27 16:50:18 my kernel: #1 0xffffffff80770c7d at lockmgr_lock_fast_path+0x18d Aug 27 16:50:18 my kernel: #2 0xffffffff80c8e980 at VOP_LOCK1_APV+0xb0 Aug 27 16:50:18 my kernel: #3 0xffffffff8087fdb5 at _vn_lock+0x65 Aug 27 16:50:18 my kernel: #4 0xffffffff808656a8 at vfs_domount+0xdc8 Aug 27 16:50:18 my kernel: #5 0xffffffff80864317 at vfs_donmount+0x7d7 Aug 27 16:50:18 my kernel: #6 0xffffffff80863b11 at sys_nmount+0x71 Aug 27 16:50:18 my kernel: #7 0xffffffff80bac12e at amd64_syscall+0x2be Aug 27 16:50:18 my kernel: #8 0xffffffff80b82c3e at fast_syscall_common+0xf8
Click to expand...
This lines look like kernel panic in the some kernel modules.
 
On what interface are those two addresses bound to?
Code: 
Aug 27 16:50:18 my sshd[1111]: error: Bind to port 22 on <IP of real IF> failed: Can't assign requested address.
Aug 27 16:50:18 my kernel: .
Aug 27 16:50:18 my sshd[1111]: error: Bind to port 22 on <IP of second real IF> failed: Can't assign requested address.

Are they the same interface(s) and IP addresses where dhcpd and apache are configured on? Those fail to start too. I would suggest trying the GENERIC kernel, just to rule out any issues with your custom kernel configuration.
 
Are they the same interface(s) and IP addresses where dhcpd and apache are configured on?
Click to expand...
Yes. There are only the two LAN IP-addresses (the ports) in the sshd_config. The dhcpd has other VLAN:s in addition. It looks like the sshd can be restarted a time after bootup when the interfaces are there already. When it tries to start the interfaces are not there. The previous kernel was 12.2-STABLE r369603 GENERIC where the rc.conf order worked ok.

This lines look like kernel panic in the some kernel modules.
Click to expand...
There were multiple debug options enabled in the kernel config. The lines look like filesystem error messages.

I think I have updated an old 13.1 to 13.4-RELEASE kernel from source before trying to update this one and I had to change the order of startup sections in rc.conf to repair the update. This fixed the problem if I remember correctly. This did not help in the version 12. The computer was different.

Thanks for the help.
 
I have an Ethernet interface that's slow to come up, have used /etc/rc.d/netwait to fix the problem:
Code: 
[sherman.133] $ grep netwait_ /etc/rc.conf
netwait_enable="YES"
netwait_timeout="5"
netwait_ip="${defaultrouter}"
netwait_if="ue0" # ASIX Electronics Corp. AX88179 Gigabit Ethernet
 
/etc/rc.d/netwait helped. dmesg was clear of errors after booting 12.4-RELEASE-p4 releng/12.4-595be1fc5. Thanks.
 
You must log in or register to reply here.
Back
Top