FreeBSD 13.2 + OpenSSH 9.3 = 1 connection limit

tnpimatt

tnpimatt

I have a small fleet of FreeBSD hosts and all are nearly identical.

Code: 
# uname -a
FreeBSD ****.tnpi.net 13.2-RELEASE FreeBSD 13.2-RELEASE releng/13.2-n254617-525ecfdad597 GENERIC amd64

Prior to upgrading to FreeBSD 13.2, I regularly open multiple concurrent ssh sessions to the servers and it has always worked. I authenticate ONLY with a password protected ssh key. The only changes in my sshd_config are to set ListenAddress (to constrain the bind to a few IPs) and PermitRootLogin without-password. After upgrading to 13.2, after I've connected with one session, subsequent concurrent ssh sessions hang forever. If I disconnect the first session, then the subsequent session will log in and work as expected. This behavior persists regardless of the user I log in as. I have tried connecting from different IP addresses and it makes no difference.

If, while connected with the first ssh session, I restart sshd with service sshd restart, then I can connect with another session. Just as often, the sshd process will wait interminably, apparently unwilling to exit while a ssh session is open. If I background the restart process and send it a kill -9, then sshd dutifully exits and then I can open a concurrent session.

Here's what it looks like on the client side when a concurrent connection is made:

Code: 
$ ssh -vv ****.tnpi.net
OpenSSH_8.8p1, OpenSSL 1.1.1o-freebsd  3 May 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolving "****.tnpi.net" port 22
debug1: Connecting to ****.tnpi.net [2605:7900:**:**::**] port 22.
debug1: Connection established.
debug1: identity file /home/matt/.ssh/id_rsa type 0
debug1: identity file /home/matt/.ssh/id_rsa-cert type -1
debug1: identity file /home/matt/.ssh/id_dsa type -1
debug1: identity file /home/matt/.ssh/id_dsa-cert type -1
debug1: identity file /home/matt/.ssh/id_ecdsa type -1
debug1: identity file /home/matt/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/matt/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/matt/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/matt/.ssh/id_ed25519 type -1
debug1: identity file /home/matt/.ssh/id_ed25519-cert type -1
debug1: identity file /home/matt/.ssh/id_ed25519_sk type -1
debug1: identity file /home/matt/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/matt/.ssh/id_xmss type -1
debug1: identity file /home/matt/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.8 FreeBSD-20211221
<insert eery silence>

It stays stuck there until the first ssh session is dropped. I tried running sshd with LogLevel DEBUG and DEBUG3. When the first session connects, there's a firehose of messages. Nothing at all is logged when the subsequent session connects and stalls. The behavior is the same when connecting over IPv4 and IPv6. I have upgraded 4 machines to 13.2 and 3 of 4 are behaving this way.

Anyone got any ideas?
 
Definitely updated userland?

Tried restarting sshd (looks like a yes) or rebooting?

You‘ve definitely missed a step somewhere - I’ve upgraded a few machines to 13.2 and not seen this.

13.2 updated sshd so check OpenSSH 9.2 release notes.
 
Definitely working for me - more than one connection, keys-only:
Code: 
$ ssh -vv somehost.somewhere
OpenSSH_9.3, LibreSSL 3.7.2
debug1: Reading configuration data /etc/ssh/ssh_config...

All the way through to a prompt. Opened another terminal window, no issues logging-in there.

On a side-note looks like the 13.2 notes are wrong - they say OpenSSH 9.2 but looks like it's actually 9.3 (as you said)
 
Yes, OpenSSH_8.8p1 is the client side (mac osx), which really ought not matter.

Server side is as reported. Since sshd is part of the userland, I wouldn't have this newer sshd version without an updated userland, right?. And yes, reboots after updating kernel & userland were done.

Code: 
# sshd -V
OpenSSH_9.3, OpenSSL 1.1.1t-freebsd  7 Feb 2023

On a FreeBSD 13.1 host, it looks like this:

Code: 
# sshd -V
sshd: illegal option -- V
OpenSSH_8.8p1, OpenSSL 1.1.1o-freebsd  3 May 2022

The problem is not connecting FROM a FreeBSD 13.2 host, it's in connecting TO a host running 13.2.
 
Yes looks OK from the server update side of things, especially if you rebooted after userland updates.

I‘m doing the same thing - multiple clients to 13.2 hosts SSH with keys not passwords and working for me.

Almost like you’ve got it in some one client only mode.
 
VladiBG said:
Is there firewall involved?
Click to expand...
Yes, PF is running on the hosts before and after upgrading them to 13.2. So too is tcpwrappers, but neither seems likely to be related because the client connects to sshd (see the Connection established" line in the ssh client output above. Right?
 
VladiBG's question led me thinking about tcpwrappers. It turns out that when I disable the sshd lines in /etc/hosts.allow, the problem goes away. It seems that tcpwrappers gained a new undocumented throttling feature.
 
I recently upgraded from 13.1 to 13.2 using binary upgrade followed all steps (I have done this many times). I face too the same behaviour. What I discovered is that in /etc/hosts.allow it makes no diffeence either to have ssh lines or not. The client just waits forever.

But what I discovered is that if the sshd service is run from inetd, then there is no delay no matter how many connections I make have. If I disabled the inetd ssh line and run it standalone (/etc/rc.d/sshd onerestart) then I do have the delay.

FreeBSD 13.2-RELEASE-p1 GENERIC amd64


I disabled the standalone execution of sshd and run it through inetd. Seems OK for the time beeing (and with proper tcpwrappers lines in /etc/hosts.allow).
 
I too have the same problem. Trying to ssh into FreeBSD13.2 box after binary update to (FreeBSD 13.2-RELEASE-p2 GENERIC amd64) just hangs. No firewall configured but I am using wireguard over the link to a bastion server, but not on that box. Lowering the MTU to 1300 on the interface of the 13.2 box solved it for me. Maybe a packet fragmentation problem that sshd does not handle properly but inetd does. I don't have the problem when Wireguard is not used.
 
SirDice said:
Packet (de)fragmentation is handled by the TCP/IP stack, not the receiving service.
Click to expand...
Thank you for your reply. True but what about the don't fragment bit. Can the receiving socket not set a "Don't Fragment" (DF) flag. My thinking being that inetd does not mind a session with fragmented packets while sshd does.
 
Also having this issue after a binary upgrade to [FreeBSD 13.2-RELEASE-p2 GENERIC amd64). Running PF and remotely log into it from different countries. Currently having my daughter reset the ISP router as a work-around until I'm able to permanently fix this.
 
You must log in or register to reply here.
Back
Top