I am experiencing the same problem with https://forums.freebsd.org/threads/freebsd-13-2-openssh-9-3-1-connection-limit.88891/ when I run sshd as a standalone process (the problem was also in FreeBSD 13).
When the sshd is executed from inetd nothing happens.
Executing /etc/rc.d/sshd start makes the sshd stuck/hang after some time
FreeBSD up to date, and no load currently.
if I execute SSHD from /etc/rc.d/sshd start
then only one user can login, the rest stuck at the message Local version, even when I am sshing from my own machine and the firewall is complete open
I have tried many settings on sshd config. Currently I am using the following. Whatever i have changed the problem still exists, even from local or from remote.
Moreover, the sshd server does not restart by its own.
If I try to restart it it hangs indefinite:
only by killing -9 the PID of the main sshd process I can restart it.
the process tree (pstree) when sshd stuck always shows at least one user connected. The user is autneticated and valid.
freebsd-ids shows no errors cocnerning the ssh
After trying to restart the sshd i see processed stuck at CLOSED state in netstat forever. I was waiting for more than 30 minutes and they were still in CLOSED.
procstat of the process shows that there is one child process (the valid connectin) still in progress and it is not killed
the connection remains in CLOSED and it is not killed
NOTE that in time every user gets one connectin and then it remains in CLOSED, so after some hours/days of usage there are multiple CLOSED connections in netstat that do not dissapear.
the sshd process of that connection is in 'S' state
If I kill -9 the sshd process everything is cleared, sshd is restarted but after some minutes again the same problem arises as before.
Nothing relavent is logged /var/log/auth.log or /var/log/messages.
ps does not show anything useful:
tcpdump shows [R] reset packages sent from my server ocassionaly to the <otherip>:58478 but nothing else.
lsof (list open files) verified that the connection remains in CLOSED state.
I even tryied to ktrace the sshd process (and then the child process 74494)
ktrace -p 73968
and after 20 minutes nothing was logged. The file ktrace.out was 0 bytes and 'kdump' was empty.
procstat shows that the main sshd process is in sleeping state (why did not receive the kill signal?)
while the CLOSED state for the chld sshd process seems to wait something(?)
sshd process has empty queue in netstat:
Finaly, my client config that I am using to connect to server, utilizes multiplexing as follows:
I have been searching for many hours to find a solution.
I could use inetd as a workarround, but I would like to use the sshd process.
As i told you this is similar with: https://forums.freebsd.org/threads/freebsd-13-2-openssh-9-3-1-connection-limit.88891/
and also from a very intersting post from 2008 about sockets stuck in CLOSED state:
Thanks for any suggestions!
When the sshd is executed from inetd nothing happens.
Executing /etc/rc.d/sshd start makes the sshd stuck/hang after some time
FreeBSD up to date, and no load currently.
Code:
> uname -a
FreeBSD <myname> 14.1-RELEASE-p4 FreeBSD 14.1-RELEASE-p4 GENERIC amd64
> uptime
1:59PM up 6 days, 11:25, 7 users, load averages: 0.30, 0.25, 0.33
> ssh -V
OpenSSH_9.7p1, OpenSSL 3.0.13 30 Jan 2024if I execute SSHD from /etc/rc.d/sshd start
then only one user can login, the rest stuck at the message Local version, even when I am sshing from my own machine and the firewall is complete open
Code:
> sudo ipfw add 1 allow all from any to any
00001 allow ip from any to any
> ssh -4 -v 127.0.0.1
OpenSSH_9.7p1, OpenSSL 3.0.13 30 Jan 2024
debug1: Connecting to 127.0.0.1 [127.0.0.1] port 22.
debug1: Connection established.
debug1: identity file /home/mdasyg/.ssh/id_rsa type -1
debug1: identity file /home/mdasyg/.ssh/id_rsa-cert type -1
debug1: identity file /home/mdasyg/.ssh/id_ecdsa type -1
debug1: identity file /home/mdasyg/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/mdasyg/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/mdasyg/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/mdasyg/.ssh/id_ed25519 type -1
debug1: identity file /home/mdasyg/.ssh/id_ed25519-cert type -1
debug1: identity file /home/mdasyg/.ssh/id_ed25519_sk type -1
debug1: identity file /home/mdasyg/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/mdasyg/.ssh/id_xmss type -1
debug1: identity file /home/mdasyg/.ssh/id_xmss-cert type -1
debug1: identity file /home/mdasyg/.ssh/id_dsa type -1
debug1: identity file /home/mdasyg/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.7
^CI have tried many settings on sshd config. Currently I am using the following. Whatever i have changed the problem still exists, even from local or from remote.
Code:
> sudo more /etc/ssh/sshd_config | grep -v '#' | sort | uniq
AuthorizedKeysFile .ssh/authorized_keys
ChallengeResponseAuthentication no
channeltimeout global=60
ClientAliveCountMax 2
ClientAliveInterval 30
Compression no
GSSAPIAuthentication no
HostbasedAuthentication no
KerberosAuthentication no
LoginGraceTime 30
MaxStartups 50:50:200
PermitEmptyPasswords no
PermitRootLogin yes
Subsystem sftp /usr/libexec/sftp-server
UsePAM noMoreover, the sshd server does not restart by its own.
If I try to restart it it hangs indefinite:
Code:
> /etc/rc.d/sshd onerestart
Performing sanity check on sshd configuration.
Stopping sshd.
Waiting for PIDS: 73968only by killing -9 the PID of the main sshd process I can restart it.
the process tree (pstree) when sshd stuck always shows at least one user connected. The user is autneticated and valid.
Code:
> pstree 73968
-+= 73968 root sshd: /usr/sbin/sshd [listener] 0 of 50-200 startups (sshd)
\-+= 74250 root sshd: mduser [priv] (sshd)
\--- 74494 mdasygenis sshd: mduser@notty (sshd)freebsd-ids shows no errors cocnerning the ssh
Code:
> freebsd-update IDS |& grep -i ssh
/etc/ssh/moduli has 0600 permissions, but should have 0644 permissions.
/etc/ssh/ssh_config has 0600 permissions, but should have 0644 permissions.
/etc/ssh/ssh_config has SHA256 hash bc09a9b9fe785a714bc153f3e9bad32efe81d160ea921a464a1eb180ece3b983, but should have SHA256 hash d12f4f223cd089cc47121cdfcd10463965db41e63c425c68e9a2bbf7b3e26a3f.
/etc/ssh/sshd_config has 0600 permissions, but should have 0644 permissions.
/etc/ssh/sshd_config has SHA256 hash fc9459564ab8273c76431712c012d965dad94f950c5af69437199d7eed147f72, but should have SHA256 hash 726ea8f0217e8a89fd3b2dd3128b4f681939c19ef434f522eea479320341c201.After trying to restart the sshd i see processed stuck at CLOSED state in netstat forever. I was waiting for more than 30 minutes and they were still in CLOSED.
Code:
>sysctl net.inet.tcp.keepidle ; sysctl net.inet.tcp.keepintvl ; sysctl net.inet.tcp.msl
net.inet.tcp.keepidle: 300000
net.inet.tcp.keepintvl: 15000
net.inet.tcp.msl: 15000procstat of the process shows that there is one child process (the valid connectin) still in progress and it is not killed
Code:
> procstat -f 73968
PID COMM FD T V FLAGS REF OFFSET PRO NAME
73968 sshd trace v r rw------ - - - /root/ktrace.out
73968 sshd text v r r------- - - - /usr/sbin/sshd
73968 sshd cwd v d r------- - - - /
73968 sshd root v d r------- - - - /
73968 sshd 0 v c rw------ 3 0 - /dev/null
73968 sshd 1 v c rw------ 3 0 - /dev/null
73968 sshd 2 v c rw------ 3 0 - /dev/null
73968 sshd 3 s - rw---n-- 1 0 TCP 0 0 ::.22 ::.0
73968 sshd 4 s - rw---n-- 1 0 TCP 0 458954624 *:22 *:0
73968 sshd 5 s - rw---n-- 1 0 TCP 0 43 <myip>22 <otherip>:58478the connection remains in CLOSED and it is not killed
Code:
> netstat -an | grep 58478
tcp4 43 <myip>22 <otherip>:58478 CLOSEDNOTE that in time every user gets one connectin and then it remains in CLOSED, so after some hours/days of usage there are multiple CLOSED connections in netstat that do not dissapear.
the sshd process of that connection is in 'S' state
Code:
> ps axuww | grep 74494
mduser 74494 0.0 0.1 22904 10296 - S 12:36 0:00.05 sshd: mduser@notty (sshd)If I kill -9 the sshd process everything is cleared, sshd is restarted but after some minutes again the same problem arises as before.
Nothing relavent is logged /var/log/auth.log or /var/log/messages.
ps does not show anything useful:
Code:
> ps -fp 74494
PID TT STAT TIME COMMAND
74494 - S 0:00.06 sshd: mduser@notty (sshd)
> ps -fp 73968
PID TT STAT TIME COMMAND
73968 - IWs 0:00.00 sshd: /usr/sbin/sshd [listener] 0 of 50-200 startups (sshd)tcpdump shows [R] reset packages sent from my server ocassionaly to the <otherip>:58478 but nothing else.
lsof (list open files) verified that the connection remains in CLOSED state.
Code:
> lsof -i tcp:22
sshd 73968 root 3u IPv6 0xfffff8010cef2540 0 TCP *:ssh (LISTEN)
sshd 73968 root 4u IPv4 0xfffff801caeb9a80 458954624 TCP *:ssh->*:* (LISTEN)
sshd 73968 root 5u IPv4 0xfffff8023002b000 43 TCP <myip>:ssh-><otherip>:58478 (CLOSED)I even tryied to ktrace the sshd process (and then the child process 74494)
ktrace -p 73968
and after 20 minutes nothing was logged. The file ktrace.out was 0 bytes and 'kdump' was empty.
procstat shows that the main sshd process is in sleeping state (why did not receive the kill signal?)
while the CLOSED state for the chld sshd process seems to wait something(?)
Code:
procstat -k 73968
PID TID COMM TDNAME KSTACK
73968 137151 sshd - mi_switch sleepq_catch_signals sleepq_wait_sig _sleep kern_wait6 sys_wait4 amd64_syscall fast_syscall_common
procstat -k 74494
PID TID COMM TDNAME KSTACK
74494 138290 sshd - mi_switch sleepq_catch_signals sleepq_timedwait_sig _cv_timedwait_sig_sbt seltdwait kern_poll_kfds kern_poll sys_ppoll amd64_syscall fast_syscall_commonsshd process has empty queue in netstat:
Code:
netstat -Lan | grep 22
tcp4 14/0/128 *.22
tcp6 0/0/128 *.22Finaly, my client config that I am using to connect to server, utilizes multiplexing as follows:
Code:
>cat ~/.ssh/config
host *
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p
ControlPersist yes
ControlPersist 3600
ServerAliveInterval 300
ServerAliveCountMax 2I have been searching for many hours to find a solution.
I could use inetd as a workarround, but I would like to use the sshd process.
As i told you this is similar with: https://forums.freebsd.org/threads/freebsd-13-2-openssh-9-3-1-connection-limit.88891/
and also from a very intersting post from 2008 about sockets stuck in CLOSED state:
Thanks for any suggestions!
