Can't ssh from a new upgraded FreeBSD 12.4 machine

I have 7 servers all running FreeBSD 12.3 and just the other day upgraded one machine to FreeBSD 12.4 however I now can't SSH into any of the FreeBSD 12.3 machines from the 12.4 machine however, I can SSH from all the 12.3 machines to the 12.4 machine just fine. All the machines are setup and configured exactly the same with the only exception being the FreeBSD version.

I've been using the same setup on these machines for years and through several FreeBSD versions and I've never run into this issue. I know there is quite a big difference in SSH versions between these two versions of FreeBSD so I thought maybe that had something to do with it but I installed the latest version of OpenSSH on one of the FreeBSD 12.3 servers and generated new keys on both servers but it still doesn't work.

The error I'm getting is:
user@machine: Permission denied (publickey).
There's nothing useful in the receiving server's auth log:
sshd[56894]: Connection closed by authenticating user user 111.111.111.111 port 49203 [preauth]
Permissions are correct and exactly the same on all machines.

I'm going crazy trying to figure this out I've never had any issues with this before. Does anyone have any advice? The debug output of the ssh command is below.

Thank you for your help!

PS: after posting the debug output below I noticed these lines and figured this could be the problem
debug3: Fssh_ssh_get_authentication_socket_path: path '/tmp/ssh-nqyEe2OVRT/agent.4080'
debug2: get_agent_identities: ssh_agent_bind_hostkey: communication with agent failed
debug1: get_agent_identities: ssh_fetch_identitylist: communication with agent failed
But I can't find anything useful online about it with most info being in regards to OpenSSH on Windows.

machine7:~ $ ssh user@machine6
The authenticity of host '[machine6]:22 ([111.111.111.111]:22)' can't be established.
ED25519 key fingerprint is SHA256:
No matching host key fingerprint found in DNS.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[machine6]:22' (ED25519) to the list of known hosts.
user@machine6: Permission denied (publickey).
m7:~ $ ssh -vvv user@machine6
OpenSSH_9.1p1, OpenSSL 1.1.1q-freebsd 5 Jul 2022
debug1: Reading configuration data /home/user/.ssh/config
debug1: /home/user/.ssh/config line 1: Applying options for *
debug1: /home/user/.ssh/config line 8: Applying options for m?
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/user/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/user/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug2: resolving "machine6" port 22
debug3: resolve_host: lookup machine6:22
debug3: ssh_connect_direct: entering
debug1: Connecting to machine6 [111.111.111.111] port 22.
debug3: Fssh_set_sock_tos: set socket 3 IPV6_TCLASS 0x48
debug1: Connection established.
debug1: identity file /home/user/.ssh/id_rsa type -1
debug1: identity file /home/user/.ssh/id_rsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa type -1
debug1: identity file /home/user/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/user/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/user/.ssh/id_ed25519 type -1
debug1: identity file /home/user/.ssh/id_ed25519-cert type -1
debug1: identity file /home/user/.ssh/id_ed25519_sk type -1
debug1: identity file /home/user/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/user/.ssh/id_xmss type -1
debug1: identity file /home/user/.ssh/id_xmss-cert type -1
debug1: identity file /home/user/.ssh/id_dsa type -1
debug1: identity file /home/user/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.1 FreeBSD-20221019
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9 FreeBSD-20200214
debug1: Fssh_compat_banner: match: OpenSSH_7.9 FreeBSD-20200214 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to machine6:22 as 'user'
debug3: put_host_port: [machine6]:22
debug3: Fssh_record_hostkey: found key type ED25519 in file /home/user/.ssh/known_hosts:2
debug3: Fssh_load_hostkeys_file: loaded 1 keys from [machine6]:22
debug1: Fssh_load_hostkeys: fopen /home/user/.ssh/known_hosts2: No such file or directory
debug1: Fssh_load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: Fssh_load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: have matching best-preference key type ssh-ed25519-cert-v01@openssh.com, using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:
debug3: verify_host_key_dns
DNS lookup error: general failure
debug3: put_host_port: [111.111.111.111]:22
debug3: put_host_port: [machine6]:22
debug3: Fssh_record_hostkey: found key type ED25519 in file /home/user/.ssh/known_hosts:2
debug3: Fssh_load_hostkeys_file: loaded 1 keys from [machine6]:22
debug1: Fssh_load_hostkeys: fopen /home/user/.ssh/known_hosts2: No such file or directory
debug1: Fssh_load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: Fssh_load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '[machine6]:22' is known and matches the ED25519 host key.
debug1: Found key in /home/user/.ssh/known_hosts:2
debug3: send packet: type 21
debug2: Fssh_ssh_set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: Fssh_ssh_set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug3: Fssh_ssh_get_authentication_socket_path: path '/tmp/ssh-nqyEe2OVRT/agent.4080'
debug2: get_agent_identities: ssh_agent_bind_hostkey: communication with agent failed
debug1: get_agent_identities: ssh_fetch_identitylist: communication with agent failed
debug1: Will attempt key: /home/user/.ssh/id_rsa
debug1: Will attempt key: /home/user/.ssh/id_ecdsa
debug1: Will attempt key: /home/user/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/user/.ssh/id_ed25519
debug1: Will attempt key: /home/user/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/user/.ssh/id_xmss
debug1: Will attempt key: /home/user/.ssh/id_dsa
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: Fssh_kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/user/.ssh/id_rsa
debug3: no such identity: /home/user/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /home/user/.ssh/id_ecdsa
debug3: no such identity: /home/user/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/user/.ssh/id_ecdsa_sk
debug3: no such identity: /home/user/.ssh/id_ecdsa_sk: No such file or directory
debug1: Trying private key: /home/user/.ssh/id_ed25519
debug3: no such identity: /home/user/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /home/user/.ssh/id_ed25519_sk
debug3: no such identity: /home/user/.ssh/id_ed25519_sk: No such file or directory
debug1: Trying private key: /home/user/.ssh/id_xmss
debug3: no such identity: /home/user/.ssh/id_xmss: No such file or directory
debug1: Trying private key: /home/user/.ssh/id_dsa
debug3: no such identity: /home/user/.ssh/id_dsa: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
user@machine6: Permission denied (publickey).
 
you need to restart sshd after upgrading to 12.4 (after the last freebsd-update install/ after reboot)
login at console and restart sshd
 
My first look on ssh issues is always if
/etc/ssh/sshd_config
is still okay.
 
Thanks for all the replies so far. I even did a clean install of 12.4 and have gone over all the configs, keys, home directories, and everything I can think of with a fine-tooth comb and everything matches how it was before and the other 12.3 servers as well but I can't for the life of me SSH out to those servers from the 12.4 server. I'm totally stumped.
 
the 12.4 machine can't connect to the 12.3 machines, is that the executive summary?
If so, can a 12.3 machine connect to the 12.4?
are both sides configured to use the same algorithms? Reason for asking there was a point where a lot of servers disabled one of the older key types, so clients could not connect to them, solution was changing the servers to allow the disabled key type until the clients could be updated.

The "no such identity" lines are odd, not exactly sure what they mean in this case.
 
the 12.4 machine can't connect to the 12.3 machines, is that the executive summary? If so, can a 12.3 machine connect to the 12.4?

Correct. 12.4 to 12.3 does not work. 12.3 to 12.4 and 12.3 works.

I've looked over every single config, file permission, directory permission, and everything I can think of and it all looks the same to me so I can't figure out what is wrong.

I think it has to be something with the ssh-agent as on the 12.4 server I get the error
debug3: Fssh_ssh_get_authentication_socket_path: path '/tmp/ssh-Te6ORsCdfL/agent.6345'
debug2: get_agent_identities: ssh_agent_bind_hostkey: communication with agent failed
debug1: get_agent_identities: ssh_fetch_identitylist: communication with agent failed
But on the 12.3 servers, I see
debug1: Will attempt key: ED25519 SHA256:.... agent
And everything works from there.

I know there's quite a difference in OpenSSH versions with 12.4 using OpenSSH_9.1p1 and 12.3 using OpenSSH_7.9p1 but I can't see anything that really stands out when looking over the release notes but maybe I missed something.

are both sides configured to use the same algorithms?
That's a good question. I've never explicitly set this anywhere so I can't answer with any confidence and didn't even think to check as I've never had this issue before when upgrading to any major or minor FreeBSD version. I will look into it.
 
I’ve not got the details to hand but certainly newer OpenSSH releases deprecated older algorithms. So maybe 12.4 defaulting to newer algorithm and 12.3 can’t match that?

But re-reading your posts it looks like something else going on - as you say with ssh-agent. Can you try without ssh-agent? Might have to set-up simple 12.3 and 12.4 machines and start with basic SSH setup (e.g. with passwords) and then move to keys and then ssh-agent.

Or wait and see if someone comes along and says “a-ha just change line 3 in your config”!
 
I’ve not got the details to hand but certainly newer OpenSSH releases deprecated older algorithms. So maybe 12.4 defaulting to newer algorithm and 12.3 can’t match that?

But re-reading your posts it looks like something else going on - as you say with ssh-agent. Can you try without ssh-agent? Might have to set-up simple 12.3 and 12.4 machines and start with basic SSH setup (e.g. with passwords) and then move to keys and then ssh-agent.

Or wait and see if someone comes along and says “a-ha just change line 3 in your config”!
Thank you for this! Your tip about "going back to the basics" helped and I was able to figure out the problem.

Turns out there was a bug with the key agent I was using and OpenSSH 8.9+ servers.

Thank you to everyone who replied for your help and time!
 
Back
Top