Solved ssh login stopped working after upgrade to 12.4-RELEASE

byrnejb

byrnejb

Since upgrading a system to 12.4-RELEASE from 12.3 logins from a remote host using authorized_keys has stopped functioning. Running ssh from the remote shows this:

Code: 
ssh -vv vhost01
OpenSSH_8.8p1, OpenSSL 1.1.1o-freebsd  3 May 2022
debug1: Reading configuration data /root/.ssh/config
debug1: /root/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolving "vhost01.hamilton.harte-lyne.ca" port 22
debug1: Connecting to vhost01.hamilton.harte-lyne.ca [216.185.71.41] port 22.
debug1: Connection established.
debug1: identity file /root/.ssh/id_rsa type 0
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type 2
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa_sk type -1
debug1: identity file /root/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type 3
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_ed25519_sk type -1
debug1: identity file /root/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /root/.ssh/id_xmss type -1
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.8 FreeBSD-20211221
debug1: Remote protocol version 2.0, remote software version OpenSSH_9.1 FreeBSD-20221019
debug1: Fssh_compat_banner: match: OpenSSH_9.1 FreeBSD-20221019 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to vhost01.hamilton.harte-lyne.ca:22 as 'root'
debug1: Fssh_load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: Fssh_load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: Fssh_load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:q2N77sbjHlZ35RzCZCCP1+V/yqiVzS56oFhH4UzX0zk
debug2: ldns: got 8 answers from DNS
debug2: ldns: trying to validate RRset
debug2: ldns: got 1 signature(s) (RRTYPE 46) from DNS
debug2: ldns: RRset validation failed: General LDNS error
debug1: found 8 insecure fingerprints in DNS
debug1: Fssh_verify_host_key_dns: matched SSHFP type 4 fptype 1
debug1: Fssh_verify_host_key_dns: matched SSHFP type 4 fptype 2
debug1: matching host key fingerprint found in DNS
debug1: Fssh_load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: Fssh_load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: Fssh_load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'vhost01.hamilton.harte-lyne.ca' is known and matches the ED25519 host key.
debug1: Found key in /root/.ssh/known_hosts:74
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /root/.ssh/id_rsa RSA SHA256:14qSPzqINY1ejGYgNGlkSMtVVBtek36Q/CrsM07+pgE
debug1: Will attempt key: /root/.ssh/id_dsa
debug1: Will attempt key: /root/.ssh/id_ecdsa ECDSA SHA256:9+DpY9vFXemvVeyR5JYVR+HKc2u9t+5I7J0YM0Jw5W8
debug1: Will attempt key: /root/.ssh/id_ecdsa_sk
debug1: Will attempt key: /root/.ssh/id_ed25519 ED25519 SHA256:rg5CuR8qHh6U/SR1cUh/qWqof5TVXM4Ew1f1Fi/R20I
debug1: Will attempt key: /root/.ssh/id_ed25519_sk
debug1: Will attempt key: /root/.ssh/id_xmss
debug2: pubkey_prepare: done
debug1: SSH2_MSG_EXT_INFO received
debug1: Fssh_kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com>
debug1: Fssh_kex_input_ext_info: publickey-hostbound@openssh.com (unrecognised)
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering public key: /root/.ssh/id_rsa RSA SHA256:14qSPzqINY1ejGYgNGlkSMtVVBtek36Q/CrsM07+pgE
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: /root/.ssh/id_rsa RSA SHA256:14qSPzqINY1ejGYgNGlkSMtVVBtek36Q/CrsM07+pgE
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Offering public key: /root/.ssh/id_ecdsa ECDSA SHA256:9+DpY9vFXemvVeyR5JYVR+HKc2u9t+5I7J0YM0Jw5W8
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: /root/.ssh/id_ecdsa ECDSA SHA256:9+DpY9vFXemvVeyR5JYVR+HKc2u9t+5I7J0YM0Jw5W8
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Trying private key: /root/.ssh/id_ecdsa_sk
debug1: Offering public key: /root/.ssh/id_ed25519 ED25519 SHA256:rg5CuR8qHh6U/SR1cUh/qWqof5TVXM4Ew1f1Fi/R20I
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: /root/.ssh/id_ed25519 ED25519 SHA256:rg5CuR8qHh6U/SR1cUh/qWqof5TVXM4Ew1f1Fi/R20I
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Trying private key: /root/.ssh/id_ed25519_sk
debug1: Trying private key: /root/.ssh/id_xmss
debug2: we did not send a packet, disable method
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug2: we sent a keyboard-interactive packet, wait for reply
debug2: input_userauth_info_req: entering
debug2: input_userauth_info_req: num_prompts 1
(root@vhost01.hamilton.harte-lyne.ca) Password for root@vhost01.hamilton.harte-lyne.ca:

I do not see where the error is. It seems that all the offered keys are excepted and then ignored. I note that the two system are running different versions of OpenSSH, but surely that cannot be the problem.

Code: 
debug1: Local version string SSH-2.0-OpenSSH_8.8 FreeBSD-20211221

debug1: Remote protocol version 2.0, remote software version OpenSSH_9.1 FreeBSD-20221019

The remote host is running 13.1-RELEASE-p2 and OpenSSH_8.8p1, OpenSSL 1.1.1o-freebsd 3 May 2022. The local host is running OpenSSH_9.1 FreeBSD-20221019
 
The issue seems to be with the PAM authentication, but I do not understand what the problem is:

Code: 
debug3: monitor_read: checking request 106
debug3: mm_answer_pam_query
debug3: PAM: sshpam_query entering
debug3: ssh_msg_recv entering
debug3: Fssh_ssh_msg_send: type 9
debug3: PAM: Authentication error
PAM: Authentication error for root from vhost06.hamilton.harte-lyne.ca
debug3: mm_request_send: entering, type 107
debug3: mm_sshpam_query: pam_query returned -1 [preauth]
debug2: auth2_challenge_start: devices <empty> [preauth]
debug3: mm_sshpam_free_ctx [preauth]
debug3: mm_request_send: entering, type 110 [preauth]
debug3: mm_sshpam_free_ctx: waiting for MONITOR_ANS_PAM_FREE_CTX [preauth]
debug3: mm_request_receive_expect: entering, type 111 [preauth]
 
Can you check /etc/ssh/sshd_config on the host you're connecting to? Maybe there are some merge artifacts left over. Although I would expect sshd(8) to stop working completely if that's the case. Verify /etc/ssh/ssh_config on the 'client' system. It may have similar issues.

byrnejb said:
Code: 
(root@vhost01.hamilton.harte-lyne.ca) Password for root@vhost01.hamilton.harte-lyne.ca:
Click to expand...
Keep in mind that logging in as root remotely is disabled by default. That may be part of the problem?
 
You must log in or register to reply here.
Back
Top