SSH problems after upgrade from 13.1 - 13.3

C

carmoda

Hi Everyone,

I experienced a particularly messy upgrade from 13.1 - 13.3 on my personal web and mail server. 5 or so config file merge conflicts. I rambled through using 'vi' - but I typically use 'ee' so It's possible i stuffed up somewhere.

The end result is a working server, however without the ability to remotely login via SSH. I am having to use my ISPs supplied console. My auth log is as such:

Code: 
Aug 18 23:10:10 brideshead sshd[804]: Received signal 15; terminating.
Aug 18 23:10:11 brideshead sshd[25578]: Server listening on
45.32.188.155 port 44.
Aug 18 23:10:11 brideshead sshd[25578]: Server listening on 0.0.0.0 port
 44.
Aug 18 23:10:39 brideshead sshd[25580]: error: Fssh_kex_input_kexinit:
unknown kex type 9 [preauth]
Aug 18 23:10:39 brideshead sshd[25580]: Fssh_ssh_dispatch_run_fatal:
Connection from 180.181.142.239 port 60645: unexpected internal error
[preauth]
Aug 19 04:26:51 brideshead sshd[27673]: error: Fssh_kex_protocol_error:
type 34 seq 1 [preauth]
Aug 19 04:26:53 brideshead sshd[27673]: Connection closed by

I have had a close look at my sshd config, groups, generating new public and private keys. i'm stumped.
 
try to run sshd with this option
-T Extended test mode. Check the validity of the configuration file, output the effective configuration to stdout and then exit.
 
Hi,

I found an old bug report similar to your issue which implies that the service sshd should be restarted to fix the issue, I don't know if it can work but it's worth a try.

carmoda said:
5 or so config file merge conflicts. I rambled through using 'vi' - but I typically use 'ee' so It's possible i stuffed up somewhere.
Click to expand...
If you are not sure here is the default sshd_config file, better to have a clean one:
github.com

freebsd-src/crypto/openssh/sshd_config at releng/13.3 · freebsd/freebsd-src

The FreeBSD src tree publish-only repository. Experimenting with 'simple' pull requests.... - freebsd/freebsd-src
github.com github.com

Advice for the next updates don't put your own modifications in /etc/ssh/sshd_config directly, instead put them in a dedicated file, best way to avoid being locked out:
forums.freebsd.org

Solved - freebsd-update: files have been modified locally

Hello everybody, when running freebsd-update on my 13.2-RELEASE-p2 boxes, I get: Fetching metadata signature for 13.2-RELEASE from update1.freebsd.org... done. Fetching metadata index... done. Inspecting system... done. Preparing to download files... done. The following files are affected by...
forums.freebsd.org forums.freebsd.org
 
gotnull said:
Hi,

I found an old bug report similar to your issue which implies that the service sshd should be restarted to fix the issue, I don't know if it can work but it's worth a try.


If you are not sure here is the default sshd_config file, better to have a clean one:
github.com

freebsd-src/crypto/openssh/sshd_config at releng/13.3 · freebsd/freebsd-src

The FreeBSD src tree publish-only repository. Experimenting with 'simple' pull requests.... - freebsd/freebsd-src
github.com github.com

Advice for the next updates don't put your own modifications in /etc/ssh/sshd_config directly, instead put them in a dedicated file, best way to avoid being locked out:
forums.freebsd.org

Solved - freebsd-update: files have been modified locally

Hello everybody, when running freebsd-update on my 13.2-RELEASE-p2 boxes, I get: Fetching metadata signature for 13.2-RELEASE from update1.freebsd.org... done. Fetching metadata index... done. Inspecting system... done. Preparing to download files... done. The following files are affected by...
forums.freebsd.org forums.freebsd.org
Click to expand...
i've restarted sshd, a few times in fact, and a reboot.

the only changes were to my port.
 
output from sshd -T

can't say i see anything obvious.

Code: 
port 44
addressfamily any
listenaddress 0.0.0.0:44
listenaddress [::]:44
usepam yes
logingracetime 60
x11displayoffset 10
maxauthtries 3
maxsessions 10
clientaliveinterval 0
clientalivecountmax 3
requiredrsasize 1024
streamlocalbindmask 0177
unusedconnectiontimeout none
permitrootlogin no
ignorerhosts yes
ignoreuserknownhosts no
hostbasedauthentication no
hostbasedusesnamefrompacketonly no
pubkeyauthentication yes
kerberosauthentication no
kerberosorlocalpasswd yes
kerberosticketcleanup yes
gssapiauthentication no
gssapicleanupcredentials yes
passwordauthentication no
kbdinteractiveauthentication yes
printmotd yes
x11forwarding yes
x11uselocalhost yes
permittty yes
permituserrc yes
strictmodes yes
tcpkeepalive yes
permitemptypasswords no
compression yes
gatewayports no
usedns yes
allowtcpforwarding yes
allowagentforwarding yes
disableforwarding no
allowstreamlocalforwarding yes
streamlocalbindunlink no
fingerprinthash SHA256
exposeauthinfo no
useblacklist no
pidfile /var/run/sshd.pid
modulifile /etc/ssh/moduli
xauthlocation /usr/local/bin/xauth
ciphers
chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-
gcm@openssh.com,aes256-gcm@openssh.com
macs
umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-
etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-
etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-
sha2-256,hmac-sha2-512,hmac-sha1
banner none
forcecommand none
chrootdirectory none
trustedusercakeys none
revokedkeys none
securitykeyprovider internal
authorizedprincipalsfile none
versionaddendum FreeBSD-20230719
authorizedkeyscommand none
authorizedkeyscommanduser none
authorizedprincipalscommand none
authorizedprincipalscommanduser none
hostkeyagent none
kexalgorithms
sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-
sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-
nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-
sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
casignaturealgorithms
ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-
nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-
nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
hostbasedacceptedalgorithms
ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-
v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-
nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-
ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-
v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-
v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-
nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-
nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
hostkeyalgorithms
ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-
v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-
nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-
ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-
v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-
v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-
nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-
nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
pubkeyacceptedalgorithms
ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-
v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-
nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-
ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-
v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-
v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-
nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-
nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
loglevel INFO
syslogfacility AUTH
authorizedkeysfile .ssh/authorized_keys
hostkey /etc/ssh/ssh_host_rsa_key
hostkey /etc/ssh/ssh_host_ecdsa_key
hostkey /etc/ssh/ssh_host_ed25519_key
authenticationmethods any
channeltimeout none
subsystem sftp /usr/libexec/sftp-server
maxstartups 10:30:100
persourcemaxstartups none
persourcenetblocksize 32:128
permittunnel no
ipqos af21 cs1
rekeylimit 0 0
permitopen any
permitlisten any
permituserenvironment no
pubkeyauthoptions none
 
carmoda said:
13.1 - 13.3
Click to expand...

If you can:
  1. activate then boot the ZFS boot environment that preceded the upgrade
  2. update 13.1
  3. upgrade from 13.1 to 13.2
  4. upgrade from 13.2 to 13.3
  5. be prepared to upgrade from 13.3 to 13.4-RELEASE.

carmoda said:
… I rambled through using 'vi' - but I typically use 'ee' so It's possible i stuffed up somewhere. …
Click to expand...

Edit/check:
  • /etc/csh.cshrc
  • /etc/profile
  • /root/.cshrc
  • /root/.profile
After you get the vi stuff out, ee in: log out, log in.
 
i've never experienced such a long and messy upgrade process in 25 years using FreeBSD. its very disappointing to see.
 
Most people haven‘t encountered this behaviour at all so most people haven’t seen it.

I‘ve upgraded about 25 machines to 13.3 (from 13.1 then 13.2) and haven’t seen anything like this.

Definitely no <<< or similar in your sshd_config?

Definitely did all upgrade steps including userland?

Upgraded with freebsd-update or something else?

What port did you change and why?
 
This looks off:
Code: 
versionaddendum FreeBSD-20230719
That should be FreeBSD-20240806 on 13.3-RELEASE-p5.

richardtoohey2 said:
I‘ve upgraded about 25 machines to 13.3 (from 13.1 then 13.2) and haven’t seen anything like this.
Click to expand...
Same for me, upgraded plenty of systems from 13.1 -> 13.2 -> 13.3 and haven't encountered this issue.
 
carmoda said:
… disappointing …
Click to expand...

From the announcement for 13.1-RELEASE: "… will be supported until at least three months after FreeBSD 13.2-RELEASE. …". 13.2 was released in April 2023.

13.1 reached end of life at the end of July 2023. That was expected, not disappointing.
 
Cath O'Deray said:
From the announcement for 13.1-RELEASE: "… will be supported until at least three months after FreeBSD 13.2-RELEASE. …". 13.2 was released in April 2023.

13.1 reached end of life at the end of July 2023. That was expected, not disappointing.
Click to expand...
Cath, I'm a retired CIO come farmer who dabbles with agritech for a laugh. Not a System Administrator. I'll say this. 25 years ago when I first got into FreeBSD i never witnessed deliberately unhelpful replies.
 
richardtoohey2 said:
Most people haven‘t encountered this behaviour at all so most people haven’t seen it.

I‘ve upgraded about 25 machines to 13.3 (from 13.1 then 13.2) and haven’t seen anything like this.

Definitely no <<< or similar in your sshd_config?

Definitely did all upgrade steps including userland?

Upgraded with freebsd-update or something else?

What port did you change and why?
Click to expand...
I was taught to always change the default port for SSH as an extra 'security through obscurity' step, even starting to use pub/priv keys. I just kept it. I could try 22.

I have checked/set all the 'merge conflict' affected configs to their default settings. The errors in the auth log didn't change. I don't get the opportunity to see anything at the client end as i'm using a windows client (kitty). apologies, I haven't used windows in decades. I can see a facility to output the log. I'll try that.

Affirmative, freebsd-update was used. Again, there were half a dozen merge conflicts. 1-0 maybe is not usual. There was no indication anything went wrong other than the conflicts. Most services were running OK. Just unable to remotely SSH in.
 
Simple suggestion: You probably have messed up your sshd_config, big time. Try moving it aside, and find a stock 13.3 version of sshd.conf and put it in place there. If things start working magically, then you can start comparing the two files. If the diffs are too large to work through, do a binary search: first half from one file, second half from the other, does it work now? Quarter files?

Hope it helps.
 
carmoda said:
I was taught to always change the default port for SSH as an extra 'security through obscurity' step, even starting to use pub/priv keys. I just kept it. I could try 22.
Click to expand...
I do the same; just checking you meant network port as opposed to ported program (didn’t think so but just to eliminate any possible misunderstanding).

It definitely feels like there have been a few more config merge files recently so I know what you mean. If you look at reply #3 maybe there’s a way of reducing your local edits (need to do it myself).

Sounds like you’ve upgraded the same way I have been with freebsd-update (other than I’ve done each dot release) so really does feel like a mangled sshd_config (hence the suggestions to try a vanilla configuration). I’m not sure but think sshd itself will be the userland step of the update hence the question about checking that.

If I get time I’ll try a 13.1 install and then upgrade to 13.3 to see if I can get the same results as you.

In the 13.x upgrades I think one authentication scheme (not the right wording) was deprecated and there was one upgrade where you had to be careful to restart sshd or reboot so maybe one of those caught you out - but seems more likely a config file.
 
Off-topic warning:

carmoda said:
I was taught to always change the default port for SSH as an extra 'security through obscurity' step, ...
Click to expand...

This really helped, for about 2 decades. I did the same thing. I regularly looked at my logs (the daily security report helpfully shows them to you), and on machines that used non-standard ssh ports and are on the public internet, there were typically ZERO attempts to even use that port. I typically use large 5-digit numbers for the ports.

This changed within the last year. There are now several entities on the web that scan *ALL* ports of *ALL* reachable IP addresses, and are smart enough to distinguish what server is responding, so they can tell ssh servers from https servers and all others (only those two are really relevant these days on public-facing ports). Then they publish or sell the results of these scans, for all black-hat hackers to see. Some of them claim to do so under the guise of security or research. My (paranoid?) theory is that they are just supplies to the hacking industry.

Since this has happened, my formerly well-hidden ssh ports get hundreds or thousands of hack attempts per day. Sadly, this means that I'll have to invest (precious) time into better security for my ssh ports (like self-closing packet filtering rules).

Grumble ...
 
I upped the log levels fairly early on and I didn't see anything in there (in addition to my original post) I could use to further investigate the issue. All config files we painfully checked and tested.

It's an 8 year old VM so perhaps it's time to retire it.
 
You must log in or register to reply here.
Back
Top