SSH closes connection attempts before password query after failish update from 13.1 -> 13.2

Juggs

Juggs

I upgraded my VPS from 13.1 to 13.2-RELEASE

Before upgrade I did pkg update & upgrade, then freebsd-update fetch, install and upgrade -r 13.2-RELEASE
After that freebsd-update install, reboot when required and after that the second freebsd-update install.
At this phase I accidentally run out of space on device. I freed space and re-did last steps. After that seemed that the upgrade went through.
All this I did over SSH, as said this is running on VPS.
At this point I noticed that creating new SSH connections failed. I restarted the sshd service from the SSH connection I had alive as I remembered I had similar problem with 13.0 -> 13.1 upgrade which was resolved simply rebooting.
So that's what I did.
After this the server comes up, Nginx is working, but all SSH attempts fail with connection closed, with no password query.
VPS admin can locally log in. He noticed that chown seems to be missing, so this leads me to suspect that something has failed in the upgrade.

Code: 
user@host:~ $ ssh -vv user@remote.host -p22
OpenSSH_8.8p1, OpenSSL 1.1.1o-freebsd  3 May 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolving "remote.host" port 22
debug1: Connecting to remote.host [1.2.3.4] port 22.
debug1: Connection established.
debug1: identity file /home/user/.ssh/id_rsa type -1
debug1: identity file /home/user/.ssh/id_rsa-cert type -1
debug1: identity file /home/user/.ssh/id_dsa type -1
debug1: identity file /home/user/.ssh/id_dsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa type -1
debug1: identity file /home/user/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/user/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/user/.ssh/id_ed25519 type -1
debug1: identity file /home/user/.ssh/id_ed25519-cert type -1
debug1: identity file /home/user/.ssh/id_ed25519_sk type -1
debug1: identity file /home/user/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/user/.ssh/id_xmss type -1
debug1: identity file /home/user/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.8 FreeBSD-20211221
debug1: Remote protocol version 2.0, remote software version OpenSSH_9.3 FreeBSD-20211221
debug1: Fssh_compat_banner: match: OpenSSH_9.3 FreeBSD-20211221 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to remote.host:22 as 'user'
debug1: Fssh_load_hostkeys: fopen /home/user/.ssh/known_hosts2: No such file or directory
debug1: Fssh_load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: Fssh_load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
Connection closed by 1.2.3.4 port 22

All this takes like 0 seconds.

I'm having a feeling that something has gotten lost from the server, but the sshd service is running and can be seen with nmap:
Code: 
PORT     STATE SERVICE
22/tcp open  sdserver
As I'm writing this I notice that this service should probably be called "ssh"?
Okay with different parameters nmap gave:
Code: 
PORT     STATE SERVICE VERSION
22/tcp open  ssh     OpenSSH 9.3 (FreeBSD 20211221; protocol 2.0)

I have tried to connect with several FreeBSD clients and PuTTY. I'm drawing blanks here.
All the clues are appreciated.

-Jukka
 
Did you use ZFS? freebsd-update(8) creates boot environments when upgrading, so you can easily rollback the upgrade. Then I would recommend just redoing the upgrade, making sure you have enough disk space this time :)

If you don't have ZFS you won't have the boot environments, you could 'restart' the upgrade process: freebsd-update -r 13.2-RELEASE --currently-running 13.1-RELEASE upgrade

I wouldn't reboot after the first freebsd-update install and just run freebsd-update install twice in a row, then update my packages, then reboot.
 
Thanks for the swift reply SirDice !

Using ZFS yes, however, the VPS I use has merely 10GB SSD for system (+ 1TB HDD for storage).
From this minus the swap leaves me with about 5+ GB for /. Before starting the upgrade there was 1.9GB free which I thought would suffice, I was proven wrong.
When the device was out of space I deleted all the ZFS snapshots.

I should propably take a look if I can move /usr and /var to the 1TB space after I get this trainwreck fixed.

Will try that reinstall method posthaste!
 
Juggs said:
Using ZFS yes, however, the VPS I use has merely 10GB SSD for system (+ 1TB HDD for storage).
Click to expand...
Then it's likely freebsd-update(8) created boot environments. That may have been the reason why it filled up, lots of left over 'older' update/upgrade BEs.
Code: 
# Create a new boot environment when installing patches
# CreateBootEnv yes

You can check for the boot environments: bectl list. Remove old, unnecessary, ones: bectl destroy <name>.

Code: 
root@maelcum:~ # bectl list
BE                             Active Mountpoint Space Created
13.2-RELEASE_2023-06-25_184542 -      -          313M  2023-06-25 18:45
default                        NR     /          1.97G 2023-06-03 08:31
The one marked NR is de Running BE and Next booting BE. You want to keep that one :)
 
Little progress here... regained access to the VPS after service provider installed OpenSSH-portable on it.
Still missing some of the core utilities, chown seems to be in /rescue as well.
I'm hesitant to re-try the system upgrade before I manage to move /var and /usr away from the zroot.
Would the zfs send/receive be the way to go with this?
My VPS admin did A LOT of remote support with me for obvious reasons and I'd rather not FUBAR this again :)
 
You must log in or register to reply here.
Back
Top