sshd preventing multiple logins, /etc/rc.d/sshd restart ... hangs

[ProServ]

Hi, logged into a FreeBSD 13.3-RELEASE-p1 server, trying to restart sshd ....

 # /etc/rc.d/sshd restart
Performing sanity check on sshd configuration.
Stopping sshd.
Waiting for PIDS: 2613

The above cmd hangs until ctl + c

# ps auxw |grep sshd
root      2613   0.0  0.0   21048   5540  -  Is   11:28      0:00.02 sshd: /usr/sbin/sshd [listener] 1 of 10-100 startups (sshd)
root      6708   0.0  0.1   21492   7288  -  Ss   23:00      0:00.11 sshd: root@pts/0 (sshd)

In /etc/ssh/sshd_config have:

PermitRootLogin yes
#StrictModes yes
MaxAuthTries 8
MaxSessions 10
MaxStartups 6

Logged out of the ssh session and now cannot get back in. The only way is to pay someone to go there and reboot.

ssh: connect to host host.domain.tld port xxxxx: Network is unreachable

Tried to ssh to same server from another server and it's refusing ssh connection:

ssh: connect to host hostname.domain.tld port xxxxx: Connection refused

Any idea what is causing this?
Thanks in advance for your assistance.

 

[iRobbery]

It looks like the rc.d command was stopping ssh with PID 2613, which is running still in the ps aux overview. I assume it did stop after your logout and hence you get the refused, typically that means nothing is listening on that port (22).

If hands on support is really pricey and you have option to power cycle the machine, and data isn't too critical on the machine, I would just consider issuing a power cycle first. If that doesn't work, you can always rent some hands on.

 

[ProServ]

I have left /etc/rc.d/sshd restart for > 1 hour after which CTL+C and then exit. Now impossible to get back via ssh. Something is preventing sshd from restarting and I have looked in various logs and don't see anything that alerts me to this issue. I have put another request to the DC to just reboot it. When it does reboot, I can get in via ssh, however from same remote source or other servers we have unable to login, no error, just refused connection or a blinking putty screen presumably waiting for a response which it never gets.

The only change made to sshd_config was to comment out and IP address so the default of #0.0.0.0 works. I don't see how that would change anything really in terms of causing sshd to not restart.

 

[SirDice]

I have left /etc/rc.d/sshd restart for > 1 hour after which CTL+C and then exit.

Looking the output it seems to fail to stop one of the processes. The ctrl-c was fine to do but you shouldn't have exited and kept the session active.

Something is preventing sshd from restarting

That one listener on PID 2613 refuses to stop, if it's not stopped it won't restart.

no error, just refused connection

Yes, it's still busy trying to stop, so the 'main' sshd(8) daemon won't accept any new connections.

The only change made to sshd_config was to comment out and IP address so the default of #0.0.0.0 works.

You don't need to uncomment it, sshd(8) listens by default on 0.0.0.0 and ::

I don't see how that would change anything really in terms of causing sshd to not restart.

The process is idling (I) so it's not doing something, shouldn't be a problem to stop it. Instead of exiting the shell (and locking yourself out) a kill -9 2613 would probably have moved things along. Still weird why it would fail to stop though.

 

[ProServ]

What do you think it can be preventing sshd from restarting? Would ...

# freebsd-update fetch
# freebsd-update install
# service sshd restart

fix it?
or

# freebsd-update -r 14.1-RELEASE upgrade

Thanks

 

[SirDice]

What do you think it can be preventing sshd from restarting?

The better question is; why does the process refuse to stop. And I honestly have no idea. I've had this happen in the past, that was due to a failing harddisk. But then I would have expected the process to be in a D state, not I.

Anyway, it's all water under the bridge now as you've lost access. Might still be worthwhile to check the disk status when you regain access. Even if it's just to rule it out as a possible cause.