Solved sshd preventing multiple logins, /etc/rc.d/sshd restart ... hangs

Hi, logged into a FreeBSD 13.3-RELEASE-p1 server, trying to restart sshd ....
Code:
 # /etc/rc.d/sshd restart
Performing sanity check on sshd configuration.
Stopping sshd.
Waiting for PIDS: 2613

The above cmd hangs until ctl + c

Code:
# ps auxw |grep sshd
root      2613   0.0  0.0   21048   5540  -  Is   11:28      0:00.02 sshd: /usr/sbin/sshd [listener] 1 of 10-100 startups (sshd)
root      6708   0.0  0.1   21492   7288  -  Ss   23:00      0:00.11 sshd: root@pts/0 (sshd)

In /etc/ssh/sshd_config have:

Code:
PermitRootLogin yes
#StrictModes yes
MaxAuthTries 8
MaxSessions 10
MaxStartups 6

Logged out of the ssh session and now cannot get back in. The only way is to pay someone to go there and reboot.
Code:
ssh: connect to host host.domain.tld port xxxxx: Network is unreachable

Tried to ssh to same server from another server and it's refusing ssh connection:

Code:
ssh: connect to host hostname.domain.tld port xxxxx: Connection refused

Any idea what is causing this?
Thanks in advance for your assistance.
 
It looks like the rc.d command was stopping ssh with PID 2613, which is running still in the ps aux overview. I assume it did stop after your logout and hence you get the refused, typically that means nothing is listening on that port (22).

If hands on support is really pricey and you have option to power cycle the machine, and data isn't too critical on the machine, I would just consider issuing a power cycle first. If that doesn't work, you can always rent some hands on.
 
I have left /etc/rc.d/sshd restart for > 1 hour after which CTL+C and then exit. Now impossible to get back via ssh. Something is preventing sshd from restarting and I have looked in various logs and don't see anything that alerts me to this issue. I have put another request to the DC to just reboot it. When it does reboot, I can get in via ssh, however from same remote source or other servers we have unable to login, no error, just refused connection or a blinking putty screen presumably waiting for a response which it never gets.

The only change made to sshd_config was to comment out and IP address so the default of #0.0.0.0 works. I don't see how that would change anything really in terms of causing sshd to not restart.
 
I have left /etc/rc.d/sshd restart for > 1 hour after which CTL+C and then exit.
Looking the output it seems to fail to stop one of the processes. The ctrl-c was fine to do but you shouldn't have exited and kept the session active.
Something is preventing sshd from restarting
That one listener on PID 2613 refuses to stop, if it's not stopped it won't restart.
no error, just refused connection
Yes, it's still busy trying to stop, so the 'main' sshd(8) daemon won't accept any new connections.
The only change made to sshd_config was to comment out and IP address so the default of #0.0.0.0 works.
You don't need to uncomment it, sshd(8) listens by default on 0.0.0.0 and ::

I don't see how that would change anything really in terms of causing sshd to not restart.
The process is idling (I) so it's not doing something, shouldn't be a problem to stop it. Instead of exiting the shell (and locking yourself out) a kill -9 2613 would probably have moved things along. Still weird why it would fail to stop though.
 
What do you think it can be preventing sshd from restarting? Would ...
Code:
# freebsd-update fetch
# freebsd-update install
# service sshd restart
fix it?
or
Code:
# freebsd-update -r 14.1-RELEASE upgrade

Thanks
 
What do you think it can be preventing sshd from restarting?
The better question is; why does the process refuse to stop. And I honestly have no idea. I've had this happen in the past, that was due to a failing harddisk. But then I would have expected the process to be in a D state, not I.

Anyway, it's all water under the bridge now as you've lost access. Might still be worthwhile to check the disk status when you regain access. Even if it's just to rule it out as a possible cause.
 
Ran fsck -y nothing really wrong there.

Ran
Code:
# freebsd-update fetch
# freebsd-update install

Installing updates...
Restarting sshd after upgrade
Performing sanity check on sshd configuration.
Stopping sshd.
Waiting for PIDS: 1021

And it just sits there. Unable to ssh in again from another serve as it won't answer or complete login.

So the question is while still connected to this server, run
Code:
CTL+C
and then # kill -9 1021 ?

Thanks.
 
Ran #kill -9 1021 and then started sshd.

Upgraded this server to 14.1-RELEASE FreeBSD 14.1-RELEASE releng/14.1-n267679-10e31f0946d8 GENERIC amd64

But it still has the issue with multiple ssh logins, only 1. And it will not restart...


# service sshd restart
Performing sanity check on sshd configuration.
Stopping sshd.
Waiting for PIDS: 1103


Complete loss as to what is preventing SSHD from restarting.

Thanks for any help with this disaster.
 
Complete loss as to what is preventing SSHD from restarting.
Anything weird in /etc/ssh/sshd_config? Maybe some leftover merge issues? The default sshd_config should be fine, did you make any changes to it?
 
But it still has the issue with multiple ssh logins, only 1. And it will not restart...
As you're now on 14.1-RELEASE. Can you clarify how easily repeatable this issue (stopping the service where a relevant PID just does not want to die) is?
Seems like very easily repeatable every time you have multiple ssh logins, but perhaps I'm misinterpreting.
 
Anything weird in /etc/ssh/sshd_config? Maybe some leftover merge issues? The default sshd_config should be fine, did you make any changes to it?
SirDice: the only diff in sshd_config is Port as we use a specific port. And change passwd auth to no,. Nothing else is altered
 
As you're now on 14.1-RELEASE. Can you clarify how easily repeatable this issue (stopping the service where a relevant PID just does not want to die) is?
Seems like very easily repeatable every time you have multiple ssh logins, but perhaps I'm misinterpreting.
How easy to repeat? Get someone to reboot that sucker and then ssh in. Fine. Now, open another shell and ssh -p blah blah and nothing happens. Just sits there. And I am talking about ssh from a different server on a completely different subnet. The while still ssh'd in the problematic FreeBSD server run either
# service sshd restart or
# /etc/rc.d/sshd restart

And it will hang until andromeda collides with the milky way. Or, as SirDice mentioned, just
# kill -9 pid of sshd.

So right now I have a script running every 30 seconds to run
# /etc/rc.d/sshd restart

And right now we are able to login and have multiple connections. But this should not be happening in the first instance.

Proserv
 
And change passwd auth to no
That's the default.
Code:
# Change to yes to enable built-in password authentication.
# Note that passwords may also be accepted via KbdInteractiveAuthentication.
#PasswordAuthentication no
#PermitEmptyPasswords no

How easy to repeat? Get someone to reboot that sucker and then ssh in. Fine. Now, open another shell and ssh -p blah blah and nothing happens.
I have a whole bunch of systems running 14.1-RELEASE and don't have this issue on any of them. I cannot reproduce the problem. On a client network there are about 20 13.3-RELEASE systems and can't reproduce it there either.
 
That's the default.
Code:
# Change to yes to enable built-in password authentication.
# Note that passwords may also be accepted via KbdInteractiveAuthentication.
#PasswordAuthentication no
#PermitEmptyPasswords no


I have a whole bunch of systems running 14.1-RELEASE and don't have this issue on any of them. I cannot reproduce the problem. On a client network there are about 20 13.3-RELEASE systems and can't reproduce it there either.
THis issue has nothing to do with 13 or 14. Just one day it started happening. This is a freak thing I've never encountered before.
 
Maybe a bit of a long shot, but what's in /etc/hosts.allow, and do you perhaps still have a /etc/hosts.deny?
 
Maybe a bit of a long shot, but what's in /etc/hosts.allow, and do you perhaps still have a /etc/hosts.deny?
No /etc/hosts.deny.

hosts.allow
Code:
# hosts.allow access control file for "tcp wrapped" applications.
#
# NOTE: The hosts.deny file is deprecated.
#       Place both 'allow' and 'deny' rules in the hosts.allow file.
#       See hosts_options(5) for the format of this file.
#       hosts_access(5) no longer fully applies.
#
#   _____                                      _          _
#  | ____| __  __   __ _   _ __ ___    _ __   | |   ___  | |
#  |  _|   \ \/ /  / _` | | '_ ` _ \  | '_ \  | |  / _ \ | |
#  | |___   >  <  | (_| | | | | | | | | |_) | | | |  __/ |_|
#  |_____| /_/\_\  \__,_| |_| |_| |_| | .__/  |_|  \___| (_)
#                                     |_|
# !!! This is an example! You will need to modify it for your specific
# !!! requirements!

sshd : /var/db/sentry/hosts.deny : deny
sshd : ALL : spawn perl /var/db/sentry/sentry.pl -c --ip=%a : allow

# Start by allowing everything (this prevents the rest of the file
# from working, so remove it when you need protection).
# The rules here work on a "First match wins" basis.
ALL : ALL : allow

# Wrapping sshd(8) is not normally a good idea, but if you
# need to do it, here's how
#sshd : .evil.cracker.example.com : deny

# Protect against simple DNS spoofing attacks by checking that the
# forward and reverse records for the remote host match. If a mismatch
# occurs, access is denied, and any positive ident response within
# 20 seconds is logged. No protection is afforded against DNS poisoning,
# IP spoofing or more complicated attacks. Hosts with no reverse DNS
# pass this rule.
ALL : PARANOID : RFC931 20 : deny

# Allow anything from localhost.  Note that an IP address (not a host
# name) *MUST* be specified for rpcbind(8).
ALL : localhost 127.0.0.1 : allow
# Comment out next line if you build libwrap without IPv6 support.
ALL : [::1] : allow
#ALL : my.machine.example.com 192.0.2.35 : allow

# To use IPv6 addresses you must enclose them in []'s
#ALL : [fe80::%fxp0]/10 : allow
#ALL : [fe80::]/10 : deny
#ALL : [2001:db8:2:1:2:3:4:3fe1] : deny
#ALL : [2001:db8:2:1::]/64 : allow

# Sendmail can help protect you against spammers and relay-rapers
sendmail : localhost : allow
#sendmail : .nice.guy.example.com : allow
#sendmail : .evil.cracker.example.com : deny
sendmail : ALL : allow

# Exim is an alternative to sendmail, available in the ports tree
exim : localhost : allow
#exim : .nice.guy.example.com : allow
#exim : .evil.cracker.example.com : deny
exim : ALL : allow

# Rpcbind is used for all RPC services; protect your NFS!
# Rpcbind should be running with -W option to support this.
# (IP addresses rather than hostnames *MUST* be used here)
#rpcbind : 192.0.2.32/255.255.255.224 : allow
#rpcbind : 192.0.2.96/255.255.255.224 : allow
rpcbind : ALL : deny

# NIS master server. Only local nets should have access
# (Since this is an RPC service, rpcbind needs to be considered)
ypserv : localhost : allow
#ypserv : .unsafe.my.net.example.com : deny
#ypserv : .my.net.example.com : allow
ypserv : ALL : deny

# Provide a small amount of protection for ftpd
ftpd : localhost : allow
#ftpd : .nice.guy.example.com : allow
#ftpd : .evil.cracker.example.com : deny
ftpd : ALL : allow

# You need to be clever with finger; do _not_ backfinger!! You can easily
# start a "finger war".
fingerd : ALL \
        : spawn (echo Finger. | \
         /usr/bin/mail -s "tcpd\: %u@%h[%a] fingered me!" root) & \
        : deny

# The rest of the daemons are protected.
ALL : ALL \
        : severity auth.info \
        : twist /bin/echo "You are not welcome to use %d from %h."
 
Code:
sshd : /var/db/sentry/hosts.deny : deny 
sshd : ALL : spawn perl /var/db/sentry/sentry.pl -c --ip=%a : allow
I think we might be getting somewhere. What if you put ALL : ALL : allow above those lines? Do you still get the odd behavior of sshd(8)?
 
I think we might be getting somewhere. What if you put ALL : ALL : allow above those lines? Do you still get the odd behavior of sshd(8)?
You are suggesting this:
Code:
ALL : ALL : allow
sshd : /var/db/sentry/hosts.deny : deny
sshd : ALL : spawn perl /var/db/sentry/sentry.pl -c --ip=%a : allow

That top line .. ALL:ALL:allow was already below the above 3 lines ...

Code:
ALL : ALL : allow
sshd : /var/db/sentry/hosts.deny : deny
sshd : ALL : spawn perl /var/db/sentry/sentry.pl -c --ip=%a : allow

# Start by allowing everything (this prevents the rest of the file
# from working, so remove it when you need protection).
# The rules here work on a "First match wins" basis.
ALL : ALL : allow

After posting the above, disabled the crontab line to restart sshd...

#!/bin/sh
## Restart sshd periodically as it seems to get stuck as of Novemenber/23
/etc/rc.d/sshd restart


Now wait to see if sshd behaves nicely
Thanks!
 
That top line .. ALL:ALL:allow was already below the above 3 lines ...
Yes, but first match wins, so we've effectively disabled that script for sshd(8). I suspect that script logs the IP address in /var/db/sentry/hosts.deny which causes the next ssh connection from that IP address to be denied. Those two lines are the reason why you can only make a single connection.
 
Hi SirDice, what you suggested must have worked. Can restart sshd and login multiple times without any issue.
Thank you so much!
 
Back
Top