FreeBSD does not return you to local prompt when disconnecting

A

anbj

Compare this video of an ssh-session being terminated by the issue of shutdown -p now.

View: https://www.youtube.com/watch?v=h6co9J2DQdM


OpenBSD terminates cleanly and you're returned to the local prompt.
FreeBSD just keeps you hanging. Why is this?

Also, both shutdown and reboot is located in /sbin. Why does which only report the path to reboot?

----
$ which reboot
/sbin/reboot
$ which shutdown
$
----
 
It takes a while for the SSH session to time-out. Compare these settings:
Code: 
     ConnectionAttempts
             Specifies the number of tries (one per second) to make before
             exiting.  The argument must be an integer.  This may be useful in
             scripts if the connection sometimes fails.  The default is 1.

     ConnectTimeout
             Specifies the timeout (in seconds) used when connecting to the
             SSH server, instead of using the default system TCP timeout.
             This value is used only when the target is down or really
             unreachable, not when it refuses the connection.
Code: 
     ServerAliveCountMax
             Sets the number of server alive messages (see below) which may be
             sent without ssh(1) receiving any messages back from the server.
             If this threshold is reached while server alive messages are
             being sent, ssh will disconnect from the server, terminating the
             session.  It is important to note that the use of server alive
             messages is very different from TCPKeepAlive (below).  The server
             alive messages are sent through the encrypted channel and
             therefore will not be spoofable.  The TCP keepalive option
             enabled by TCPKeepAlive is spoofable.  The server alive mechanism
             is valuable when the client or server depend on knowing when a
             connection has become inactive.

             The default value is 3.  If, for example, ServerAliveInterval
             (see below) is set to 15 and ServerAliveCountMax is left at the
             default, if the server becomes unresponsive, ssh will disconnect
             after approximately 45 seconds.

     ServerAliveInterval
             Sets a timeout interval in seconds after which if no data has
             been received from the server, ssh(1) will send a message through
             the encrypted channel to request a response from the server.  The
             default is 0, indicating that these messages will not be sent to
             the server.
Code: 
     TCPKeepAlive
             Specifies whether the system should send TCP keepalive messages
             to the other side.  If they are sent, death of the connection or
             crash of one of the machines will be properly noticed.  However,
             this means that connections will die if the route is down
             temporarily, and some people find it annoying.

             The default is yes (to send TCP keepalive messages), and the
             client will notice if the network goes down or the remote host
             dies.  This is important in scripts, and many users want it too.

             To disable TCP keepalive messages, the value should be set to no.
             See also ServerAliveInterval for protocol-level keepalives.
 
SirDice said:
It takes a while for the SSH session to time-out. Compare these settings:
Click to expand...

Yes, if you have keepalive active, then it should terminate, after quite some time.
But what we see in the other video, is an immediate Connection closed by remote host.

What they might be doing, is, kill the running sshd instances. (Is there an other means to actively log out the current users?)

Whereas, what we do, is, work thru the rc.d shutdown scripts. And some of these scripts will take down the interfaces. Then the active session's sshd are still lingering in memory, and will be killed in a later step, but then cannot close the TCP connection anymore.

This could be fixed, if, when stopping the sshd service, all active daemons would also be killed. But then, you do not always want that behaviour.
 
Oh, there's very much a difference between reboot(8) and shutdown(8). A reboot doesn't stop your services, it just kills the processes. A shutdown -r now will actually run all service stop scripts (in reverse order). Do NOT use the reboot(8) command unless you absolutely have to.

I have no idea how things work on OpenBSD. They might be killing services or have some other way of defining order of scripts (we have rcorder(8) for that). Also note that it's a 13.0-CURRENT in the video. It may have been something that's been fixed already. Note that -CURRENT is now 14.0, 13.0-CURRENT doesn't exist anymore (13/stable has been branched off).
 
anbj said:
Also, both shutdown and reboot is located in /sbin. Why does which only report the path to reboot?
Click to expand...
On my installation, either none or both are reported.

But then, permissions are quite different, so it may be, what You get reported is just what You are allowed to invoke as the current user:
Code: 
$ ls -la /sbin/reboot /sbin/shutdown 
-r-xr-xr-x  4 root  wheel     10904 Dec 11 17:44 /sbin/reboot
-r-sr-xr--  2 root  operator  15912 Dec 11 17:44 /sbin/shutdown
 
(FreeBSD) reboot does it cleanly:

----
$ doas reboot
Connection to freebsd closed by remote host.
Connection to freebsd closed.
----
 
Not cleanly.
reboot(8):
Code: 
The halt and reboot utilities flush the file system cache to disk,	send
     all running processes a SIGTERM (and subsequently a SIGKILL) and, respec-
     tively, halt or restart the system.

Compare this with shutdown(8) that sends a signal to init(8):
Code: 
     When shutting down the machine, init will try to run the /etc/rc.shutdown
     script.  This script can be used to cleanly terminate specific programs
     such as innd (the InterNetNews server).  If this script does not
     terminate within 120 seconds, init will terminate it.  The timeout can be
     configured via the sysctl(8) variable kern.init_shutdown_timeout.

Now, that said, as PMc noted it's possible there are some ordering issues at play here. The /etc/rc.shutdown scripts gets a list of scripts using rcorder(8) then reverses that list. Each script is then issued a faststop command.
Code: 
files=`rcorder ${rcorder_opts} /etc/rc.d/* ${local_rc} 2>/dev/null`

for _rc_elem in `reverse_list $files`; do
        debug "run_rc_script $_rc_elem faststop"
        run_rc_script $_rc_elem faststop
done

Either /etc/rc.d/sshd doesn't cleanly stop running sshd(8) processes, or NETWORKING is shutdown before sshd. I'm actually leaning towards the first. If you restart the sshd service remotely for example your current session keeps running and doesn't actually get stopped and started. So something in sshd(8) is keeping it alive until it gets its rug pulled out from under it.
 
SirDice said:
Either /etc/rc.d/sshd doesn't cleanly stop running sshd(8) processes
Click to expand...
It does not at all stop them.
And that's usually what you want, because when changing some sshd configuration and then doing service sshd restart, you would not want all your running sessions being killed.

anbj said:
(FreeBSD) reboot does it cleanly:

----
$ doas reboot
Connection to freebsd closed by remote host.
Connection to freebsd closed.
----
Click to expand...
That is to be expected. reboot does not execute rc.d scripts, and kills the processes immediately. So the network interfaces are not brought down, and the sshd processes can communicate their termination to the peer.

But then, the behaviour of the ssh sessions is the least important thing here. The important question is: how do you need to bring down your services, in order to have them cleanly come up again. And that, obviousely, depends on the kind of services installed.
 
PMc said:
It does not at all stop them.
And that's usually what you want, because when changing some sshd configuration and then doing service sshd restart, you would not want all your running sessions being killed.
Click to expand...
With a restart, yes. I wouldn't like to have my remote session killed with that. But for a stop it should definitely stop the processes but apparently that's not the case. Although an absentminded service sshd stop would be rather annoying if you're working remotely. So they might err on the safe side and just keep your current session alive.
 
Thank you all for great answers.

When I say 'cleanly', I guess I mean 'apparently cleanly'; as in; my session was closed and I am returned to my local prompt. This just has a better feel to it compared to ending up with a seemingly nonresponding connection that I have to force close. My thoughts did not include any tubes and plumbing behind the scenes.

Thank you SirDice et al., for educating me.
 
anbj, I had a similar issue some months ago on one of my machines, which had been upgraded to a more recent version at that time. Apparently, on shutdown, the sshd mother process was taken down in an orderly fashion, but its child processes which maintain the connection were not, and therefore ssh clients did not receive the session terminated signal and would hang until timeout. At that time I was hurrying on a tight schedule, and I did not investigate this further, but simply put the following command into /etc/rc.shutdown.local

/usr/bin/killall -TERM sshd

That hack resolved the issue quickly but dirty. I do not remember anymore all the details. I think this happened with 12.1-RELEASE, but I may be wrong, because I experimented also with some 13.0-CURRENT snapshots at that time.

Anyway, with the latest 12.2-RELEASE and 13.0-ALPHA2 said hack is no more necessary, since everything behaves normal, i.e. no hang of a ssh client on shutdown.
 
Just for completeness;

After waiting a couple of seconds and pressing enter, ssh understands this connection is gone with a packet_write_wait: Connection to 192.168.10.3 port 22: Broken pipe

Code: 
[root@nuc10 /usr/src]# shutdown -r now
Shutdown NOW!
shutdown: [pid 48551]
[root@nuc10 /usr/src]#
*** FINAL System shutdown message from anbj@nuc10 ***


System going down IMMEDIATELY






System shutdown time has arrived
packet_write_wait: Connection to 192.168.10.3 port 22: Broken pipe

This is on
Code: 
$ uname -a
FreeBSD nuc10 13.0-ALPHA2 FreeBSD 13.0-ALPHA2 #3 stable/13-c256213-g03d837b565a: Wed Jan 27 09:08:35 CET 2021     anbj@nuc10:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64
 
Delighted to see that on FreeBSD 14.0-CURRENT/FreeBSD 13*, it gets you back to your local prompt without having to time out (or whatever happens).
 
Last edited:
You must log in or register to reply here.
Back
Top