No reply to SSH request

fitz

fitz

Hello,

I have recently built/rebuilt my first FreeBSD server, which, has up until now been working wonderfully. However, now when I attempt to ssh to the server, I get no response. No request for a password. I have to manually/hard boot the server to get to a point where I can ssh successfully into the server again.

If I leave the server for 30+ minutes - the power is on, but there is no response from the server after typing in ssh fitz@192.168.9.10.
 
Depending on the exact message a few things could be wrong. If you get a connection time-out there may be a firewall that's blocking access. If you get "connection refused" it may be that your SSH daemon is not running (maybe it crashed). When SSH isn't working are you still able to ping(8) the machine?
 
Also try one or more -v flags to SSH to troubleshoot in more detail:
Code: 
     -v      Verbose mode.  Causes ssh to print debugging messages about its
             progress.  This is helpful in debugging connection, authentication,
             and configuration problems.  Multiple -v options increase the ver‐
             bosity.  The maximum is 3.
 
Thanks DutchDaemon -v is a little gem to know about rather than staring at a cursor! It has given me the following response:
Code: 
OpenSSH_6.4, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 51: Applying options for *
debug1: Connecting to 192.168.98.10 [192.168.98.10] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1_hpn13v11 FreeBSD-20140420
debug1: match: OpenSSH_6.6.1_hpn13v11 FreeBSD-20140420 pat OpenSSH*
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 2b:b3:23:ab:22:ac:52:ad:58:ab:40:76:22:b0:1a:cd
debug1: Host '192.168.98.10' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/id_rsa
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Next authentication method: keyboard-interactive

Any ideas please?
 
Have you tried typing your username, or your username and password in the blind? You're apparently connected. The other side may be waiting for a DNS timeout as well, but that should not be thirty minutes.

Also try a telnet to port 22. You should see, at the very least, the server's version string
Code: 
OpenSSH_6.6.1_hpn13v11 FreeBSD-20140420
 
Just tried the username and password blind - no luck...

And the result of telnet is
Code: 
[root@master Master]# telnet 192.168.98.10 22
Trying 192.168.98.10...
Connected to 192.168.98.10.
Escape character is '^]'.
SSH-2.0-OpenSSH_6.6.1_hpn13v11 FreeBSD-20140420

Protocol mismatch.
Connection closed by foreign host.
Thanks :)
 
The host may be having an issue. I've had this happen on a few heavily loaded servers. You'd get the banner but everything else just stalls. It could be disk related, if one drive has failed for example, the system would block everything and waits indefinitely for the disk to respond to I/O. It's also possible the server has run out of memory, and I mean really run out, no memory free and a full swap. That would also result in connections seemingly hanging.
 
Other than that, the 'protocol mismatch' may be problematic, though I don't believe either your client or your server are trying to use a Protocol 1 nowadays.
 
Thanks SirDice & DutchDaemon

I have configured it so that it does a SMART HDD scan every morning at 03:00 - and this has been OK so far - but I do check daily at /var/mail/root.

Knowing that I can hard boot it and then successfully log in and coming from Linux (thus being a newbie) - which logs should/can I read to check out what has been going on?

I have checked through some in /var/mail/ but, it doesn't seem like I have found the 'right' logs to check.
 
Usually /var/log/messages is the log to check (sometimes /var/log/debug.log has something interesting). You can also temporarily (and manually) run sshd -d on the server to see what it produces, preferably in a tmux or screen session.
 
Looks like authentication does not work. Delete or temporarily rename ~/.ssh then try again. The SSH client seems to be a non FreeBSD system on the LAN which shows a lower SSH version. Post its /etc/ssh/ssh_config. The SSH server seems to be an actual FreeBSD system. Post its /etc/ssh/sshd_config (Note: sshd).

Have you ever tried to ssh from the server to the client? Does that work? Are there any firewalls on client and/or server or between? Next to ssh -v there is ssh -vv which is more verbose and ssh -vvv which is maximal verbose.

On my boxes the most verbose SSH logs can been found in /var/log/auth.log.
 
Hurumph...

Now I cannot ssh in at all - thus no news regarding any logs.

I've tried a hard boot a couple of times.

Some Services are running on the Server as I am able to mount NFS shares which I can then access, but, the squid proxy server service is down for example.

From a different client (running Xubuntu) - the following SSH response:

Code: 
root@hpnote:/home/fitz# ssh -v fitz@192.168.98.10
OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 192.168.98.10 [192.168.98.10] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2

and using -vvv getopt

Code: 
root@hpnote:/home/fitz# ssh -vvv fitz@192.168.98.10
OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 192.168.98.10 [192.168.98.10] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2

Could anyone advise as to how I might be able to log back on in order to view my logs and hopefully resolve the problem?

Many thanks getopt and DutchDaemon
 
I'd reboot the server once more, then immediately login. While you have a connection use something like sysutils/tmux to keep several screens open at the same time. Run top(1) and tail(1) a few logfiles. Keep an eye on it until it starts losing connectivity. Hopefully you'd be able to see what's happening when the server starts stalling.
 
Thanks SirDice - got in!

I've posted some logs:

getopt said:
Post its /etc/ssh/sshd_config (Note: sshd)
Click to expand...

Code: 
#  $OpenBSD: sshd_config,v 1.93 2014/01/10 05:59:19 djm Exp $
#  $FreeBSD: releng/10.1/crypto/openssh/sshd_config 264692 2014-04-20 12:4$

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

# Note that some of FreeBSD's defaults differ from OpenBSD's, and
# FreeBSD has a few additional options.

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0

getopt said:
On my boxes the most verbose SSH logs can been found in /var/log/auth.log
Click to expand...

Code: 
Dec  7 23:48:18 backup5 sshd[902]: Connection closed by 192.168.98.99 [preauth]
Dec  7 23:48:47 backup5 sshd[907]: error: PAM: authentication error for root from 192.168.98.99
Dec  7 23:48:51 backup5 last message repeated 2 times
Dec  7 23:48:51 backup5 sshd[907]: Connection closed by 192.168.98.99 [preauth]
Dec  7 23:49:00 backup5 sshd[912]: Accepted keyboard-interactive/pam for fitz from 192.168.98.99 port 32778 ssh2
Dec  7 23:49:06 backup5 su: fitz to root on /dev/pts/0
Dec  7 23:51:24 backup5 webmin[930]: Successful login as admin from 192.168.98.99
Dec  7 23:53:38 backup5 sshd[915]: Received disconnect from 192.168.98.99: 11: disconnected by user
Dec  7 23:55:08 backup5 sshd[1068]: Accepted keyboard-interactive/pam for fitz from 192.168.98.99 port 32910 ssh2
Dec  7 23:55:13 backup5 su: fitz to root on /dev/pts/0
Dec  7 23:57:14 backup5 sshd[1071]: Received disconnect from 192.168.98.99: 11: disconnected by user
Dec  8 03:23:22 backup5 sendmail[3199]: gethostby*.gethostanswer: asked for "backup5.localdomain IN AAAA", got type "A"
Dec  8 03:23:28 backup5 sendmail[3148]: gethostby*.gethostanswer: asked for "backup5.localdomain IN AAAA", got type "A"
Dec  8 03:23:28 backup5 sendmail[3171]: gethostby*.gethostanswer: asked for "backup5.localdomain IN AAAA", got type "A"
Dec  8 03:23:28 backup5 sendmail[3172]: gethostby*.gethostanswer: asked for "backup5.localdomain IN AAAA", got type "A"
Dec  8 03:23:28 backup5 sendmail[3171]: gethostby*.gethostanswer: asked for "10.98.168.192.in-addr.arpa IN PTR", got type "A"
Dec  8 03:40:12 backup5 sshd[3242]: Accepted keyboard-interactive/pam for fitz from 192.168.98.99 port 33328 ssh2
Dec  8 03:40:16 backup5 su: fitz to root on /dev/pts/0
Dec  8 03:43:45 backup5 sshd[3245]: Received disconnect from 192.168.98.99: 11: disconnected by user
Dec  8 04:33:02 backup5 sshd[781]: Server listening on :: port 22.
Dec  8 04:33:02 backup5 sshd[781]: Server listening on 0.0.0.0 port 22.
Dec  8 04:33:13 backup5 sshd[851]: Accepted keyboard-interactive/pam for fitz from 192.168.98.99 port 57566 ssh2
Dec  8 04:33:18 backup5 su: fitz to root on /dev/pts/0
Dec  8 04:41:33 backup5 sshd[760]: Server listening on :: port 22.
Dec  8 04:41:33 backup5 sshd[760]: Server listening on 0.0.0.0 port 22.
Dec  8 04:41:50 backup5 sshd[829]: Accepted keyboard-interactive/pam for fitz from 192.168.98.99 port 57729 ssh2
Dec  8 04:43:04 backup5 su: fitz to root on /dev/pts/0
Dec  9 08:37:45 backup5 sshd[3937]: Accepted keyboard-interactive/pam for fitz from 192.168.98.99 port 34337 ssh2
Dec  9 08:37:50 backup5 su: fitz to root on /dev/pts/0
Dec  9 08:42:08 backup5 sshd[775]: Server listening on :: port 22.
Dec  9 08:42:08 backup5 sshd[775]: Server listening on 0.0.0.0 port 22.
Dec  9 08:47:19 backup5 sshd[850]: Accepted keyboard-interactive/pam for fitz from 192.168.98.99 port 34537 ssh2
Dec  9 08:47:25 backup5 su: fitz to root on /dev/pts/0
Dec  9 08:52:15 backup5 sshd[853]: Received disconnect from 192.168.98.99: 11: disconnected by user
Dec  9 09:20:24 backup5 sshd[772]: Server listening on :: port 22.
Dec  9 09:20:24 backup5 sshd[772]: Server listening on 0.0.0.0 port 22.
Dec  9 09:21:49 backup5 sshd[842]: Accepted keyboard-interactive/pam for fitz from 192.168.98.99 port 46359 ssh2
Dec  9 09:21:53 backup5 su: fitz to root on /dev/pts/0
Dec  9 09:22:30 backup5 sshd[845]: Received disconnect from 192.168.98.99: 11: disconnected by user
Dec 10 13:25:16 backup5 sshd[757]: Server listening on :: port 22.
Dec 10 13:25:16 backup5 sshd[757]: Server listening on 0.0.0.0 port 22.
Dec 11 06:17:18 backup5 sshd[740]: Server listening on :: port 22.
Dec 11 06:17:18 backup5 sshd[740]: Server listening on 0.0.0.0 port 22.
Dec 11 06:17:40 backup5 sshd[810]: Accepted keyboard-interactive/pam for fitz from 192.168.98.127 port 35500 ssh2
Dec 11 06:17:48 backup5 su: fitz to root on /dev/pts/0

DutchDaemon said:
Usually /var/log/messages is the log to check
Click to expand...

Code: 
Nothing to note here

So I tried /var/mail/root
Code: 
From root@backup5.localdomain Mon Dec  8 03:23:22 2014
Return-Path: <root@backup5.localdomain>
Received: from backup5.localdomain (localhost [127.0.0.1])
  by backup5.localdomain (8.14.9/8.14.9) with ESMTP id sB83NMc1003201
  (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO)
  for <root@backup5.localdomain>; Mon, 8 Dec 2014 03:23:22 GMT
  (envelope-from root@backup5.localdomain)
Received: (from root@localhost)
  by backup5.localdomain (8.14.9/8.14.9/Submit) id sB83NMlB003199
  for root; Mon, 8 Dec 2014 03:23:22 GMT
  (envelope-from root)
Date: Mon, 8 Dec 2014 03:23:22 GMT
From: Charlie Root <root@backup5.localdomain>
Message-Id: <201412080323.sB83NMlB003199@backup5.localdomain>
To: root@backup5.localdomain
Subject: backup5.localdomain daily run output


Removing stale files from /var/preserve:

Cleaning out old system announcements:

Removing stale files from /var/rwho:

Backup passwd and group files:

Verifying group file syntax:
/etc/group is fine

Backing up mail aliases:

Disk status:
Filesystem  Size  Used  Avail Capacity  Mounted on
zroot/ROOT/default  1.3T  4.4G  1.3T  0%  /
devfs  1.0K  1.0K  0B  100%  /dev
zroot/Media  5.3T  3.9T  1.3T  75%  /mnt/Media
zroot/tmp  1.3T  218K  1.3T  0%  /tmp
zroot/usr/home  1.3T  218K  1.3T  0%  /usr/home
zroot/usr/ports  1.3T  959M  1.3T  0%  /usr/ports
zroot/usr/src  1.3T  154K  1.3T  0%  /usr/src
zroot/var/crash  1.3T  154K  1.3T  0%  /var/crash
zroot/var/log  1.3T  793K  1.3T  0%  /var/log
zroot/var/mail  1.3T  160K  1.3T  0%  /var/mail
zroot/var/tmp  1.3T  154K  1.3T  0%  /var/tmp

Network interface status:
Name  Mtu Network  Address  Ipkts Ierrs Idrop  Opkts Oerrs  Coll  Drop
bge0  1500 <Link#1>  9c:b6:54:07:d3:75 146850030  0  0 76893877  0  0  0
bge0  - 192.168.98.0  192.168.98.10  146843454  -  - 76893507  -  -  -
lo0  16384 <Link#2>  1408  0  0  942  0  0  0
lo0  - localhost  ::1  4  -  -  4  -  -  -
lo0  - fe80::1%lo0  fe80::1%lo0  0  -  -  0  -  -  -
lo0  - your-net  localhost  126  -  -  938  -  -  -

Local system status:
3:01AM  up  3:14, 0 users, load averages: 1.44, 1.21, 1.22
Mail in local queue:
/var/spool/mqueue is empty
  Total requests: 0

Mail in submit queue:
/var/spool/clientmqueue is empty
  Total requests: 0

Security check:
  (output mailed separately)

Checking for rejected mail hosts:

Backing up pkgng database:

SMART status:
Checking health of /dev/ada0: OK
Checking health of /dev/ada1: OK
Checking health of /dev/ada2: OK
Checking health of /dev/ada3: OK
Checking health of /dev/ada4: OK

-- End of daily output --

From root@backup5.localdomain Mon Dec  8 03:23:28 2014
Return-Path: <root@backup5.localdomain>
Received: from backup5.localdomain (localhost [127.0.0.1])
  by backup5.localdomain (8.14.9/8.14.9) with ESMTP id sB83NSur003205
  (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO)
  for <root@backup5.localdomain>; Mon, 8 Dec 2014 03:23:28 GMT
  (envelope-from root@backup5.localdomain)
Received: (from root@localhost)
  by backup5.localdomain (8.14.9/8.14.9/Submit) id sB83NSv7003148
  for root; Mon, 8 Dec 2014 03:23:28 GMT
  (envelope-from root)
Date: Mon, 8 Dec 2014 03:23:28 GMT
From: Charlie Root <root@backup5.localdomain>
Message-Id: <201412080323.sB83NSv7003148@backup5.localdomain>
To: root@backup5.localdomain
Subject: backup5.localdomain daily security run output


Checking setuid files and devices:

Checking negative group permissions:

Checking for uids of 0:
root 0
toor 0

Checking for passwordless accounts:

Checking login.conf permissions:

backup5.localdomain kernel log messages:
+++ /tmp/security.hNR9Ryd2  2014-12-08 03:23:09.967381049 +0000
+CPU: AMD Turion(tm) II Neo N54L Dual-Core Processor (2196.40-MHz K8-class CPU)
+Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
+ugen1.1: <ATI> at usbus1
+Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
+ugen1.1: <ATI> at usbus1
+uhub0: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
+uhub1: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
+uhub2: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
+uhub3: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
+uhub4: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
+uhub5: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
+uhub3: 4 ports with 4 removable, self powered
+uhub1: 5 ports with 5 removable, self powered
+uhub5: 5 ports with 5 removable, self powered
+uhub0: 5 ports with 5 removable, self powered
+Timecounter "TSC-low" frequency 1098198663 Hz quality 800

backup5.localdomain login failures:
Dec  7 23:44:59 backup5 sshd[8175]: error: PAM: authentication error for root from 192.168.98.99
Dec  7 23:45:09 backup5 sshd[8182]: error: PAM: authentication error for root from 192.168.98.99
Dec  7 23:48:15 backup5 sshd[902]: error: PAM: authentication error for root from 192.168.98.99
Dec  7 23:48:47 backup5 sshd[907]: error: PAM: authentication error for root from 192.168.98.99

backup5.localdomain refused connections:

Checking for packages with security vulnerabilities:
Database fetched: Sun Dec  7 03:13:36 GMT 2014

-- End of security output --
Apologies if I have included too much code, but, being a newbie to FreeBSD, I didn't want to miss anything.

I've pretty much decided to re-install FreeBSD, now to a separate drive (a USB/KingstonData) and keep the 5xHDDs purely ZFS / DATA, but, if you could spot anything of any warning value, it would be greatly appreciated.

Or, indeed, if there is anything to add to the new configuration to log so that, should the same happen, I know where to look, that would be a great help also.

Many thanks DutchDaemon SirDice getopt :)
 
You must log in or register to reply here.
Back
Top