ssh server side not completing logins

P

ProServ

Hi, few days ago noticed I cannot ssh to FreeBSD 13.2 server. Have been able to for many years. Being very remote the only way I can think to get sshd to work again is to reboot server. Trying to avoid that as it is costly to get someone to do it. FTP logins work, web services are working. It just seems sshd doesn't work. Example:

Code: 
# ssh -p 10069 -v f.q.d.n.tld
OpenSSH_9.3p2, OpenSSL 1.1.1t-freebsd  7 Feb 2023
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to f.q.d.n [xx.xxx.xx.5] port 10069.
debug1: Connection established.
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa_sk type -1
debug1: identity file /root/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_ed25519_sk type -1
debug1: identity file /root/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /root/.ssh/id_xmss type -1
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.3 FreeBSD-20230719

Since we use keys only, there is no login prompt. The ssh keys on client side are valid and have been used for a long time.

Any idea on how I might be able to login?

Thanks.
 
It seems you might be missing a keyfile on the client you are connecting from.
All identity keys returns -1, if found on the client it should return 0.
something like:
Code: 
debug1: identity file /root/.ssh/id_rsa type 0
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1

in this example, the rsa key is found and used.
 
Root is not allowed to login remotely, with or without keys, PermitRootLogin defaults to 'no' and should be kept that way.
 
We have always allowed root login and have always had set:
Code: 
PasswordAuthentication no
ChallengeResponseAuthentication no

We have not changed /etc/ssh/sshd_config in the longest time. Probably since setting up the server years ago. It's just the last several days, sshd doesn't seem to answer.

When connecting it will never ask for password. Just deny.

Code: 
# ssh -p 19872 other.freebsd_server.tld
root@other.freebsd_server.tld: Permission denied (publickey).

Either one has the keys or not. In this case, it doesn't respond at all.
Thanks.
 
ProServ said:
Code: 
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa_sk type -1
debug1: identity file /root/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_ed25519_sk type -1
debug1: identity file /root/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /root/.ssh/id_xmss type -1
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
Click to expand...
BjarneB said:
It seems you might be missing a keyfile on the client you are connecting from.
All identity keys returns -1, if found on the client it should return 0.
Click to expand...

Did someone mess up the permissions/ownership of root's ~/.ssh directory? That's a common cause of failure. No keys are found, so nothing is used?
 
SirDice said:
Did someone mess up the permissions/ownership of root's ~/.ssh directory? That's a common cause of failure. No keys are found, so nothing is used?
Click to expand...
~> ll -d .ssh
drwx------ 2 name name 4096 Oct 30 01:05 .ssh
ll .ssh
total 172
drwx------ 2 name name 4096 Oct 30 01:05 .
drwxr-xr-x 99 name name 16384 Oct 31 07:28 ..
-rw------- 1 name name 3954 Aug 8 11:47 authorized_keys
-rw------- 1 name name 56 Dec 17 2018 config
-rw------- 1 name name 1679 Apr 13 2014 id_rsa
-rw-r--r-- 1 name name 401 Apr 13 2014 id_rsa.pub
-rw------- 1 name name 68502 Jun 8 13:22 known_hosts
-rw------- 1 name name 68058 Jun 7 18:10 known_hosts.old

Nothing is changed here. Something is wrong on the server. It doesn't matter which client I use either. Have about 25 servers available I can ssh to (with the exact same keys as above) and can get to all - as I have tried to be sure. But trying to ssh to the one FreeBSD Server, it just doesn't respond. But everything else on that FreeBSD server seems to be working except one web site as the login script complains it cannot connect to sql db.

Code: 
FATAL: Connection to Cacti database failed.
 
Hi *,
This is very strange. This morning, sshd answered and logged in as root. Have not been able to ssh to this server since the day this thread was created. The server was not rebooted as uptime shows no reboot for 26 days.

Messages does show:
Code: 
 error: Fssh_kex_exchange_identification: banner line contains invalid characters

/etc/ssh/sshd_config is not using any banner:
Code: 
# no default banner path
#Banner none

Also very suspicious, mysql server is gone. /var/db/mysql is present with the db's

pkg info has
Code: 
mysql80-client-8.0.33_3
but no server.

/var/log/authlog shows many attempts from a microsoft address
Code: 
 grep  20.206.169.124 /var/log/auth.log | wc -l
 6735

So lets start fixing microsoft from trying to break into this FreeBSD Server
Code: 
 pfctl -t bruteforce -T add 20.192.0.0/10


any ideas??
 

Attachments

  • 1699790708969.png
    1699790708969.png
    1.1 MB · Views: 47
After seeing a ton of messages saying invalid characters in banner for sshd, uncommented Banner none

The attempted to restart sshd:
Code: 
 # /etc/rc.d/sshd restart
Performing sanity check on sshd configuration.
Stopping sshd.
Waiting for PIDS: 22158

And now it is stuck there. Cannot open a new ssh connection to this server.

Pressed ^c
and then see what the PID is stuck on:

Code: 
 # ps auxw |grep 22158
root      22158   0.0  0.0   21068   5048  -  Is   02:43        0:00.01 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups (sshd)

Luckily still logged into this server. Not sure what is wrong.

???
 
grep 20.206.169.124 /var/log/auth.log | wc -l
6735

whois is your friend. It doesn't matter what a browser displays when attacker from that address hits 6735 times. Just block it. In this case blocked a /10
 
ProServ said:
whois is your friend.
Click to expand...

Thanks.

Code: 
% whois  20.206.169.124
% IANA WHOIS server
% for more information on IANA, visit http://www.iana.org
% This query returned 1 object

refer:        whois.arin.net

inetnum:      20.0.0.0 - 20.255.255.255
organisation: Administered by ARIN
status:       LEGACY

whois:        whois.arin.net

changed:      1994-10
source:       IANA

# whois.arin.net

NetRange:       20.192.0.0 - 20.255.255.255
CIDR:           20.192.0.0/10
NetName:        MSFT
NetHandle:      NET-20-192-0-0-1
Parent:         NET20 (NET-20-0-0-0-0)
NetType:        Direct Allocation
OriginAS:       
Organization:   Microsoft Corporation (MSFT)
RegDate:        2017-10-18
Updated:        2021-12-14
Ref:            https://rdap.arin.net/registry/ip/20.192.0.0



OrgName:        Microsoft Corporation
OrgId:          MSFT
Address:        One Microsoft Way
City:           Redmond
StateProv:      WA
PostalCode:     98052
Country:        US
RegDate:        1998-07-10
Updated:        2023-10-23
Comment:        To report suspected security issues specific to traffic emanating from Microsoft online services, including the distribution of malicious content or other illicit or illegal material through a Microsoft online service, please submit reports to:
Comment:        * https://cert.microsoft.com. 
Comment:       
Comment:        For SPAM and other abuse issues, such as Microsoft Accounts, please contact:
Comment:        * abuse@microsoft.com. 
Comment:       
…
 
ProServ said:
… Being very remote the only way I can think to get sshd to work again is to reboot server. Trying to avoid that as it is costly to get someone to do it. …
Click to expand...

ProServ said:
… This morning, sshd answered and logged in as root. …
Click to expand...

Not to suggest that you should rush to this, but if you reboot the server (whilst logged in as root), will doing so avoid the cost of getting someone else to do it?

NB your answer to <https://forums.freebsd.org/posts/628234> above may be relevant to how to proceed.
 
Well.... Def not a favela.... dclau

en.wikipedia.org

Campinas - Wikipedia

en.wikipedia.org en.wikipedia.org

Essentially (Campinas) is the Silicon Valley of Brazil....

My 2 cents = I have seen huge spike recently in ddos/bruteforce attempts in my DNS servers from Brazil literally I am about to ban all of Brazil IPs, that's how big of an issue is becoming (hacker revolution in Brazil?)...

Usually you get Germany, Russia, and China but first time I see such huge spike from Brazil.
 
GoNeFast_01 said:
Well.... Def not a favela.... dclau

en.wikipedia.org

Campinas - Wikipedia

en.wikipedia.org en.wikipedia.org

Essentially (Campinas) is the Silicon Valley of Brazil....

My 2 cents = I have seen huge spike recently in ddos/bruteforce attempts in my DNS servers from Brazil literally I am about to ban all of Brazil IP, that's how big of an issue is becoming (hacker revolution in Brazil?)...

Usually you get Germany, Russia, and China but first time I see such huge spike from Brazil.
Click to expand...
Ban Brazil for a couple of days if you don't have any customers there. I would consider N. Korea as well. But I walk in my own shoes, you in yours.
 
So you permitted 6735 faulty logins without blocking it right after the 3th attempt? I would suggest you to use some IPS like fail2ban or sshguard which monitor the logs and after N fail login they add the offensive IP address into the firewall table.
 
forums.freebsd.org

sshd stuck on restart

Have a problem with sshd restarting. It hangs.... # /etc/rc.d/sshd restart Performing sanity check on sshd configuration. Stopping sshd. Waiting for PIDS: 22158 # ps auxw |grep sshd root 22158 0.0 0.0 21068 5048 - Ss 02:43 0:00.01 sshd: /usr/sbin/sshd [listener] 0 of...
forums.freebsd.org forums.freebsd.org
 
VladiBG said:
So you permitted 6735 faulty logins without blocking it right after the 3th attempt? I would suggest you to use some IPS like fail2ban or sshguard which monitor the logs and after N fail login they add the offensive IP address into the firewall table.
Click to expand...
Or just add to sshd_config

StrictModes yes
MaxAuthTries 3
MaxSessions 2

??

Wouldn't this stop the 6735 am I missing something ? (I use fail2ban in my linux vps hosting asterisk server for flood protection and when I do I enable the ssh ddos setting in jail.conf)....

Is there something else as fail2ban for fbsd (I am seeing fail2ban in python39 app in pkg) for other ports outside of the ssh (sshguard you can use in other ports)?
 
GoNeFast_01 said:
Or just add to sshd_config

StrictModes yes
MaxAuthTries 3
MaxSessions 2

??

Wouldn't this stop the 6735 am I missing something ? (I use fail2ban in my linux vps hosting asterisk server for flood protection and when I do I enable the ssh ddos setting in jail.conf)....

Is there something else as fail2ban for fbsd (I am seeing fail2ban in python39 app in pkg) for other ports outside of the ssh (sshguard you can use in other ports)?
Click to expand...
It will only disconnect the client after MaxAuthTries but it won't block it for another new connect and attempt. fail2ban and sshguard monitor the log file for those fail attempts and put the offensive client ip address into firewall rule for specific time for example for 24h so no other attempts can be done from the same ip.
To avoid majority of the bot scan and security organizations in internet you can change the default ssh port from 22 to something else this will filter all automatic scan and won't fill your firewall table with those ip addresses. Another good approach is to use a jump host and allow only connection from a specific ip address.
 
VladiBG said:
To avoid majority of the bot scan and security organizations in internet you can change the default ssh port from 22 to something else...
Click to expand...
I found that this helped enormously with these types of scans. Of course, many will observe that this only stops the uninitiated, but so many of those scans do come from uninitiated script kiddies. If moving the server away from port 22 does not help, you may wish to consider port knocking (knockd(1)).
 
You must log in or register to reply here.
Back
Top