Strange problem with ssh and reaching accounts.google.com

S

sjohn

I've been banging my head on this problem and want to know if anyone else has any ideas.
PRETTY_NAME="FreeBSD 13.1-RELEASE-p5"

So I have 2 problems that may or may not be related and I have a FreeBSD bare metal server in Dallas and clients FreeBSD/Dell r720 Debain/r620 in the office.
I'll start with the ssh symptoms first since they are pretty reproducible and I have rebooted both the FreeBSD server and the FreeBSD client in the office.

I can't ssh from the FreeBSD client to the server directly but I have a VM on the server with ipfw forwarding port 55522 on the server to the VM on port 22 and I can ssh directly to that.
It doesn't seem to be a DNS issue as ssh directly by IP address doesn't work either.
I can connect from the FreeBSD client to the Debian machine in the same office and the Debian machine can ssh directly to the FreeBSD server.
Running a second sshd damon on the server port 62222 on the server ssh from FreeBSD client does not work but ssh from the Debian machine does work.
I have a ip dectection daemon running on the server (to assist with DDNS) and running the ipdect client on the FreeBSD client works eg socket connection in perl works.

I wonder if I should enable packet reassembly in the ipfw rules? I'll include an edited version of ipfw.rules below.

The other strange thing is I can access most sites on the internet including google.com and my own mail server (the vm on the FreeBSD server) but not mail.google.com which hangs on loading accounts.google.com. I have purged all my history. Nightly/LibreWolf also doesn't work with accounts.google.com but if I enable tor socks 127.0.0.1 :9050 I am able to use gmail normally. I did have bind running as a caching name server on the client as well as dnsmasq on 127.0.0.1 to handle /etc/hosts and it worked great for several months, and when this problem started I had made no changes. I have tried disabling dnsmasq and now running bind with resolvconf disabled and resolv.conf nameserver set to 127.0.0.1 and nslookups work well.

Both of these issues started at the same time. I work in networking and have been running FreeBSD for 25 years but am at a loss as to where to go from here, any ideas?

SJohn
 

Attachments

  • ipfw5rules.txt
    5.9 KB · Views: 41
ssh directly to the bare metal server is what you are trying to do?
If so, do you have sshd on the server listening on port 22?
If so, do you have rules in place that allow inbound connections to port 22 from your client address?
 
mer said:
ssh directly to the bare metal server is what you are trying to do?
If so, do you have sshd on the server listening on port 22?
If so, do you have rules in place that allow inbound connections to port 22 from your client address?
Click to expand...
Yes that is what I am trying to do, it just stopped working but only stopped working from one FreeBSD machine.
Yes sshd is running on port 22, yes the firewall allows connections, I am able to ssh from a Debian machine in the same office to the server.
Yes, I have inbound rules to allow port 22 on the server, I have included the ipfw.rules file with obvious IP addresses change for privacy.

Thanks in advance.
 
Thanks. I figured that was all good, but double checking doesn't hurt.
have you tried adding "-vvv" to the client ssh command?
Last time I ran into something like this root cause was default ciphers allowed. Things were deprecated in interest of security, one side was upgraded, the other wasn't adding verbosity to the client invocation showed that issue up quickly.
What versions of the client are across the various machines? Both working and not working. ssh -V that may give a clue.
 
mer said:
Thanks. I figured that was all good, but double checking doesn't hurt.
have you tried adding "-vvv" to the client ssh command?
Last time I ran into something like this root cause was default ciphers allowed. Things were deprecated in interest of security, one side was upgraded, the other wasn't adding verbosity to the client invocation showed that issue up quickly.
What versions of the client are across the various machines? Both working and not working. ssh -V that may give a clue.
Click to expand...
Good idea, tried that ... looks like it just doesn't get a response, like there is a dumb appliance which thinks this connection is not allowed and blocks it.

OpenSSH_9.3p1, OpenSSL 1.1.1t-freebsd 7 Feb 2023
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 20: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/root/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/root/.ssh/known_hosts2'
debug2: resolving "example.com" port 22
debug3: resolve_host: lookup example.com:22
debug3: ssh_connect_direct: entering
debug1: Connecting to example.com [199.71.214.193] port 22.
debug3: Fssh_set_sock_tos: set socket 4 IP_TOS 0x48
debug1: connect to address 111.222.111.222 port 22: Operation timed out
ssh: connect to host example.com port 22: Operation timed out

I also am able to connect from a VM on the FreeBSD client to the FreeBSD server, no problem.
I tried changing my network interfaces a bit swapped bge0 and bge1, cleaner anyway since it matches the external lan and the vm lan, still no luck.
Maybe some device in the office but heck I checked the microtik router and nope the boss didn't add deny client to server port 22, which would pretty much be what would be required to cause this problem but allow all other machines to work. Hmm maybe try change my local IP address, wait I try and then come back and edit this.
 
Okay I fixed it but .... err not really I had to change my IP address.

I changed my ip from ending in .231 to .10 and the ssh now works and so does gmail.
I really have no idea other than it appears to be network related (external) and probably nothing to do with either machine.
Had to change configuration in a few places, expected if changing IP address to get bind to work right.
I wouldn't call this solved, but that is the work around, and why did it affect both account.google.com and ssh to 1 server specifically since ssh to other external PBX machines that I manage was working fine as well.

Hmmm maybe one of the switches is screwing, but why only these services, must be a layer 3 or higher device, layer 3 at least to filter IP and port, and for gmail only layer 7?
As far as I know we don't have any such equipment on site.

Go Figure!

Thanks for the help though! Cheers!
 
Yeah, and the new IP stopped working too after awhile. I checked the Microtik router and seems at one point there was a machine on the address I used and there is a MAC assigned, so I changed address again and everything works again ... so far. Seems to me it has to be a layer 7 device ... ssh from ports 22 and 62222 to the server didn't work but ssh from other machines and from the problem machine to a VM on the server with the same IP using an ipfw forward from port 55522 to 22 did work. The strange thing is it worked when I changed the IP then stopped working again though that might have something to do with the DHCP entry with a different MAC. I did ping the address I wanted to use before using it but who knows, it could be in use. I am still thinking smart firewall device ... we are on a domain, I don't think Windows Server has this functionality? We are trialing a splynx server for customer management as well but that router isn't configured to use it.
 
the machines that work, are they in a different subnet? Back up in #7 sounds like you changed the last octet, perhaps the upstream block is on a whole subnet.
Maybe flip the IPs on a working machine with the not working one and see if its really IP related or "incoming port" related.
 
You must log in or register to reply here.
Back
Top