Solved SSH boomerang from bhyve host to Linux guest back to bhyve host

A

AndAlooongCameFreeBSD

Hi all,

First post. I run a Linux guest (192.0.2.200) in a FreeBSD 14 bhyve host (192.0.2.10). I'm connected to the host from a laptop in the 192.0.2.0/24 network, also via SSH if that matters. Static IPs only. When I try to SSH from the host to the guest something weird happens: SSH reaches the guest as expected but then somehow comes back to the host where the connection originated from. Consequently, the connection fails as the verbose SSH output below shows.

Is this similar to the network stack problem that VNET jails solve? Does IPFW perhaps do some sort of implicit redirection? Is there something wrong with my configuration? Did I overlook somehing in a man page or do I better prepare to spend 3 weeks reading source code if I want do understand what happens?

Here is my configuration and the verbose SSH output:

Code: 
Linux guest IP configuration:

1: enp0s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 13:37:42:4a:01:62 brd ff:ff:ff:ff:ff:ff
    inet 192.0.2.200/24 brd 192.0.2.255 scope global noprefixroute enp0s2
       valid_lft forever preferred_lft forever
    inet6 fe80::1337:1337:1337:1337/64 scope link noprefixroute
       valid_lft forever preferred_lft forever


FreeBSD bhyve host configuration and output:

# From /var/log/auth.log
<auth.info> host_running_bhyve sshd[23571]: Connection from 192.0.2.10 port 49998 on 192.0.2.200 port 22
<auth.info> host_running_bhyve sshd[23571]: Failed publickey for root from 192.0.2.10 port 49998 ssh2: ED25519 SHA256:Q5     Z7Ukyadayadayada
<auth.info> host_running_bhyve sshd[23571]: Connection closed by authenticating user root 192.0.2.10 port 49998 [preauth     ]


# From /var/log/messages
<kern.err> host_running_bhyve kernel: [92959] arp: 13:37:42:4a:01:62 is using my IP address 192.0.2.200 on bridge0!

# Relevant IPFW rule active on 192.0.2.10 which is "me" in this case:
# 00220 allow log logamount 5 tcp from any to me 22 setup via re0 keep-state :default

$ sysctl net.inet.ip.forwarding
net.inet.ip.forwarding: 0
$ sysctl net.link.tap.up_on_open
net.link.tap.up_on_open: 1


ifconfig_re0="inet 192.0.2.10/24"
defaultrouter="192.0.2.1"
cloned_interfaces="bridge0 tap0"
ifconfig_bridge0_descr="Bridge for LINUX VMs"
ifconfig_tap0_descr="tap for 1st LINUX VM"
# net.link.tap.up_on_open=1 so probably not necessary: ifconfig_tap0="up"
ifconfig_bridge0="inet 192.0.2.200/24 addm re0 addm tap0 up"


$ ifconfig re0
re0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
    ether 13:37:42:5e:a5:61
    inet 192.0.2.10 netmask 0xffffff00 broadcast 192.0.2.255
    media: Ethernet autoselect (100baseTX <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

$ ifconfig bridge0
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: Bridge for LINUX VMs
    options=0
    ether 13:37:42:10:c3:77
    inet 192.0.2.200 netmask 0xffffff00 broadcast 192.0.2.255
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 4 priority 128 path cost 2000000
    member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 55
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>

$ ifconfig tap0
tap0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: tap for 1st LINUX VM
    options=80000<LINKSTATE>
    ether 13:37:42:00:09:10
    groups: tap
    media: Ethernet 1000baseT <full-duplex>
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    Opened by PID 1337


$ ssh -i ~/.ssh/key2connect2LINUXmachine root@192.0.2.200
OpenSSH_9.5p1, OpenSSL 3.0.12 24 Oct 2023
debug1: Reading configuration data /home/host_running_bhyve/.ssh/config
debug1: /home/host_running_bhyve/.ssh/config line 6: Applying options for redhat01
debug1: /home/host_running_bhyve/.ssh/config line 30: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolve_canonicalize: hostname 192.0.2.200 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/host_running_bhyve/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/host_running_bhyve/.ssh/known_hosts2'
debug3: ssh_connect_direct: entering
debug1: Connecting to 192.0.2.200 [192.0.2.200] port 22.
debug3: Fssh_set_sock_tos: set socket 3 IP_TOS 0x48
debug1: Connection established.
debug1: identity file /home/host_running_bhyve/.ssh/key2connect2LINUXmachine type 3
debug1: identity file /home/host_running_bhyve/.ssh/key2connect2LINUXmachine-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.5
debug1: Remote protocol version 2.0, remote software version OpenSSH_9.5 FreeBSD-20231004
debug1: Fssh_compat_banner: match: OpenSSH_9.5 FreeBSD-20231004 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 192.0.2.200:22 as 'root'
debug3: Fssh_record_hostkey: found key type ED25519 in file /home/host_running_bhyve/.ssh/known_hosts:1
debug3: Fssh_load_hostkeys_file: loaded 1 keys from 192.0.2.200
debug1: Fssh_load_hostkeys: fopen /home/host_running_bhyve/.ssh/known_hosts2: No such file or directory
debug1: Fssh_load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: Fssh_load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: have matching best-preference key type ssh-ed25519-cert-v01@openssh.com, using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,kex-strict-c-v00@openssh.com
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,kex-strict-s-v00@openssh.com
debug2: host key algorithms: ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug3: Fssh_kex_choose_conf: will use strict KEX ordering
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:swFyakCEtX2ZkwLv7DuULDO8QNb5Tr0iT+yayLHfxZ4
debug3: Fssh_record_hostkey: found key type ED25519 in file /home/host_running_bhyve/.ssh/known_hosts:1
debug3: Fssh_load_hostkeys_file: loaded 1 keys from 192.0.2.200
debug1: Fssh_load_hostkeys: fopen /home/host_running_bhyve/.ssh/known_hosts2: No such file or directory
debug1: Fssh_load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: Fssh_load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '192.0.2.200' is known and matches the ED25519 host key.
debug1: Found key in /home/host_running_bhyve/.ssh/known_hosts:1
debug3: send packet: type 21
debug1: Fssh_ssh_packet_send2_wrapped: resetting send seqnr 3
debug2: Fssh_ssh_set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: Fssh_ssh_packet_read_poll2: resetting read seqnr 3
debug1: SSH2_MSG_NEWKEYS received
debug2: Fssh_ssh_set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/host_running_bhyve/.ssh/key2connect2LINUXmachine ED25519 SHA256:Q5F9Wbxi1wXgV0UM9Xy2crQWigYV/Z4iHIsSveffqkI explicit
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: Fssh_kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com,ssh-dss,ssh-rsa,rsa-sha2-256,rsa-sha2-512>
debug1: Fssh_kex_ext_info_check_ver: publickey-hostbound@openssh.com=<0>
debug1: Fssh_kex_ext_info_check_ver: ping@openssh.com=<0>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 53
debug3: input_userauth_banner: entering
#####################################################
-----------------------------------------------------
THIS BANNER IS SET IN THE BHYVE HOST'S sshd_config
AND TELLS ME THAT I'M SSH-BOOMERANGING BACK TO MYSELF
-----------------------------------------------------
#####################################################
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/host_running_bhyve/.ssh/key2connect2LINUXmachine ED25519 SHA256:Q5F9Wbxi1wXgV0UM9Xy2crQWigYV/Z4iHIsSveffqkI explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
root@192.0.2.200: Permission denied (publickey).
 
OK, that helped, thanks.
I think by having an IP address in my bridge configuration in rc.conf I created something what Cisco people would call a switched virtual interface (SVI).
Code: 
ifconfig_bridge0="inet 192.0.2.200/24 addm re0 addm tap0 up"

And as SirDice pointed out that caused an IP conflict since what is connected to the tap0 end of the bridge (the Linux VM) has the same IP address. Accidental arp spoofing so to speak.

I became a little obsessed with what exactly caused that boomerang effect back to the originating machine so I grepped that error message from /var/log/messages in the source code.

Code: 
<kern.err> host_running_bhyve kernel: [92959] arp: 13:37:42:4a:01:62 is using my IP address 192.0.2.200 on bridge0!

grep -r "is using my IP" /usr/src

That led me to arp(4):
Code: 
"arp: %x:%x:%x:%x:%x:%x is using my IP address %d.%d.%d.%d on %s!"
ARP has discovered another host on the local network which responds to
mapping requests for its own Internet address with a different Ethernet
address, generally indicating that two hosts are attempting to use the
same Internet address.

That also led me to /usr/src/sys/netinet/if_ether.c. The relevant part starts in line 917:
Code: 
917      if (!bridged && !carped && isaddr.s_addr == myaddr.s_addr &&
918        myaddr.s_addr != 0) {
919        ARP_LOG(LOG_ERR, "%*D is using my IP address %s on %s!\n",
920            ifp->if_addrlen, (u_char *)ar_sha(ah), ":",
921            inet_ntoa_r(isaddr, addrbuf), ifp->if_xname);
922        itaddr = myaddr;
923        ARPSTAT_INC(dupips);
924        goto reply;
925      }

Honestly, as an occasional C reader with rudimentary programming skills and some basic networking knowledge it would have taken me a year to fully understand what that file does in detail. But I could cheat with an old book that covers the Unix TCP/IP implementation. The code has changed a little more than just a bit during that last three decades it seems but it says:
that book on page 689 where the mid-90s version of if_ether.c is discussed said:
The sender's IP address and target IP address are copied into isaddr and itaddr.
Click to expand...

So itaddr is the target address which is the Linux VM where I tried to SSH into, assuming that itaddr in line 922 in the code above is the still the same itaddr from that book. That if-condition evaluated to true, otherwise I would not have that message in my /var/log/messages, so the code got executed and the variable containing my Linux VM's IP address, itaddr, was reassigned with myaddr, the IP of my bhyve host (I guess) if I interpret line 922 correctly.

Seems to make sense, at least for me.

I'd like to know what those packet types from the SSH debug logs mean but enough rabbit hole for today:)
 
SirDice said:
Setting an IP address on the bridge is not the problem. Having duplicate IP addresses on your network is the issue.
Click to expand...
Yes, you're right and I understood. My remark about the SVI was a conceptual note to self kind of thing. Thanks for your help, problem solved (I think I cannot mark as solved by myself yet).
 
You must log in or register to reply here.
Back
Top