Solved VM guest can access internet except can't 'git clone'

wison · Mar 17, 2024

Hi guys, question about bhyve VM networking, I install ArchLinux vm guest, can access internet from vm (ping google.co.nz works, pacman install works, curl file works) except can't git clone (Failed to connect to github.com port 443 after 133428 ms: Couldn't connect to server). Need a guide to solve this

- Host OS version:

Bash:

FreeBSD th80 14.0-RELEASE-p5 FreeBSD 14.0-RELEASE-p5 #0: Tue Feb 13 23:37:36 UTC 2024     root@amd64-builder.daemonology.net:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64

- Host pf disabled

Bash:

doas service pf status
Status: Disabled for 0 days 00:41:16          Debug: Urgent

- Host VM switch and routing table (internet via igc0 ethernet, not WIFI)

Bash:

doas vm switch list
# NAME      TYPE      IFACE        ADDRESS         PRIVATE  MTU  VLAN  PORTS
# internal  standard  vm-internal  192.168.2.1/24  no       -    -     igc0


netstat  -rn4
# Routing tables
#
# Internet:
# Destination        Gateway            Flags     Netif Expire
# default            192.168.1.200      UGS        igc0
# 127.0.0.1          link#2             UH          lo0
# 192.168.1.0/24     link#3             U         wlan0
# 192.168.1.160      link#2             UHS         lo0
# 192.168.1.190      link#2             UHS         lo0
# 192.168.2.0/24     link#5             U      vm-inter
# 192.168.2.1        link#2             UHS         lo0

- Host ifconfig when vm guest is running

Bash:

ifconfig
igc0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=4a420b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWTSO,RXCSUM_IPV6,HWSTATS,MEXTPG>
        ether 58:47:ca:70:14:b5
        inet 192.168.1.160 netmask 0xffffff00 broadcast 192.168.1.255
        groups: pfnic
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
wlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=0
        ether a0:80:69:f4:cd:1a
        inet 192.168.1.190 netmask 0xffffff00 broadcast 192.168.1.255
        groups: wlan pfnic
        ssid JI-LE-SHI-JIE channel 40 (5200 MHz 11a) bssid 34:58:40:ca:f8:bc
        regdomain APAC2 country NZ authmode WPA2/802.11i privacy ON
        deftxkey UNDEF AES-CCM 2:128-bit AES-CCM 3:128-bit txpower 17 bmiss 7
        mcastrate 6 mgmtrate 6 scanvalid 60 wme roaming MANUAL
        parent interface: iwlwifi0
        media: IEEE 802.11 Wireless Ethernet OFDM/54Mbps mode 11a
        status: associated
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pflog0: flags=0 metric 0 mtu 33152
        options=0
        groups: pflog
vm-internal: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=0
        ether 8e:4a:c9:b2:73:03
        inet 192.168.2.1 netmask 0xffffff00 broadcast 192.168.2.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000000
        member: igc0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge vm-switch viid-d1efa@
        nd6 options=9<PERFORMNUD,IFDISABLED>
tap0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: vmnet/my-arch/0/internal
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:ff:9e
        groups: tap vm-port
        media: Ethernet 1000baseT <full-duplex>
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 31357

- VM ip and route table

Bash:

ip add
# 2: enp0s5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
#     link/ether 58:9c:fc:0b:40:26 brd ff:ff:ff:ff:ff:ff
#     inet 192.168.2.10/24 brd 192.168.2.255 scope global enp0s5
#        valid_lft forever preferred_lft forever
#     inet6 2407:7000:98b6:b200:5a9c:fcff:fe0b:4026/64 scope global dynamic mngtmpaddr noprefixroute
#        valid_lft 1295905sec preferred_lft 647905sec
#     inet6 fe80::5a9c:fcff:fe0b:4026/64 scope link proto kernel_ll
#        valid_lft forever preferred_lft forever


ip route show
# default via 192.168.2.1 dev enp0s5 proto static
# 192.168.2.0/24 dev enp0s5 proto kernel scope link src 192.168.2.10

Welcome any idea about how to debug and solve, many thanks

richardtoohey2 · Mar 17, 2024

Does using curl to https://github.com work or give you the same message? Just to eliminate something special about git.

And you can definitely connect to https sites (using curl)?

SirDice · Mar 17, 2024

Might need to install security/ca_root_nss for the root CAs.

wison · Mar 17, 2024

richardtoohey2 said:
Does using curl to https://github.com work or give you the same message? Just to eliminate something special about git.

And you can definitely connect to https sites (using curl)?

Curl on HOST no problem:

Bash:

curl https://jsonplaceholder.typicode.com/users | wc -l

# % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
#                                  Dload  Upload   Total   Spent    Left  Speed
# 100  5645    0  5645    0     0  50829      0 --:--:-- --:--:-- --:--:-- 51318
#      231

curl https://github.com | wc -l

#  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
#                                  Dload  Upload   Total   Spent    Left  Speed
# 100  217k    0  217k    0     0   708k      0 --:--:-- --:--:-- --:--:--  710k
#     2379

Curl on VM no problem except 'github.com':

Bash:

curl https://jsonplaceholder.typicode.com/users | wc -l

# % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
#                                  Dload  Upload   Total   Spent    Left  Speed
# 100  5645    0  5645    0     0   7039      0 --:--:-- --:--:-- --:--:--  7038
# 231

c   curl https://github.com | wc -l
#   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
#                                  Dload  Upload   Total   Spent    Left  Speed
#   0     0    0     0    0     0      0      0 --:--:--  0:00:07 --:--:--     0^C⏎

#
# Waiting.....
#

# curl: (28) Failed to connect to github.com port 443 after 133314 ms: Couldn't connect to server
# 0

wison · Mar 17, 2024

SirDice said:
Might need to install security/ca_root_nss for the root CAs.

Just installed:

Bash:

 pkg info --all | rg ca_root
ca_root_nss-3.93_2             Root certificate bundle from the Mozilla Project
 N [ FreeBSD ] wison | /home/wison   pkg info --all | rg ca_root^C
 I [ FreeBSD ] wison | /home/wison   pkg info --list-files ca_root_nss
ca_root_nss-3.93_2:
        /etc/ssl/cert.pem
        /usr/local/etc/ssl/cert.pem.sample
        /usr/local/openssl/cert.pem.sample
        /usr/local/share/certs/ca-root-nss.crt
        /usr/local/share/licenses/ca_root_nss-3.93_2/LICENSE
        /usr/local/share/licenses/ca_root_nss-3.93_2/MPL20
        /usr/local/share/licenses/ca_root_nss-3.93_2/catalog.mk

Reboot HOST and try again, still doesn't work, any extra config step needed? Actually, I was wondering why install ca_root_nss should fix?

One more information, my Arch Linux guest doesn't install =X= (I don't need GUI at all), I just ssh into it to use (for coding test purpose), I mentioned this because I saw ca_root_nss related to Firefox and Thunderbird....[/icode]

richardtoohey2 · Mar 18, 2024

And can curl connect to github from the host?

wison · Mar 18, 2024

richardtoohey2 said:
And can curl connect to github from the host?

Yes, absolutely, see curl result above

richardtoohey2 · Mar 18, 2024

Oops, sorry, missed that part!

Check DNS (it can't be DNS!) - but make sure resolving on guest and host to something reasonable.

Try verbose on the git clone command (the internet says it doesn't do much useful) - but verbose output on curl might show something?

fmotsch · Mar 19, 2024

Did you try to ssh into the VM and try 'git clone' then ?

richardtoohey2 · Mar 19, 2024

That’s the problem the OP is having. Git clone doesn’t work inside the VM but everything else does, and git clone works from the host.

Hiroo Ono · Mar 19, 2024

Did you check the pf or ipfw status in the VM?
Also check if curl to https://some_other_host_than_github works.

richardtoohey2 · Mar 19, 2024

OP checked firewall in host, but don't think guest/VM.

OP has tried curl for https sites and they work from guest/VM or host.

github.com doesn't like connections from the guest/VM.

I suggested checking DNS to make sure github.com in VM definitely the right github.com.

So something in the networking from the guest/VM is making traffic to/from github.com get lost.

Hiroo Ono · Mar 19, 2024

Sorry I also missed that the guest is ArchLinux. So checking iptables setting in the guest is what I should have advised.

rootbert · Mar 19, 2024

also please try e1000 in your vm config instead of virtio-net

taiwan740 · Mar 19, 2024

Try ping 140.82.121.3

wison · Mar 20, 2024

fmotsch said:
Did you try to ssh into the VM and try 'git clone' then ?

Of course

wison · Mar 20, 2024

richardtoohey2 said:
That’s the problem the OP is having. Git clone doesn’t work inside the VM but everything else does, and git clone works from the host.

Actually, that's not right, and I figured out how it works right now, git clone works well in VM (and it should be), I will make a summary when I done with my documentation later

wison · Mar 20, 2024

Hiroo Ono said:
Sorry I also missed that the guest is ArchLinux. So checking iptables setting in the guest is what I should have advised.

Arch Linux VM doesn't apply any firewall, as I don't want to setup each different firewall for every VM (I got Arch Linux, Alpine, OpenBSD, etc...), I plan to use host's PF as the central firewall and network traffic control, I think I can make it

wison · Mar 20, 2024

Ok, guys, thanks for all the friendly replies above, I believe that I was using the wrong way to implement my virtual network topology, and I've already found a temporary (regular) solution: Just back to normal bridge solution (all VMs connect to the same LAN), then everything works fine

Bash:

Create public bridge virtual switch

If you want all VMs attached to your current (FreeBSD Bhyve Host) network just like another computer on your LAN, then you should create a VM switch in this way.

For example, your local LAN network is '192.168.1.0/24', the network topology would like this:

----------------------------------------------------
                192.168.1.0/24
----------------------------------------------------
        |             |        |        |
        |             |        |        |
        |             |        |        |
  /------------\   /-----\  /-----\  /-------\
  |  FreeBSD   |   | VM1 |  | VM2 |  | VM... |
  | Bhyve host |   |     |  |     |  |       |
  \------------/   \-----/  \-----/  \-------/


Pros:
    - All VMs can access the internet via your host NIC without extra configuration.

Crons:
    - Your 'PF' can't protect all VMs, they have to setup their firewalls.

Here is the virtual switch:

Bash:

NAME    TYPE      IFACE      ADDRESS  PRIVATE  MTU  VLAN  PORTS
public  standard  vm-public  -        no       -    -     igc0

All VMs use the same configuration like the host: DHCP. As the virtual switch bind to physical igc0 NIC, then ech VM assigns an 192.168.1.x/24 IP and use the same DNS with host (192.168.1.1 which my default home router), but this is just a temporary solution, as that's NOT the init topology I wanted.

Actually, I want this:

Bash:

Create internal virtual switch

----------------------------------------------------
                192.168.1.0/24
----------------------------------------------------
                         |          
                   /------------\      
                   |  FreeBSD   |      
                   | Bhyve host |
                   \------------/      
                         |          
                    /----------\    NAT
                    | Internal | ------------------
                    |  Switch  |   192.168.2.0/24 
                    \----------/ ------------------
                         |
                /-----------------\
                |        |        |
                |        |        |
                |        |        |
             /-----\  /-----\  /-------\
             | VM1 |  | VM2 |  | VM... |
             |     |  |     |  |       |
             \-----/  \-----/  \-------/

All VMS hide in the internally isolated network and use NAT to route the traffic between VM and outside world, then I can use the host's PF to control all detail traffics I want to pass or deny.

Because the virtual switch has the groups: bridge vm-switch viid-4c918@ groups attribute bound to the actual bridge NIC, So I believe I can setup detailed PF rules targets to that particular NIC to reach what I want.

The previous failure should be related I missed the dnsmasq on the host, and then internal VM DNS didn't work well, but I will give it a try later when I have time, and I think I can make it

Hiroo Ono · Mar 20, 2024

Edit: sorry, if curl from the VM to hosts other than github, you should have set it. Though I still think it is the gateway or NAT problem...

In the latter case (VMs in the isolated network), the FreeBSD host must be a gateway.
Did you have

Code:

gateway_enable="YES"

in the /etc/rc.conf? And if you use IPv6, also

Code:

ipv6_gateway_enable="YES"

Hiroo Ono · Mar 20, 2024

Maybe an MTU problem?

richardtoohey2 · Mar 20, 2024

wison said:
DNS didn't work

So it was DNS? Wonder if there's a meme about that

wison · Mar 21, 2024

Ok, finally, I made it happen, and everything works fine, here is the entire documentation, just in case someone wanted it:

*** Create internal virtual switch

If you want all VMs attached to your current (FreeBSD Bhyve Host) network just like another computer on your LAN, then you should create a VM switch in this way.

For example, your local LAN network is 192.168.1.0/24, the network topology would like this:

Bash:

----------------------------------------------------
                192.168.1.0/24
----------------------------------------------------
                         |          
                   /------------\      
                   |  FreeBSD   |      
                   | Bhyve host |
                   \------------/      
                         |          
                    /----------\    NAT
                    | Internal | ------------------
                    |  Switch  |   192.168.2.0/24 
                    \----------/ ------------------
                         |
                /-----------------\
                |        |        |
                |        |        |
                |        |        |
             /-----\  /-----\  /-------\
             | VM1 |  | VM2 |  | VM... |
             |     |  |     |  |       |
             \-----/  \-----/  \-------/


Pros:
    - Protect all VMs behind the isolated network
    - Detailed control network traffic between VMs and the outside world

Crons:
    - Your [icode]PF[/icode] rules become more complicated

**** Create a standard type virtual switch

- -t: Specify virtual switch type, standard is the default value if you don't provided.
- -a: Specify an IP adress that is assigned to the bridge interface.
- -p: Enable private mode, guests can't access each other, even they're conntected the same virtual switch!!!
- internal: The name of virtual switch, maximum length of a switch name is also limited to 12 characters.

Bash:

  doas vm switch create \
      -t standard \
      -a 192.168.2.1/24 \
      internal

Plz DO NOT attach a physical interface to the virtual switch, so all network packets go through the virtual switch NIC, then you can use PF rules to control everything

**** List and info

Bash:

  doas vm switch list
  # NAME      TYPE      IFACE        ADDRESS         PRIVATE  MTU  VLAN  PORTS
  # internal  standard  vm-internal  192.168.2.1/24  no       -    -     -

  doas vm switch info internal
  # ------------------------
  # Virtual Switch: internal
  # ------------------------
  #   type: standard
  #   ident: vm-internal
  #   vlan: -
  #   physical-ports: -
  #   bytes-in: 0 (0.000B)
  #   bytes-out: 0 (0.000B)

**** What actual change on the host

When you created a virtual switch, you can see ifconfig list a new bridge NIC like this:

The new NIC's name will be vm-YOUR_VIRTUAL_SWITCH_NAME

Bash:

  vm-internal: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
          options=0
          ether 2e:48:ac:4d:0e:a3
          inet 192.168.2.1 netmask 0xffffff00 broadcast 192.168.2.255
          id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
          maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
          root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
          groups: bridge vm-switch viid-d1efa@
          nd6 options=9<PERFORMNUD,IFDISABLED>

Also, new rules are added to routing table:

Bash:

  netstat  -rn4

  # Routing tables
  # 
  # Internet:
  # Destination        Gateway            Flags     Netif Expire
  # default            192.168.1.200      UGS        igc0
  # 127.0.0.1          link#2             UH          lo0
  # 192.168.1.0/24     link#1             U          igc0
  # 192.168.1.160      link#2             UHS         lo0
  192.168.2.0/24     link#5             U      vm-inter
  192.168.2.1        link#2             UHS         lo0

**** Enable IP forwarding

You have to enable IP forwarding on the host to make NAT works!!!

Add the following settings to /etc/rc.conf:

Bash:

  gateway_enable="yes"

Reboot and then print the system setting to confirm:

Bash:

  sysctl net.inet.ip | rg forward

  # net.inet.ip.forwarding: 1

That menas IP forwarding works.

**** PF settings

Suppose that your PF applies a block all policy, which means you only care about the pass rules.

In the following examples, they use these PF macros, feel free to change them to match yours:

Bash:

  # ==============================================================================
  #
  # Macros
  #
  # - `nic_eth`: Default ethernet NIC
  # - `nic_wlan`: Default wireless NIC
  # - `nic_group `: The specified group name that applied to your NICs. You need
  #                 to add the `group pfnic` parameter to `ifconfig_xxx` inside
  #                 `/etc/rc.conf`.
  # - `$nic_group:network`: Get back the NIC assigned network, e.g. 192.168.1.0/24
  # - `($nic_eth)` or `($nic_wlan)`: Get back the exact NIC assigned IP
  #
  # ==============================================================================
  nic_eth = "igc0"
  nic_wlan = "wlan0"
  nic_group = "pfnic"
  nic_vm_switch = "vm-internal"
  nic_vm_group = "vm-switch"
  trust_vm_node = "{192.168.2.10, 192.168.2.20}"

Plz keep that in mind, all network packets between the host and VMs are through by the internal virtual switch (192.168.2.1).

Bash:

  vm-internal: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
         ether ea:36:15:e6:c3:02
         inet 192.168.2.1 netmask 0xffffff00 broadcast 192.168.2.255
         groups: bridge vm-switch viid-d1efa@

That's why the following macros give you back the following values:

- nic_vm_switch: Get back the vm-internal NIC.
- nic_vm_group: Get back the vm-internal NIC by the group name
- nic_vm_switch:network: Get back the virtual switch assigned network: 192.168.2.0/24

***** NAT (Network Address Translation)

Bash:

  # ==============================================================================
  # NAT for VM
  #
  # Since translation occurs before filtering the filter engine will see packets
  # as they look after any addresses and ports have been translated.
  # ==============================================================================
  nat on $nic_eth from $nic_vm_switch:network to any -> ($nic_eth)

***** VM pass rules

Usually, your PF settings should already have the pass rules for the external NIC (nic_eth in this example) that connects to your router to reach the internet. So, the following pass rules only on the virtual network part!!!

Also, keep that in mind, a network packet go through the following path (or steps) before going out to your default gateway via nic_eth (in this example):

VM NIC (192.168.2.10) --> Virtual Switch NIC (192.168.2.1) --> Gateway NIC (nic_eth) --> NAT (tranlsate 192.168.2.0/24 to your nic_eth network) --> Router

As you've already had your pass rules on nic_eth, that's why you don't need to care about that part anymore, you only focus on VM NIC --> Virtual Switch NIC (before NAT) rules!!!

Bash:

  # ==============================================================================
  #
  # All VM related
  #
  # ==============================================================================

  #
  # Ping outside
  #
  pass in quick on $nic_vm_group proto icmp from $nic_vm_switch:network to any icmp-type echoreq

  #
  # This make VM DNS and NTP works!!!
  #
  pass in quick on $nic_vm_group proto udp from $nic_vm_switch:network to any port {domain,ntp,mdns,ssdp,bootpc}

  #
  # Allow dialy app to use: all outgoing TCP and related reply
  #
  pass in quick on $nic_vm_group proto tcp from $nic_vm_switch:network to any

  #
  # Youtube needed
  # https       443/udp
  # ssdp        1900/udp   #Selective Service Discovery Protocol (UPnP)
  #
  pass in quick on $nic_vm_group proto udp from $nic_vm_switch:network to any port {https,ssdp}

  #
  # Allow host to ssh to vm
  #
  # Packets path: Virtual switch NIC --> out ( from virtual switch IP trust VM IP )
  #
  pass out quick on $nic_vm_group proto tcp from ($nic_vm_switch) to $trust_vm_node port ssh

Solved VM guest can access internet except can't 'git clone'

Administrator