Hi everybody,
I've been using FreeBSD for quite some time, and I'm currently stuck with a problem that is most probably due to a dumb configuration mistake on my side. But I've been struggling with it for two days now and I just can't find out what I'm doing wrong. This is why I decided to ask.
My setup is:
- A FreeBSD server with a public IP address - let's say 1.2.3.4 (I've redacted the IP in all config files due to privacy reasons) - on interface em0
- A network bridge "bridge0" with network 192.168.100.0/24 assigned to it
- A bhyve VM using IP address 192.168.100.101 (assigned via DHCP from a dhcpd running on the host). The VM is running an nginx webserver on port 80.
I can:
- ping 192.168.100.101 from the FreeBSD host
- "telnet 192.168.100.101 80" from the FreeBSD host
I'm trying to forward incoming traffic on the external interface em0, port 80, to the nginx server running in the bhyve VM and I can't get it to work.
Here's my ifconfig:
Here's my /etc/rc.conf:
Here's my /etc/sysctl.conf:
and here's my /etc/pf.conf, assembled from multiple documentations I found, and apart from the tcp port forwarding into the bhyve VM, it seems to do what I want it to do:
When I try to access 1.2.3.4:80 from the outside world, I see packets arriving on em0, and can also see the "rdr" pf rule being triggered. But nothing else, the connection is not forwarded to bridge0:
I'm sorry for being so verbose, but I want to provide everything that could be necessary in tracking down my mistake. I'm really lost here and would be very glad if somebody with more experience in configuring pf could take his time to help me out. Thank you in advance!
I've been using FreeBSD for quite some time, and I'm currently stuck with a problem that is most probably due to a dumb configuration mistake on my side. But I've been struggling with it for two days now and I just can't find out what I'm doing wrong. This is why I decided to ask.
My setup is:
- A FreeBSD server with a public IP address - let's say 1.2.3.4 (I've redacted the IP in all config files due to privacy reasons) - on interface em0
- A network bridge "bridge0" with network 192.168.100.0/24 assigned to it
- A bhyve VM using IP address 192.168.100.101 (assigned via DHCP from a dhcpd running on the host). The VM is running an nginx webserver on port 80.
I can:
- ping 192.168.100.101 from the FreeBSD host
- "telnet 192.168.100.101 80" from the FreeBSD host
I'm trying to forward incoming traffic on the external interface em0, port 80, to the nginx server running in the bhyve VM and I can't get it to work.
Here's my ifconfig:
Code:
root@mentat:~ # ifconfig
em0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=4e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 30:9c:23:b8:c0:8a
inet 1.2.3.4 netmask 0xffffffc0 broadcast 1.2.3.255
inet6 [redacted] prefixlen 64
inet6 fe80::329c:23ff:feb8:c08a%em0 prefixlen 64 scopeid 0x1
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=0
ether 58:9c:fc:10:ff:ac
inet 192.168.100.1 netmask 0xffffff00 broadcast 192.168.100.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 2000000
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 2000000
groups: bridge vm-switch viid-4c918@
nd6 options=9<PERFORMNUD,IFDISABLED>
tap0: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
ether 58:9c:fc:10:8e:79
groups: tap
media: Ethernet 1000baseT <full-duplex>
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pflog0: flags=1000141<UP,RUNNING,PROMISC,LOWER_UP> metric 0 mtu 33152
options=0
groups: pflog
tap1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: vmnet/debian/0/public
options=80000<LINKSTATE>
ether 58:9c:fc:10:ff:a9
groups: tap vm-port
media: Ethernet 1000baseT <full-duplex>
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Opened by PID 1868Here's my /etc/rc.conf:
Code:
hostname="[redacted]"
zfs_enable="YES"
pf_enable="YES"
pflog_enable="YES"
sshd_enable="YES"
tinyproxy_enable="YES"
gateway_enable="YES"
vm_enable="YES"
vm_dir="zfs:zfs/vm"
vm_list="debian"
vm_delay="5"
ifconfig_em0="DHCP"
ifconfig_em0_ipv6="inet6 [redacted]"
cloned_interfaces="bridge0 tap0"
vmm_load="YES"
kld_list="nmdm vmm"
ifconfig_bridge0="inet 192.168.100.1 netmask 255.255.255.0 broadcast 192.168.100.255 addm tap0"
dhcpd_enable="YES"Here's my /etc/sysctl.conf:
Code:
net.link.tap.up_on_open=1
net.inet.ip.forwarding=1and here's my /etc/pf.conf, assembled from multiple documentations I found, and apart from the tcp port forwarding into the bhyve VM, it seems to do what I want it to do:
Code:
## Set your public interface ##
ext_if="em0"
## Set your server public IP address ##
ext_if_ip="1.2.3.4"
## Set and drop these IP ranges on public interface ##
martians = "{ 127.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, \
10.0.0.0/8, 169.254.0.0/16, 192.0.2.0/24, \
0.0.0.0/8, 240.0.0.0/4 }"
# network for bhyve VMs
vm_if="bridge0"
vmnet="192.168.100.0/24"
debian_vm="192.168.100.101"
## Set http(80)/https (443) port here ##
webports = "{http, https}"
## enable these services ##
int_tcp_services = "{domain, ntp, smtp, www, https, ftp, ssh, 23}"
int_udp_services = "{domain, ntp}"
## Sets the interface for which PF should gather statistics such as bytes in/out and packets passed/blocked ##
set loginterface $ext_if
# Deal with attacks based on incorrect handling of packet fragments
scrub in all
## Skip loop back interface - Skip all PF processing on interface ##
set skip on lo
# bhyve nat
nat on $ext_if from $vmnet to any -> ($ext_if)
#### ------------------ route http/https traffic into vm debian ---------------------------
rdr pass log on $ext_if inet proto tcp from any to any port 80 -> $debian_vm port 80
pass in on $vm_if inet proto tcp from any to $debian_vm port 80 keep state
#### what is wrong in the above two lines - or in the overall configuration? -----------
## Set default policy ##
block return in log all
block out all
# Drop all Non-Routable Addresses
block drop in quick on $ext_if from $martians to any
block drop out quick on $ext_if from any to $martians
## Blocking spoofed packets
antispoof quick for $ext_if
## Use the following rule to enable ssh for ALL users from any IP address #
pass in inet proto tcp to $ext_if port ssh
# Allow Ping-Pong stuff. Be a good sysadmin
pass inet proto icmp icmp-type echoreq
# All access to our Nginx/Apache/Lighttpd Webserver ports
pass proto tcp from any to $ext_if port $webports
pass proto tcp from $vmnet to $vmnet
# Allow access for VM
pass proto tcp from $vmnet to port 8888
pass proto tcp from $vmnet to port 22
pass proto udp from $vmnet to port {67, 68}
# Allow essential outgoing traffic
pass out quick on $ext_if proto tcp to any port $int_tcp_services
pass out quick on $ext_if proto udp to any port $int_udp_servicesWhen I try to access 1.2.3.4:80 from the outside world, I see packets arriving on em0, and can also see the "rdr" pf rule being triggered. But nothing else, the connection is not forwarded to bridge0:
Code:
root@freebsd:/etc # tcpdump -n -i em0 port 80
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on em0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
17:36:45.921406 IP 156.67.139.105.44916 > 1.2.3.4.80: Flags [S], seq 1394833432, win 64240, options [mss 1452,sackOK,TS val 1059157925 ecr 0,nop,wscale 7], length 0
## new connection attempt, therefore different outgoing port
root@mentat:~ # tcpdump -n -e -ttt -i pflog0 port 80
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on pflog0, link-type PFLOG (OpenBSD pflog file), snapshot length 262144 bytes
00:00:00.000000 rule 0/0(match): rdr in on em0: 156.67.139.105.11297 > 1.2.3.4.80: Flags [S], seq 2447330776, win 64240, options [mss 1452,sackOK,TS val 1059240751 ecr 0,nop,wscale 7], length 0
## but nothing is happening on bridge0
root@mentat:/etc # tcpdump -n -i bridge0 port 80
## whereas when executing "telnet 192.168.100.101 80" on the freeebsd host, I can see traffic on bridge0 and the connection is established:
root@mentat:/etc # tcpdump -n -i bridge0 port 80
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on bridge0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
17:40:19.092079 IP 192.168.100.1.15556 > 192.168.100.101.80: Flags [S], seq 2352298141, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2299072963 ecr 0], length 0
17:40:19.092448 IP 192.168.100.101.80 > 192.168.100.1.15556: Flags [S.], seq 1124584372, ack 2352298142, win 64308, options [mss 1410,sackOK,TS val 3482475983 ecr 2299072963,nop,wscale 7], length 0
17:40:19.092492 IP 192.168.100.1.15556 > 192.168.100.101.80: Flags [.], ack 1, win 1036, options [nop,nop,TS val 2299072964 ecr 3482475983], length 0
...I'm sorry for being so verbose, but I want to provide everything that could be necessary in tracking down my mistake. I'm really lost here and would be very glad if somebody with more experience in configuring pf could take his time to help me out. Thank you in advance!
Last edited by a moderator:
