IPFW Ipfw and internal nat

stpitn · Mar 11, 2024

Good Day
I am having a little issue with IPFW and I looking for some guidance. My network layout is

Wireless ISP -----HP-T730 with an Intel 350 Copper 4 nic networking card --- unmanaged switch
Network config in rc.conf

Currently I am able to ping 10.10.10.103 from 10.0.0.101. However the reverse is not true. Also ping from 10.10.10.103 to 1.1.1.1 does not work.

/var/log/security is not very chatty

Mar 10 14:15:54 hptserver kernel: ipfw: 900 Accept TCP 10.0.0.102:36493 10.0.0.101:22 in via igb0
Mar 10 15:01:00 hptserver kernel: ipfw: 900 Accept TCP 10.0.0.102:11872 10.0.0.101:22 in via igb0

Open to suggestions.

Thanks
stpitn

Bash:

# freebsd-version -kru ; uname -aKU
14.0-RELEASE-p5
14.0-RELEASE-p5
14.0-RELEASE-p5
FreeBSD hptserver.lan 14.0-RELEASE-p5 FreeBSD 14.0-RELEASE-p5 #0: Tue Feb 13 23:37:36 UTC 2024
root@amd64-builder.daemonology.net:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 1400097 1400097

Bash:

/etc/rc.conf
clear_tmp_enable="YES"
syslogd_flags="-ss"
hostname="hptserver.lan"
keymap="us.kbd"
sshd_enable="YES"
ntpd_enable="YES"
powerd_enable="YES"
moused_nondefault_enable="NO"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="NO"
zfs_enable="YES"

# Networking
  gateway_enable="YES"
  ifconfig_igb0="inet 10.0.0.101 netmask 255.255.255.0 -tso -lro -rxcsum -txcsum -vlanhwtso"
  defaultrouter="10.0.0.1"
  ifconfig_igb1="inet 10.10.10.2/24 10.10.10.1 -rxcsum -txcsum -txcsum -lro -tso -vlanhwtso"
  ifconfig_igb2="inet 10.10.20.2/24 10.10.10.1 -rxcsum -txcsum -txcsum -lro -tso -vlanhwtso"
  ifconfig_igb3="inet 10.10.30.2/24 10.10.10.1 -rxcsum -txcsum -txcsum -lro -tso -vlanhwtso"

# MODULES/COMMON/BASE
#  kld_list="${kld_list} ipfw ipfw_nat"
  kld_list="${kld_list} fusefs coretemp cpuctl geom_eli"

# DHCP
#  dhcpd_enable="YES"

# Firewall
  firewall_enable="YES"
  firewall_logging="YES"
  firewall_logif="YES"
  firewall_script="/usr/local/etc/ipfw.rules"

# NAT
  firewall_nat_enable="YES"

Bash:

/etc/sysctl.conf
#security.bsd.see_other_uids=0
security.bsd.see_jail_proc=0
security.bsd.unprivileged_read_msgbuf=0
security.bsd.unprivileged_proc_debug=0
kern.randompid=1
vfs.zfs.min_auto_ashift=12
security.jail.allow_raw_sockets=1
security.jail.chflags_allowed=1

# Network Address Translation
  net.inet.ip.fw.enable=1
  net.inet.ip.fw.one_pass=0
  net.inet.ip.forwarding=1
# enable logging to system logging
  net.inet.ip.fw.verbose=1
# IPFW logs to the security facility
  net.inet.ip.fw.verbose_limit=5

Code:

#!/bin/sh -
ipfw -q -f flush
# ipfw command.
fwcmd="/sbin/ipfw -q"

# set these to your outside interface network
  oif="igb0"
  onet="10.0.0.0/24"
#  onetaddr="10.0.0.101"

# set these to your inside interface network
  iif="igb1"
  inet="10.10.10.0/24"
#  wlanif="igb2"
#  wlannet="10.10.20.0/24"

BAD_ADDR_TBL=13

# Stop RFC1918 nets on the outside interface
  ${fwcmd} table ${BAD_ADDR_TBL} flush
  ${fwcmd} table ${BAD_ADDR_TBL} add 172.16.0.0/12
  ${fwcmd} table ${BAD_ADDR_TBL} add 192.168.0.0/16
  ${fwcmd} table ${BAD_ADDR_TBL} add 0.0.0.0/8
  ${fwcmd} table ${BAD_ADDR_TBL} add 44.0.0.0/8
  ${fwcmd} table ${BAD_ADDR_TBL} add 100.64.0.0/10
  ${fwcmd} table ${BAD_ADDR_TBL} add 169.254.0.0/16
  ${fwcmd} table ${BAD_ADDR_TBL} add 192.0.0.0/24
  ${fwcmd} table ${BAD_ADDR_TBL} add 192.0.2.0/24
  ${fwcmd} table ${BAD_ADDR_TBL} add 192.88.99.0/24
  ${fwcmd} table ${BAD_ADDR_TBL} add 198.18.0.0/15
  ${fwcmd} table ${BAD_ADDR_TBL} add 198.51.100.0/24
  ${fwcmd} table ${BAD_ADDR_TBL} add 224.0.0.0/4
  ${fwcmd} table ${BAD_ADDR_TBL} add 240.0.0.0/4

# Only in rare cases do you want to change these rules
  ${fwcmd} add 100 pass all from any to any via lo0
  ${fwcmd} add 200 deny all from any to 127.0.0.0/8
  ${fwcmd} add 300 deny ip from 127.0.0.0/8 to any

# Stop spoofing ipv4
  ${fwcmd} add deny all from ${inet} to any in via ${oif}
  ${fwcmd} add deny all from ${onet} to any in via ${iif}

# Define stuff we should never send out or receive in.
  ${fwcmd} add 430 deny all from any to "table($BAD_ADDR_TBL)" via ${oif}

# Drop Packets to ports where we don't want logging
  ${fwcmd} add 500 deny { tcp or udp } from any to any dst-port 23,135-139,445 in
  ${fwcmd} add 510 deny { tcp or udp } from any to any dst-port 1026,1027 in
  ${fwcmd} add 520 deny { tcp or udp } from any to any dst-port 1433,1434 in

# Broadcasts and multicasts
  ${fwcmd} add 530 deny ip from any to 255.255.255.255
  ${fwcmd} add 540 deny ip from any to 224.0.0.0/24 in

# Noise from routers
  ${fwcmd} add 550 deny udp from any to any dst-port 520 in

# Noise from webbrowsing
  ${fwcmd} add 560 deny tcp from any 80,443 to any dst-port 1024-65535 in

  ${fwcmd} add 630 check-state

# # For services permitted below
  ${fwcmd} add 650 pass tcp from any to any established

# Allow any connection out, adding state for each.
  ${fwcmd} add 700 pass tcp  from any to any setup keep-state
  ${fwcmd} add 710 pass udp  from any to any       keep-state
  ${fwcmd} add 720 pass icmp from any to any       keep-state

# Allow DHCP
  ${fwcmd} add 740 pass udp  from 0.0.0.0 68 to 255.255.255.255 67 out
  ${fwcmd} add 750 pass udp  from any 67     to me 68 in
  ${fwcmd} add 760 pass udp  from any 67     to 255.255.255.255 68 in

# Allow  ICMP
  ${fwcmd} add 770 pass icmp from any to any icmptype 8
  ${fwcmd} add 780 pass ipv6-icmp from any to any icmp6type 128,129
  ${fwcmd} add 790 pass icmp from any to any icmptype 3,4,11

# Allow access to our DNS
  ${fwcmd} add 810 pass tcp from any to me 53 keep-state
  ${fwcmd} add 820 pass udp from any to me 53
  ${fwcmd} add 830 pass udp from me 53 to any

# Allow NTP queries out in the world
  ${fwcmd} add 850 pass udp from me to any 123 keep-state

# MyServices Accept and log ssh on 22
  ${fwcmd} add 900 allow log logamount 100 tcp from any to me dst-port 22 setup
  ${fwcmd} add 910 allow tcp from any to me dst-port 22
  ${fwcmd} add 940 allow tcp from any to me dst-port 25 setup
  ${fwcmd} add 920 allow tcp from any to me dst-port 80 setup
  ${fwcmd} add 970 allow tcp from any to me dst-port 110 setup
  ${fwcmd} add 930 allow tcp from any to me dst-port 443 setup
  #${fwcmd} add 950 allow tcp from any to me dst-port 465
  ${fwcmd} add 960 allow tcp from any to me dst-port 587 setup
  ${fwcmd} add 980 allow tcp from any to me dst-port 853
  #${fwcmd} add 990 allow tcp from any to me dst-port 143
  #${fwcmd} add 1000 allow tcp from any to me dst-port 995

  ${fwcmd} add 1500 count ip from any to any

# NAT
  ${fwcmd} add 2100 reass all from any to any in
  ipfw nat 1 config if ${oif} log reset same_ports unreg_only
  ${fwcmd} add 2130 nat 1 ip from ${inet} to any in via ${oif}

# Outbound
  ${fwcmd} add 2200 nat 101 all from any to any
  ipfw nat 99 config log ip 10.0.0.101 log deny_in reset same_ports
  ${fwcmd} add 2210 pass ip from any to any out xmit ${oif}

# Allow established connections
  ${fwcmd} add 2220 pass tcp from any to any established

# Log and drop all other ports
  ${fwcmd} add 65534 deny log logamount 500 ip from any to any

# Default rule DROP, this is set by default so it's not needed
  #${fwcmd} 65535 deny ip from any to any

Code:

# sockstat -4 -l
USER     COMMAND    PID   FD  PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
root     sshd       19885 4   tcp4   *:22                  *:*
ntpd     ntpd       15034 21  udp4   *:123                 *:*
ntpd     ntpd       15034 22  udp4   10.0.0.101:123        *:*
ntpd     ntpd       15034 23  udp4   10.10.10.2:123        *:*
ntpd     ntpd       15034 24  udp4   10.10.20.2:123        *:*
ntpd     ntpd       15034 25  udp4   10.10.30.2:123        *:*
ntpd     ntpd       15034 28  udp4   127.0.0.1:123         *:*

# ping 1.1.1.1 from 10.0.0.101
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=0 ttl=56 time=36.008 ms
64 bytes from 1.1.1.1: icmp_seq=1 ttl=56 time=62.584 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=56 time=40.972 ms
^C
--- 1.1.1.1 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 28.588/39.353/62.584/12.528 ms

# ping 10.10.10.103 from 10.0.0.101
PING 10.10.10.103 (10.10.10.103): 56 data bytes
64 bytes from 10.10.10.103: icmp_seq=0 ttl=64 time=0.589 ms
64 bytes from 10.10.10.103: icmp_seq=1 ttl=64 time=0.398 ms
64 bytes from 10.10.10.103: icmp_seq=2 ttl=64 time=0.328 ms
^C
--- 10.10.10.103 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.328/0.438/0.589/0.110 ms

SirDice · Mar 12, 2024

stpitn said:

Code:

  ifconfig_igb1="inet 10.10.10.2/24 10.10.10.1 -rxcsum -txcsum -txcsum -lro -tso -vlanhwtso"
  ifconfig_igb2="inet 10.10.20.2/24 10.10.10.1 -rxcsum -txcsum -txcsum -lro -tso -vlanhwtso"
  ifconfig_igb3="inet 10.10.30.2/24 10.10.10.1 -rxcsum -txcsum -txcsum -lro -tso -vlanhwtso"

What are you trying to set with that 10.10.10.1 address?

stpitn · Mar 12, 2024

SirDice
Due to my very rudimentary knowledge of networking, 10.0.0.1 is the default gateway for the WAN and the 10.10.10.1 would be a gateway for the LAN. Is there another way to achieve this? Or is my thinking completely wrong.
Thanks

SirDice · Mar 12, 2024

That's not how you set a gateway. I suggest you remove it, I'm guessing it gets picked up as a netmask, which ruins the whole configuration (and the reason for the connection weirdness).

stpitn said:
10.0.0.1 is the default gateway for the WAN and the 10.10.10.1 would be a gateway for the LAN.

There can only be one default gateway (assuming a single routing table) and you set it with defaultrouter="10.0.0.1". The ifconfig_* sets the configuration for the interface, not routing. Static routes (you probably don't need them on this host) could be set like so:

Code:

static_routes="myroute1 myroute2"
route_myroute1="-net 172.16.4.0/24 10.10.20.1"
route_myroute2="-net 192.168.0.0/24 10.0.0.1"

stpitn said:
Or is my thinking completely wrong.

Yes,10.10.10.1 is an address that's directly connected to igb1. Your routing table is going to reflect that.

smithi · Mar 12, 2024

stpitn said:

Code:

# set these to your outside interface network
oif="igb0"
onet="10.0.0.0/24"
# onetaddr="10.0.0.101"
# set these to your inside interface network iif="igb1" inet="10.10.10.0/24

onet should include your address on the network, i.e.
onet="10.0.0.101/24"
Same for inet.

It's good that you provided so much info, but this attempt at merging the rc.firewall "simple" and "workstation" rulesets isn't going to fly well, as it stands.

"simple" is for a router between an outside net, such as internet via another gateway and a) this machine and an inside network, probably via NAT; b) services offered to outside and/or inside networks and c) possible use as a workstation as well, with care.

"workstation" specifically does not provide NAT to other hosts, and uses a mix of stateful rules for client-based services (e.g. browsing) and stateless rules for services provided to mostly the local net, or perhaps port-redirected services from upstream NAT.

This is well achievable but not the way this ruleset attempts it. NAT is done in the wrong place, check-state and allowing stateless established rules are in the wrong place. You need to clearly separate all routed traffic from local traffic, see ipfw(8) sections PACKET FLOW and STATEFUL FIREWALL for guidance.

If I were you I'd post the report to freebsd-ipfw@freebsd.org where ipfw folks will see it; in my experience they're not here.

And this much has exhausted me, sorry I can't dive in deep ...

cheers, Ian

IPFW Ipfw and internal nat

stpitn

SirDice

Administrator

stpitn

SirDice

Administrator

smithi