Confusing about default gateway with multiple FIB.

epopen · Feb 9, 2026

Hi All

I am using VIMAGE(vnet) + netgraph(4) to build independence two jail(8) as follows.

Here, NIC ng_vimage0 using jail (IP range: 10.0.2.0~10.0.3.254) to access with host and internet, FIB=1

Code:

ng_vimage0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
        options=28<VLAN_MTU,JUMBO_MTU>
        ether 02:ac:95:e3:83:d2
        hwaddr 58:9c:fc:10:8a:17
        inet 10.0.3.254 netmask 0xfffffe00 broadcast 10.0.3.255
        fib: 1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=61<PERFORMNUD,AUTO_LINKLOCAL,NO_RADR>

Host side, NIC ng0 using host access internet by PPPoE (Dynamic IPv4) using net/mpd5

Code:

ng0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
        options=0
        inet 220.132.60.27 --> 168.95.98.254 netmask 0xffffffff
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

Next: host route table @ FIB=0

Code:

Internet:
Destination        Gateway            Flags         Netif Expire
default            168.95.98.254      UGS             ng0
10.0.2.0/23        link#6             US       ng_vimage0
127.0.0.1          link#2             UH              lo0
168.95.98.254      link#5             UH              ng0
220.132.60.27      link#2             UHS             lo0

And For jail route table @ FIB=1

Code:

Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags         Netif Expire
default            link#5             US              ng0
10.0.2.0/23        link#6             U        ng_vimage0
10.0.3.254         link#2             UHS             lo0
127.0.0.1          link#2             UHS             lo0

(link#2 = lo0, link#5 = ng0 and link#6 = ng_vimage0)

The solution workable, but I confuing...
In jail route table @ FIB=1, default gateway = ng0 is correct?
(Because correct method I imagine it is that the default gateway is ng_vimage0,
and whether accessing the internet or the host from jail, it is routed through the system only (routing to ng0 when accessing the internet)
But I know I'm missing some key, so my imagination untenable).

If my current solution and imagine is incorrectly, would you can help to correct?

Thanks all very much.

AlfredoLlaquet · Feb 20, 2026

WARNING 1: Seeing that you have received no additional responses in more than 72 hours, with nothing but good intentions in my heart, especially helping you, I outsourced your case to a non-subject expert-in-training and below the horizontal line is what it had to say.

WARNING 2: Use it all only as an inspiration or starting investigative point and check everything using official sources. That it sounds certain doesn't mean it is; these non-subjects are trained to first and foremost sound certain.

WARNING 3: Read in full before doing anything. Then, decide what you want to do.

Note: If you end up solving your problem, please write a post explaining what fixed it, so it gets documented for the community.

How the fib setting on ng_vimage0 works

ifconfig(8)(): ifconfig ... fib N assigns a FIB to frames/packets received on that interface. Route lookup for those packets uses that FIB. The setting is not inherited by VLANs or other sub-interfaces.

Given ng_vimage0 has fib: 1, traffic arriving from the jail-side segment into the host via ng_vimage0 is routed using host FIB 1.

HAZARD: Connectivity loss
Any routing-table change (especially default routes) can immediately cut off remote access to the host. Apply changes from local console access and keep rollback commands ready.

Why “default via ng0” in FIB 1 is correct

The routing table shown as “fib: 1” contains host interfaces ( ng0, ng_vimage0), so it is a host routing table, not a VNET jail’s routing table.

In host FIB 1, the correct direction is:

10.0.2.0/23 → directly connected on ng_vimage0 (back toward the jail-side segment)
everything else (default) → out toward the ISP over the PPPoE link on ng0

So, a default route that uses ng0 in host FIB 1 is correct for jail-to-Internet forwarding when inbound traffic from ng_vimage0 is tagged to FIB 1.

Why “default link#… US ng0” can still be valid

route(8)() supports adding routes as interface routes (using -iface), and on point-to-point links the interface itself may be used as the “gateway” in the display output. That commonly produces “Gateway link#…” and no G flag in netstat -rn. This is not automatically wrong for PPP/PPPoE-style interfaces.

HAZARD: Do not “fix” a working PPPoE default route blindly
Replacing an interface default with a gateway default can break routing depending on how the PPP peer route is installed in that FIB. Only change this when there is a specific failure to correct and console access is available.

What the default gateway should be inside each VNET jail

VNET/VIMAGE jails have their own network stacks and their own routing tables and interfaces. They do not use the host’s FIB tables unless a shared-IP jail is used with exec.fib (different mechanism).

Inside each VNET jail, the default route should point to the host’s inside router address on that jail-facing subnet (in the example shown, 10.0.3.254 on the host side of the jail segment). It should not point to ng0, which is a host PPPoE interface.

Required forwarding/NAT pieces on the host

For RFC1918 traffic ( 10.0.2.0/23) to reach the public Internet over PPPoE, the host typically must:

Forward IPv4 packets between the inside segment and ng0 ( net.inet.ip.forwarding=1 / gateway_enable=yes).
NAT that private traffic to the public address on ng0, unless the upstream ISP routes 10.0.2.0/23 back to the host (uncommon).

HAZARD: Enabling forwarding/NAT changes host security posture
Packet forwarding and NAT turn the host into a router. Ensure firewall policy (pf/ipfw/ipf) is correctly defined before exposing services.

Verification commands (host)

Use setfib(1)() to view the effective routes for each FIB:

Code:

# setfib 1 netstat -rn
# setfib 0 netstat -rn

atax1a · Feb 20, 2026

chatgpt is not an expert in training, it is a statistical token generator

cracauer@ · Feb 20, 2026

AlfredoLlaquet · Feb 20, 2026

atax1a said:
chatgpt is not an expert in training, it is a statistical token generator

Actually, that was LLMs, but the new models with a reasoning loop are more than that. Also, I'm a writer, so I'm authorized by the failed-writers' guild to use metaphors when I please. And thirdly, I use a four-step method to generate the answer which is absolutely my own.

cracauer@ said:
wtf

Please, elaborate.

epopen · Feb 22, 2026

AlfredoLlaquet said:
Why “default via ng0” in FIB 1 is correct

The routing table shown as “fib: 1” contains host interfaces ( ng0, ng_vimage0), so it is a host routing table, not a VNET jail’s routing table.

In host FIB 1, the correct direction is:

10.0.2.0/23 → directly connected on ng_vimage0 (back toward the jail-side segment)

everything else (default) → out toward the ISP over the PPPoE link on ng0

So, a default route that uses ng0 in host FIB 1 is correct for jail-to-Internet forwarding when inbound traffic from ng_vimage0 is tagged to FIB 1.

Why “default link#… US ng0” can still be valid

route(8)() supports adding routes as interface routes (using -iface), and on point-to-point links the interface itself may be used as the “gateway” in the display output. That commonly produces “Gateway link#…” and no G flag in netstat -rn. This is not automatically wrong for PPP/PPPoE-style interfaces.

HAZARD: Do not “fix” a working PPPoE default route blindly
Replacing an interface default with a gateway default can break routing depending on how the PPP peer route is installed in that FIB. Only change this when there is a specific failure to correct and console access is available.

Hi AlfredoLlaquet

Thanks your answer & suggestion.

I'm tried

Set default gateway as ng_vimage0 to route table @ FIB :1
as follows
# route add default -interface ng_vimage0 -iface -fib 1
Result:

Code:

# netstat -nr -F 1
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags         Netif Expire
default            link#6             US       ng_vimage0
10.0.3.254         link#2             UHS             lo0
127.0.0.1          link#2             UHS             lo0

Test result: All of jail connection broken.

Set interface ng_vimage0 @ FIB :0 as follows.
# ifconfig ng_vimage0 10.0.3.254/23 fib 0
Result:

Code:

# ifconfig ng_vimage0
ng_vimage0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
        options=28<VLAN_MTU,JUMBO_MTU>
        ether 02:ac:95:e3:83:d2
        hwaddr 58:9c:fc:10:8a:17
        inet 10.0.3.254 netmask 0xfffffe00 broadcast 10.0.3.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=61<PERFORMNUD,AUTO_LINKLOCAL,NO_RADR>

Test result: All of jail connection broken also.

Mixed above two tried.
Test result: All of jail connection broken also.

Thanks you and all.

AlfredoLlaquet · 2026-02-28T18:05:51+0000

Given that you are still don't getting human answers, let's repeat the process.

WARNING 1: Seeing that you have received no additional responses in more than 72 hours, with nothing but good intentions in my heart, especially helping you, I outsourced your case to a non-subject expert-in-training and below the horizontal line is what it had to say.

WARNING 2: Use it all only as an inspiration or starting investigative point and check everything using official sources. That it sounds certain doesn't mean it is; these non-subjects are trained to first and foremost sound certain.

WARNING 3: Read in full before doing anything. Then, decide what you want to do.

Note: If you end up solving your problem, please write a post explaining what fixed it, so it gets documented for the community.

S: Restore jail connectivity after setting the wrong default route in FIB 1

HAZARD: Remote access may be lost immediately. Routing changes (especially default routes) can cut off SSH. Apply from console. Scope: host routing tables (FIB 0/1) and jail connectivity.

0) Confirm multi-FIB support is enabled

net.fibs must be 2 or more for FIB 1 to be usable. ifconfig(8)() requires multiple FIB support (ROUTETABLES / tunables) for fib N to be meaningful.

Code:

# sysctl net.fibs

HAZARD: Reboot required; boot configuration change. Setting net.fibs in /boot/loader.conf requires a reboot and can affect routing behavior system-wide.
If net.fibs is 1, set the loader tunable and reboot:

Code:

# sysrc -f /boot/loader.conf net.fibs=2

1) Tag inbound packets on ng_vimage0 to use FIB 1

This selects the FIB used for packets received on that interface; it does not create routes by itself. Also note: interface FIB is not inherited by VLANs or sub-interfaces.

HAZARD: Interface reconfiguration may disrupt connectivity. Changing interface address/FIB can interrupt traffic on the jail-side segment.

Code:

# ifconfig ng_vimage0 inet 10.0.3.254/23 fib 1
# ifconfig ng_vimage0

2) Remove the broken default route in host FIB 1

Setting default as “directly reachable” on the inside interface ( -iface) forces Internet-bound traffic back onto the jail-side segment and breaks forwarding.

HAZARD: Default route removal can drop existing sessions. Deleting default can immediately break outbound connectivity and management access depending on path selection.

Code:

# route -n -fib 1 delete default

3) Ensure FIB 1 contains the connected route back to the jail subnet

When net.add_addr_allfibs=0 (modern default), connected/interface routes may exist only in FIB 0 unless added explicitly per FIB. FreeBSD UPDATING documents the default change.

HAZARD: Routing table modification affects all traffic in that FIB. Adding/removing inside routes can break return traffic to jail IPs.

Code:

# route -n -fib 1 add -net 10.0.2.0/23 -iface ng_vimage0

4) Restore the FIB 1 default route to the PPPoE WAN (ng0)

On point-to-point links, using the interface as the default route is common and typically displays as default link#... US ng0 in netstat -rn.

HAZARD: Default route change can cause immediate outage. A wrong WAN default (or missing peer reachability in that FIB) breaks Internet access for jail-tagged traffic.

Code:

# route -n -fib 1 add default -iface ng0

5) Verify host routing tables

Use -F to select a FIB in netstat(1)():

Code:

# netstat -rn -F 1
# netstat -rn -F 0

Minimum expectation for FIB 1:

10.0.2.0/23 ... ng_vimage0
default ... ng0

6) Verify IPv4 forwarding on the host

HAZARD: Enabling forwarding changes security posture. Turning the host into a router increases exposure; ensure firewall/NAT policy is correct before relying on forwarding.

Code:

# sysctl net.inet.ip.forwarding

7) Data-plane verification (host and jail)

Use setfib(1)() to test using a specific routing table:

Code:

# setfib 1 ping -c 2 10.0.2.1    # replace with an actual jail IP
# setfib 1 ping -c 2 1.1.1.1

Inside a VNET jail, the default route should point to the host’s inside router IP (here 10.0.3.254), not ng0:

Code:

# jexec <jailname> netstat -rn
# jexec <jailname> ping -c 2 10.0.3.254
# jexec <jailname> ping -c 2 1.1.1.1

Make it robust across reconfiguration/reboot

A) Ensure interface routes are added to all FIBs automatically

FreeBSD UPDATING documents that the default for net.add_addr_allfibs changed to 0. Set it to 1 when relying on automatic connected routes in every FIB.

HAZARD: System-wide routing behavior change. This affects how interface routes appear across all FIBs; validate on console before relying on it.

Runtime:

Code:

# sysctl net.add_addr_allfibs=1

Persistent:

Code:

# sysrc -f /etc/sysctl.conf net.add_addr_allfibs=1

B) Persist ng_vimage0 address and FIB tagging

HAZARD: Boot-time network changes can prevent recovery access. Persisting incorrect interface configuration can break jail networking after reboot.

Code:

# sysrc ifconfig_ng_vimage0="inet 10.0.3.254/23 fib 1"

C) Persist the FIB 1 inside-subnet route (manual route approach)

rc.conf(5)() static routes can pass -fib 1 through to route(8)().

HAZARD: Persistent route errors survive reboot. Incorrect static routes can permanently break jail reachability until corrected from console.

Code:

# sysrc static_routes+=" jailnet_fib1"
# sysrc route_jailnet_fib1="-fib 1 -net 10.0.2.0/23 -iface ng_vimage0"

Confusing about default gateway with multiple FIB.

epopen

AlfredoLlaquet

atax1a

cracauer@

AlfredoLlaquet

epopen

AlfredoLlaquet