Confusing about default gateway with multiple FIB.

epopen

epopen

Hi All

I am using VIMAGE(vnet) + netgraph(4) to build independence two jail(8) as follows.

螢幕快照 2026-02-09 14-59-06.png


Here, NIC ng_vimage0 using jail (IP range: 10.0.2.0~10.0.3.254) to access with host and internet, FIB=1
Code: 
ng_vimage0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
        options=28<VLAN_MTU,JUMBO_MTU>
        ether 02:ac:95:e3:83:d2
        hwaddr 58:9c:fc:10:8a:17
        inet 10.0.3.254 netmask 0xfffffe00 broadcast 10.0.3.255
        fib: 1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=61<PERFORMNUD,AUTO_LINKLOCAL,NO_RADR>
Host side, NIC ng0 using host access internet by PPPoE (Dynamic IPv4) using net/mpd5
Code: 
ng0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
        options=0
        inet 220.132.60.27 --> 168.95.98.254 netmask 0xffffffff
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

Next: host route table @ FIB=0
Code: 
Internet:
Destination        Gateway            Flags         Netif Expire
default            168.95.98.254      UGS             ng0
10.0.2.0/23        link#6             US       ng_vimage0
127.0.0.1          link#2             UH              lo0
168.95.98.254      link#5             UH              ng0
220.132.60.27      link#2             UHS             lo0
And For jail route table @ FIB=1
Code: 
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags         Netif Expire
default            link#5             US              ng0
10.0.2.0/23        link#6             U        ng_vimage0
10.0.3.254         link#2             UHS             lo0
127.0.0.1          link#2             UHS             lo0
(link#2 = lo0, link#5 = ng0 and link#6 = ng_vimage0)

The solution workable, but I confuing...
In jail route table @ FIB=1, default gateway = ng0 is correct?
(Because correct method I imagine it is that the default gateway is ng_vimage0,
and whether accessing the internet or the host from jail, it is routed through the system only (routing to ng0 when accessing the internet)
But I know I'm missing some key, so my imagination untenable).

If my current solution and imagine is incorrectly, would you can help to correct?

Thanks all very much.
 
WARNING 1: Seeing that you have received no additional responses in more than 72 hours, with nothing but good intentions in my heart, especially helping you, I outsourced your case to a non-subject expert-in-training and below the horizontal line is what it had to say.

WARNING 2: Use it all only as an inspiration or starting investigative point and check everything using official sources. That it sounds certain doesn't mean it is; these non-subjects are trained to first and foremost sound certain.

WARNING 3: Read in full before doing anything. Then, decide what you want to do.

Note: If you end up solving your problem, please write a post explaining what fixed it, so it gets documented for the community.

How the fib setting on ng_vimage0 works

ifconfig(8)(): ifconfig ... fib N assigns a FIB to frames/packets received on that interface. Route lookup for those packets uses that FIB. The setting is not inherited by VLANs or other sub-interfaces.

Given ng_vimage0 has fib: 1, traffic arriving from the jail-side segment into the host via ng_vimage0 is routed using host FIB 1.

HAZARD: Connectivity loss
Any routing-table change (especially default routes) can immediately cut off remote access to the host. Apply changes from local console access and keep rollback commands ready.


Why “default via ng0” in FIB 1 is correct

The routing table shown as “fib: 1” contains host interfaces ( ng0, ng_vimage0), so it is a host routing table, not a VNET jail’s routing table.

In host FIB 1, the correct direction is:
  • 10.0.2.0/23 → directly connected on ng_vimage0 (back toward the jail-side segment)
  • everything else (default) → out toward the ISP over the PPPoE link on ng0

So, a default route that uses ng0 in host FIB 1 is correct for jail-to-Internet forwarding when inbound traffic from ng_vimage0 is tagged to FIB 1.


Why “default link#… US ng0” can still be valid

route(8)() supports adding routes as interface routes (using -iface), and on point-to-point links the interface itself may be used as the “gateway” in the display output. That commonly produces “Gateway link#…” and no G flag in netstat -rn. This is not automatically wrong for PPP/PPPoE-style interfaces.

HAZARD: Do not “fix” a working PPPoE default route blindly
Replacing an interface default with a gateway default can break routing depending on how the PPP peer route is installed in that FIB. Only change this when there is a specific failure to correct and console access is available.


What the default gateway should be inside each VNET jail

VNET/VIMAGE jails have their own network stacks and their own routing tables and interfaces. They do not use the host’s FIB tables unless a shared-IP jail is used with exec.fib (different mechanism).

Inside each VNET jail, the default route should point to the host’s inside router address on that jail-facing subnet (in the example shown, 10.0.3.254 on the host side of the jail segment). It should not point to ng0, which is a host PPPoE interface.


Required forwarding/NAT pieces on the host

For RFC1918 traffic ( 10.0.2.0/23) to reach the public Internet over PPPoE, the host typically must:
  • Forward IPv4 packets between the inside segment and ng0 ( net.inet.ip.forwarding=1 / gateway_enable=yes).
  • NAT that private traffic to the public address on ng0, unless the upstream ISP routes 10.0.2.0/23 back to the host (uncommon).

HAZARD: Enabling forwarding/NAT changes host security posture
Packet forwarding and NAT turn the host into a router. Ensure firewall policy (pf/ipfw/ipf) is correctly defined before exposing services.


Verification commands (host)

Use setfib(1)() to view the effective routes for each FIB:

Code: 
# setfib 1 netstat -rn
# setfib 0 netstat -rn
 
AlfredoLlaquet said:
Why “default via ng0” in FIB 1 is correct

The routing table shown as “fib: 1” contains host interfaces ( ng0, ng_vimage0), so it is a host routing table, not a VNET jail’s routing table.

In host FIB 1, the correct direction is:
  • 10.0.2.0/23 → directly connected on ng_vimage0 (back toward the jail-side segment)
  • everything else (default) → out toward the ISP over the PPPoE link on ng0

So, a default route that uses ng0 in host FIB 1 is correct for jail-to-Internet forwarding when inbound traffic from ng_vimage0 is tagged to FIB 1.


Why “default link#… US ng0” can still be valid

route(8)() supports adding routes as interface routes (using -iface), and on point-to-point links the interface itself may be used as the “gateway” in the display output. That commonly produces “Gateway link#…” and no G flag in netstat -rn. This is not automatically wrong for PPP/PPPoE-style interfaces.

HAZARD: Do not “fix” a working PPPoE default route blindly
Replacing an interface default with a gateway default can break routing depending on how the PPP peer route is installed in that FIB. Only change this when there is a specific failure to correct and console access is available.
Click to expand...
Hi AlfredoLlaquet

Thanks your answer & suggestion.

I'm tried
  • Set default gateway as ng_vimage0 to route table @ FIB :1
    as follows
    # route add default -interface ng_vimage0 -iface -fib 1
    Result:
    Code: 
    # netstat -nr -F 1
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags         Netif Expire
default            link#6             US       ng_vimage0
10.0.3.254         link#2             UHS             lo0
127.0.0.1          link#2             UHS             lo0
    Test result: All of jail connection broken. 🙀
  • Set interface ng_vimage0 @ FIB :0 as follows.
    # ifconfig ng_vimage0 10.0.3.254/23 fib 0
    Result:
    Code: 
    # ifconfig ng_vimage0
ng_vimage0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
        options=28<VLAN_MTU,JUMBO_MTU>
        ether 02:ac:95:e3:83:d2
        hwaddr 58:9c:fc:10:8a:17
        inet 10.0.3.254 netmask 0xfffffe00 broadcast 10.0.3.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=61<PERFORMNUD,AUTO_LINKLOCAL,NO_RADR>
    Test result: All of jail connection broken also. 🙀
  • Mixed above two tried.
    Test result: All of jail connection broken also. 🙀

Thanks you and all.
 
You must log in or register to reply here.
Back
Top