Using second NIC as FIB on different subnet

Woden27 · Jan 11, 2022

I configured rc.conf to support two interfaces with FIB

Code:

ifconfig_igb0="inet 203.1.1.7 netmask 255.255.255.192 fib 0"
ifconfig_igb1="inet 192.168.20.2 netmask 255.255.255.0 fib 1"
static_routes="Nic0 Nic1"
route_Nic0="default 203.1.1.1 -fib 0"
route_Nic1="default 192.168.20.1  -fib 1"

and the /boot/loader.conf

Code:

sysctl net.fibs
net.fibs: 2

ipfw added

Code:

00110 setfib 0 ip from any to any via igb0
00120 setfib 1 ip from any to any via igb1

if I try to ping from the machine itself with

Code:

$ setfib 1 ping 192.168.20.2
PING 192.168.20.2 (192.168.20.2): 56 data bytes
64 bytes from 192.168.20.2: icmp_seq=0 ttl=64 time=0.102 ms
64 bytes from 192.168.20.2: icmp_seq=1 ttl=64 time=0.084 ms
64 bytes from 192.168.20.2: icmp_seq=2 ttl=64 time=0.077 ms

Is pinging . But if I try to ssh to that address it's going in timeout..

$ setfib 1 ssh 192.168.20.2

Code:

$setfib 1 netstat -r4
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.20.1       UGS        igb1
localhost          link#5             UHS         lo0
192.168.20.0/24    link#2             U          igb1

What I'm missing here?

SirDice · Jan 11, 2022

Woden27 said:
But if I try to ssh to that address it's going in timeout..

Firewall on the destination host blocking the traffic?

Woden27 · Jan 11, 2022

the destination host is the second NIC itslef

SirDice · Jan 11, 2022

Oh, right. I see that now. Is sshd(8) actually listening on that address? What does sockstat -4 tell you?

covacat · Jan 11, 2022

probably listens otherwise it would be connection refused
but ssh replies from fib0 and will send to 192.168.x.x thru the internet which will fail

Woden27 · Jan 11, 2022

SirDice said:
Oh, right. I see that now. Is sshd(8) actually listening on that address? What does sockstat -4 tell you?

root sshd 3960 5 tcp4 192.168.20.2:22 *:*

Woden27 · Jan 11, 2022

covacat said:
probably listens otherwise it would be connection refused
but ssh replies from fib0 and will send to 192.168.x.x thru the internet which will fail

Probably something like this, but how I can solve it? ?

VladiBG · Jan 11, 2022

I don't know what is your end goal for this but if it's to have redundancy WAN access then you need dynamic routing or if it's not possible then go for SLA tracking.

Woden27 · Jan 11, 2022

VladiBG said:
I don't know what is your end goal for this but if it's to have redundancy WAN access then you need dynamic routing or if it's not possible then go for SLA tracking.

Just I would like to reach another network from the NIC interface igb1...nothing special !
I am trying to test the connectivity, but I cannot reach the 192.168.20.0/24 in any way..neither from the same machine and the same interface. (but when attached to the other router is the same -- timeout)

covacat · Jan 11, 2022

you can probably use ipfw to set fib 1 on any packet originating on 192.168.20.2

Woden27 · Jan 11, 2022

covacat said:
you can probably use ipfw to set fib 1 on any packet originating on 192.168.20.2

Code:

00100 allow ip from any to any via lo0
00110 setfib 0 ip from any to any via igb0
00120 setfib 1 ip from any to any via igb1
00125 setfib 1 ip from any to any via 192.168.20.2

even if I delete rule 00120 -- same
even if I disable ipfw -- same

VladiBG · Jan 11, 2022

Woden27 said:
Just I would like to reach another network from the NIC interface igb1

Then you can use static route to that network via igb1. You don't need multiple FIB not/or two default gateways.

Woden27 · Jan 11, 2022

VladiBG said:
Then you can use static route to that network via igb1. You don't need multiple FIB not/or two default gateways.

I changed to
route_Nic1="-net 192.168.20.0/24 192.168.20.1 -fib 1"

again, I can ping the NIC address , but when I try to ssh it , timeout...

ipfw service is stopped...and sshd is listening !

root sshd 1728 5 tcp4 192.168.20.2:22 *:*

VladiBG · Jan 11, 2022

sshd is running on fib0

Woden27 · Jan 11, 2022

VladiBG said:
Then you can use static route to that network via igb1. You don't need multiple FIB not/or two default gateways.

OK i solved removing all the FIB statements and using the static route for igb1.
This works, but anyway, I would like to know why with FIBs fails?

Woden27 · Jan 11, 2022

VladiBG said:
sshd is running on fib0

I even added these statements to sshd_config ...

ListenAddress 0.0.0.0
ListenAddress 192.168.20.2

VladiBG · Jan 11, 2022

The igb1 has 192.168.20.2/24 so you don't need to define a static route for it via 192.168.20.1 as it's connected route and this gateway will never be used to reach rest of the hosts in 192.168.20.0/24.

in sshd

ListenAddress 0.0.0.0

means that it will listen on all IPs. In configuration files only the last statement is apply as it's overwrite the previous one. So specifying multiple times the same statement make no sense.

If you are interested in networking i recommend you to start with CCNA Routing and Switching Lessons

Woden27 · Jan 11, 2022

The igb1 has 192.168.20.2/24 so you don't need an to define a static route for it via 192.168.20.1 as it's connected route and this gateway will never be used to reach rest of the hosts in 192.168.20.0/24.

This makes sense to me.

means that it will listen on all IPs. In configuration files only the last statement is apply as it's overwrite the previous one. So specifying multiple times the same statement make no sense.

even this sounds ok.

The fact I was confused -- and the reason I used FIBs was actually the default router statement.
If my configuration is:

ifconfig_igb0="inet 203.1.1.7 netmask 255.255.255.192"
ifconfig_igb1="inet 192.168.20.2 netmask 255.255.255.0"
defaultrouter="203.1.1.1"

How the traffic from the interface igb1 can flow without a "default router" statement or a static route?

This doesn't solve the problem with FIBs and why in that case the address is pingable but not sshable ..

VladiBG · Jan 11, 2022

Woden27 said:
How the traffic from the interface igb1 can flow without a "default router" statement or a static route?

It receive directly connected route for 192.168.20.0/24 via igb1. Same as directly connected route for 203.1.1.0/26 via igb0. Those subnets are accessible without routing the traffic trough the gateway/router/.

covacat · Jan 11, 2022

if you want all traffic from from 192.168.20.2 to go to internet thru 192.168.20.1
add allow ip from 192.168.20.0/24 to 192.168.20.0/24 via igb1
add allow ip from 192.168.20.2 to me
add fwd 192.168.20.1 ip from 192.168.20.2 to any
no fibs
ping -S 192.168.20.2 8.8.8.8
or you can probably run multiple instances of sshd or any other services and bind them separately on igb0 and igb1 and use fibs

Woden27 · Jan 12, 2022

VladiBG said:
It receive directly connected route for 192.168.20.0/24 via igb1. Same as directly connected route for 203.1.1.0/26 via igb0. Those subnets are accessible without routing the traffic trough the gateway/router/.

Actually removing the statement

defaultrouter="203.1.1.1"

will make the server accessible from the LAN but not from the WAN with PAT from the main router.

Woden27 · Jan 13, 2022

covacat said:
if you want all traffic from from 192.168.20.2 to go to internet thru 192.168.20.1
add allow ip from 192.168.20.0/24 to 192.168.20.0/24 via igb1
add allow ip from 192.168.20.2 to me
add fwd 192.168.20.1 ip from 192.168.20.2 to any
no fibs
ping -S 192.168.20.2 8.8.8.8
or you can probably run multiple instances of sshd or any other services and bind them separately on igb0 and igb1 and use fibs

Tried, I can ping the default GW (192.168.20.1) but cannot reach the server to 192.168.20.2 via ssh
I cannot ping outside on the internet neither from the interface igb1
It just goes in timeout.

This is what I have at the moment in my rc.conf

ifconfig_igb0="inet 203.1.1.7 netmask 255.255.255.192"
ifconfig_igb1="inet 192.168.20.2 netmask 255.255.255.0"
defaultrouter="203.1.1.1"
static_routes="Nic1 2df"
route_Nic1="-net 192.168.20.0/24 -iface igb1"
route_2df="default 192.168.20.1 -iface igb1"

i addedd the rules suggested with

ipfw (role number after all the allow statements) add allow ip from 192.168.20.0/24 to 192.168.20.0/24 via igb1

etc..

covacat · Jan 13, 2022

you have to have some kind of nat on 192.168.20.1 in order to access internet from 192.168.20.2

Woden27 · Jan 13, 2022

Nat is working correctly.
To prove that i just connected a laptop with windows

with the same address (static) to the same port in the same VLAN, and i can ping google without problems..

must be something else, but honestly I have now no idea

covacat · Jan 13, 2022

Code:

box1 connected to internet via ng0 (pppoe)
10.1.1.1 is on internal lan and gateway for the rest of the lan

~#netstat -rn|grep defau
default            10.0.0.1           UGS         ng0


box2 has 10.1.1.42 and an openvpn connection to another host
utun4: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1500
        inet 10.255.255.10 --> 10.255.255.9 netmask 0xffffffff

on box2
route add 11.12.13.14/32 10.255.255.9  (send traffic to 11.12.13.14 thru openvpn)
bash-3.2# netstat -rn |grep 11.12.13.14
11.12.13.14/32     10.255.255.9       UGSc         utun4      

on box1
ipfw add 5 fwd 10.1.1.42 icmp from 10.1.1.1 to 11.12.13.14
send icmp traffic from 10.1.1.1 to box2
ping -S 10.1.1.1 11.12.13.14
PING 11.12.13.14 (11.12.13.14) from 10.1.1.1: 56 data bytes

on box2
bash-3.2# tcpdump -i utun4 -nl
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on utun4, link-type NULL (BSD loopback), capture size 262144 bytes
13:17:56.527595 IP 10.1.1.1 > 11.12.13.14: ICMP echo request, id 38695, seq 11, length 64
13:17:57.531496 IP 10.1.1.1 > 11.12.13.14: ICMP echo request, id 38695, seq 12, length 64

so traffic sent from box1 10.1.1.1 to 11.12.13.14 is going thru the openvpn connection
no routes have been set on box1 just ipfw fwd

box1 is freebsd 12.3 box2 is a mac

Using second NIC as FIB on different subnet

Woden27

SirDice

Administrator

Woden27

SirDice

Administrator

covacat

Woden27

Woden27

VladiBG

Woden27

covacat

Woden27

VladiBG

Woden27

VladiBG

Woden27

Woden27

VladiBG

Woden27

VladiBG

covacat

Woden27

Woden27

covacat

Woden27

covacat