Hi everyone!
We are using jails in FreeBSD since years (9.x and up) without problems. Since then we have two network cards in the (xen virtualized) machines, where one is private (10.0.0.0/8) and one is public (here changed to 1.2.3.0/20). To have the routing done, we use setfib for the public network, which worked always without problems.
Since a few month, we move a few hundred machines to 12.0-RELEASE-p10 and now (or maybe earlier, since we do not check each machine, each day or so) we experience that the sendmail on port 25 sometimes stop to talk correctly with the outside world. The communication from and to machines within the 1.2.3.0/20 keeps working. If we restart the sendmail process, then all communication in all networks is running again (by now, we can't say when and how fast the error will come back).
In the host, there is a sendmail on port 25 on loopback, not on any of the NICs.
Currently we're just digging around and one clear difference at least between 10.x and 12.0 seems to be the view of netstat -nr with setfib from the host view:
in 10.x:
[root@10-0-4-144-host ~]# netstat -rn
[root@10-0-4-142-host ~]# setfib 2 netstat -rn
In 12.0:
[root@10-0-4-144-host ~]# netstat -rn
[root@10-0-4-144-host ~]# setfib 2 netstat -rn
/etc/rc.conf:
As one can see, on FreeBSD 10.x the netstat don't show up the routing of the other fibs as expected. The 12.0 shows it, what is not expected.
So my question is: is this just a 'false' printing in the FreeBSD 12.0, or is it wrong and therefore any change between 10.x and 12.0 regarding setfib? So is there a new way to set this up? Or is it a bug?
Many thanks in advance for your input!
Jimmy
We are using jails in FreeBSD since years (9.x and up) without problems. Since then we have two network cards in the (xen virtualized) machines, where one is private (10.0.0.0/8) and one is public (here changed to 1.2.3.0/20). To have the routing done, we use setfib for the public network, which worked always without problems.
Since a few month, we move a few hundred machines to 12.0-RELEASE-p10 and now (or maybe earlier, since we do not check each machine, each day or so) we experience that the sendmail on port 25 sometimes stop to talk correctly with the outside world. The communication from and to machines within the 1.2.3.0/20 keeps working. If we restart the sendmail process, then all communication in all networks is running again (by now, we can't say when and how fast the error will come back).
In the host, there is a sendmail on port 25 on loopback, not on any of the NICs.
Currently we're just digging around and one clear difference at least between 10.x and 12.0 seems to be the view of netstat -nr with setfib from the host view:
in 10.x:
[root@10-0-4-144-host ~]# netstat -rn
Code:
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default 10.0.0.1 UGS xn0
10.0.0.0/16 link#5 U xn0
10.0.4.144 link#5 UHS lo0
127.0.0.1 link#4 UH lo0
Internet6:
Destination Gateway Flags Netif Expire
::/96 ::1 UGRS lo0
::1 link#4 UH lo0
::ffff:0.0.0.0/96 ::1 UGRS lo0
fe80::/10 ::1 UGRS lo0
fe80::%lo0/64 link#4 U lo0
fe80::1%lo0 link#4 UHS lo0
ff01::%lo0/32 ::1 U lo0
ff02::/16 ::1 UGRS lo0
ff02::%lo0/32 ::1 U lo0[root@10-0-4-142-host ~]# setfib 2 netstat -rn
Code:
Routing tables (fib: 2)
Internet:
Destination Gateway Flags Netif Expire
default 1.2.3.4 UGS xn1
1.2.3.0/20 link#6 U xn1
1.2.3.8 link#6 UHS lo0
Internet6:
Destination Gateway Flags Netif Expire
fe80::%lo0/64 link#4 U lo0In 12.0:
[root@10-0-4-144-host ~]# netstat -rn
Code:
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default 10.0.0.1 UGS xn0
10.0.0.0/16 link#4 U xn0
10.0.4.144 link#4 UHS lo0
1.2.3.0/20 link#5 U xn1
127.0.0.1 link#1 UH lo0
Internet6:
Destination Gateway Flags Netif Expire
::/96 ::1 UGRS lo0
::1 link#1 UH lo0
::ffff:0.0.0.0/96 ::1 UGRS lo0
fe80::/10 ::1 UGRS lo0
fe80::%lo0/64 link#1 U lo0
fe80::1%lo0 link#1 UHS lo0
ff02::/16 ::1 UGRS lo0[root@10-0-4-144-host ~]# setfib 2 netstat -rn
Code:
Routing tables (fib: 2)
Internet:
Destination Gateway Flags Netif Expire
default 1.2.3.4 UGS xn1
10.0.0.0/16 link#4 U xn0
1.2.3.0/20 link#5 U xn1
1.2.3.8 link#5 UHS lo0
127.0.0.1 link#1 UH lo0
Internet6:
Destination Gateway Flags Netif Expire
::/96 ::1 UGRS lo0
::1 link#1 UH lo0
::ffff:0.0.0.0/96 ::1 UGRS lo0
fe80::/10 ::1 UGRS lo0
fe80::%lo0/64 link#1 U lo0
ff02::/16 ::1 UGRS lo0/etc/rc.conf:
Code:
ifconfig_xn0="inet 10.0.4.144/16 mtu 1500"
ifconfig_xn1="inet 1.2.3.8/20 fib 2"
defaultrouter="10.0.0.1"
static_routes="vsnetwork vsdefault"
route_vsnetwork="-net 1.2.3.0/20 -interface xn1 -fib 2"
route_vsdefault="default 1.2.3.4 -fib 2"As one can see, on FreeBSD 10.x the netstat don't show up the routing of the other fibs as expected. The 12.0 shows it, what is not expected.
So my question is: is this just a 'false' printing in the FreeBSD 12.0, or is it wrong and therefore any change between 10.x and 12.0 regarding setfib? So is there a new way to set this up? Or is it a bug?
Many thanks in advance for your input!
Jimmy
