Network Interface Difficulty

z662

z662

I recently installed 10.2 Release and am trying to narrow down a few issues I am having. I am not sure if they are related to config files I had from 10.2 Stable or general configuration problems.

My two issues are that when I reboot the server there is no network connectivity until I restart the interface and firewall via: sudo /etc/rc.d/netif restart and sudo pfctl -d && sudo pfctl -f /etc/pf.conf

This issue seems like more of a timing conflict where a service on boot is not ready to start yet and it fails to grab a DHCP lease...maybe? I have posted some config files below in hopes that the issue can be spotted. I have never experienced this with a BSD system let alone thought that restarting the FW would do the trick.

My second issue is that I am unable to clone my loopback interface which I need so that I can set up an ezjail. When following the instructions here https://www.freebsd.org/doc/handbook/jails-ezjail.html I attempt to create a cloned interface (note the line already exists in my /etc/rc.conf but when manually creating via the CLI I get no output nor does the lo1 exist:

Code: 
brad@mercury:/usr/jails$ sudo service netif cloneup
brad@mercury:/usr/jails$ sudo ifconfig
Password:
re0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
  ether b8:97:5a:23:26:32
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect (none)
  status: no carrier
re1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
  ether 00:14:d1:2b:9c:b5
  inet 192.168.0.101 netmask 0xffffff00 broadcast 192.168.0.255
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
  options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
  inet6 ::1 prefixlen 128
  inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
  inet 127.0.0.1 netmask 0xff000000
  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  ether 02:fe:4a:c8:9c:00
  nd6 options=9<PERFORMNUD,IFDISABLED>
  id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
  maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
  root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
  member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
  ifmaxaddr 0 port 5 priority 128 path cost 2000000
  member: re1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
  ifmaxaddr 0 port 2 priority 128 path cost 20000
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=80000<LINKSTATE>
  ether 00:bd:69:2e:00:00
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect
  status: active
  Opened by PID 1096
brad@mercury:/usr/jails$

Code: 
brad@mercury:/usr/jails$ cat /etc/rc.conf
hostname="mercury.milkyway"
keyrate="fast"
#ifconfig_re0="192.168.0.101 netmask 255.255.255.0"
ifconfig_re1="SYNCDHCP"
ifconfig_bridge0="addm re1 addm tap0"
cloned_interfaces="${cloned_interfaces} lo1"
cloned_interfaces="bridge0 tap0"
pf_enable="YES"
#pflog_enable="YES"
gateway_enable="YES"
pf_rules="/etc/pf.conf"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="NO"
zfs_enable="YES"
sshd_enable="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
ezjail_enable="YES"
nfs_client_enable="YES"
#kern_securelevel_enable="YES"
#kern_securelevel="3"
local_unbound_enable="YES"

Any help would be greatly appreciated! If any other config files are needed please let me know.
 
Running a firewall with interfaces that get their address from DHCP requires some additional work. Please show your firewall configuration file for PF.
 
Ok cool, hopefully something simple then.

Here you are:

Code: 
brad@mercury:/usr/jails$ cat /etc/pf.conf
set skip on lo0
interface="re1"
apacheJail="192.168.0.102"
ircJail="192.168.0.103"
plexJail="192.168.0.104"
scrub in all
rdr pass on $interface proto tcp from any to $interface port 80 -> $apacheJail
rdr pass on $interface proto tcp from any to $interface port 6667 -> $ircJail

pass in on $interface proto tcp from any to any port 2662

block in on $interface
pass in on $interface proto udp from any to any port 1900
pass in on $interface proto tcp from any to any port 32400
pass in on $interface proto tcp from any to any port 32469
#pass in on $interface proto tcp from any to any port 8324
#pass in on $interface proto tcp from any to any port 3005
#pass in on $interface proto udp from any to any port 5353
pass in on $interface proto udp from any to any port 32410:32414
pass out on $interface proto {tcp,udp,icmp} all
 
From pf.conf(5):
Code: 
  Host name resolution and interface to address translation are done
  at ruleset load-time.  When the address of an interface (or host
  name) changes (under DHCP or PPP, for instance), the ruleset must
  be reloaded for the change to be reflected in the kernel.  Sur‐
  rounding the interface name (and optional modifiers) in parentheses
  changes this behaviour.  When the interface name is surrounded by
  parentheses, the rule is automatically updated whenever the inter‐
  face changes its address.  The ruleset does not need to be
  reloaded.  This is especially useful with nat.
 
Interesting...I did not know that. So I just need to update to this then?:

Code: 
interface=(re1)

If so, any ideas as to why I cannot create a cloned loopback interface as well?
 
Setting it with this won't work as it generates a parsing error in multiple places.
Code: 
interface=(re1)

When the context of the rule is appropriate, such as "to <the outside IP address of the interface at the time>" like this below then it should be used.
Code: 
rdr pass on $interface proto tcp from any to ($interface) port 80 -> $apacheJail
 
z662 said:
...
If so, any ideas as to why I cannot create a cloned loopback interface as well?
Click to expand...

These two lines should be reversed. The first adds to whatever precedes it. In this case nothing precedes it and the second line would just completely override that anyway.
Code: 
cloned_interfaces="${cloned_interfaces} lo1"
cloned_interfaces="bridge0 tap0"
 
Looks like I am still having some issues though. I have adjusted my ruleset to the below:

Code: 
brad@mercury:/home/brad$ cat /etc/pf.conf
set skip on lo0
interface="re1"
apacheJail="192.168.0.102"
ircJail="192.168.0.103"
plexJail="192.168.0.104"
scrub in all
rdr pass on $interface proto tcp from any to ($interface) port 80 -> $apacheJail
rdr pass on $interface proto tcp from any to ($interface) port 6667 -> $ircJail

pass in on $interface proto tcp from any to any port 2662

block in on $interface
pass in on $interface proto udp from any to any port 1900
pass in on $interface proto tcp from any to any port 32400
pass in on $interface proto tcp from any to any port 32469
#pass in on $interface proto tcp from any to any port 8324
#pass in on $interface proto tcp from any to any port 3005
#pass in on $interface proto udp from any to any port 5353
pass in on $interface proto udp from any to any port 32410:32414
pass out on $interface proto {tcp,udp,icmp} all
brad@mercury:/home/brad$


Whenever I reboot, my re1 ethernet interface does not have an IP so I still have to reset networking and my firewall. I got a syntax error when I used () around the pass in lines. Perhaps the final line needs them as well?

Just to play devil's advocate here, I have never had to do this and have been running a very similar if not identical ruleset for both 10.2 Stable and 9.x branch. Is it possible I just forgot something else or am not doing something correctly on the DHCP side of things?
 
z662 said:
Code: 
rdr pass on $interface proto tcp from any to ($interface) port 80 -> $apacheJail
rdr pass on $interface proto tcp from any to ($interface) port 6667 -> $ircJail
Click to expand...
These are not needed as the jails are bound to $interface and you can simply connect to the jail's IP addresses directly. You would need them if your jails are bound to lo1 for example.
 
Thanks Dice I will update those as soon as I can get my interface cloned (still unable to do so) and install the jails.
I am still having issues getting an IP address upon boot too. I am struggling to understand why these config files worked seemingly fine in Stable but not now. Not sure if I missed a step somewhere or need to make more changes to my config files.
 
The version of -STABLE I was using was compiled about 3 months ago, maybe 4. Is that still possible/likely those updates wouldn't have made it to -RELEASE?

I did not update/build world or anything afterwards. I was only on -STABLE due to some functionality for bhyve that was not available elsewhere at the time.

Does anyone have any suggestions/ideas on how I should proceed to debug and troubleshoot this issue? I am not sure what to even try at this point especially for the cloned interface issue considering it doesn't throw an error or complain about anything, just silently ignores the command essentially.
 
I could do that, although at this point I guess I would say that the cloned interfaces are what I would want to resolve first. To me, there is no point creating the jails without giving them a separate interface to run on. Problem is I don't see why they are failing. Do you know if there is a log or verbose way to run the create command so that I can see why it doesn't get cloned?

If I can't get the interfaces cloned then I will just put the old -STABLE drives back in and pick-up where I left off. Just was hoping to stay on -RELEASE for a change.
 
I was wrong, looks like the interface wasn't being cloned on my -STABLE install either. I guess I will not worry about it, although it could be causing issues with conflicting services on the same interface. I would prefer to use DHCP but may have to go static in the meantime.

Let me know if anyone thinks of something. Thanks
 
Sorry, I don't see what you are trying to do with the cloned interfaces. Why bridge0 and tap0? The only thing the ezjail section in the Handbook uses cloned interfaces for is the separate loopback interface for the jail. The jail's main IP address is not a cloned interface, just an alias that is created when the jail is started.
 
The bridge and tap interfaces are used for a bhyve virtual install I have. Sorry if they were causing confusion; they were not related.

I was able to get the loopback interface cloned for the jails that I needed to install. I noticed that restarting the rc.d netif service is no longer necessary however I still have to reset my pf rules on reboot.
 
You must log in or register to reply here.
Back
Top