jails Network issue with vlan jail on dedicated port

S

SamKook

I've been trying to make a jail on my server that is part of a separate vlan from the host and I'm having a hard time figuring out what should be configured on the host and what should be in the jail.

At first I had my 2 network ports on the server in a lagg configuration and a bridge with epairs for the jail and things were working mostly fine in the jail. I had internet access, but there were some issues loading a web ui and other odd behavior so I figured things would be better if I undid separated the 2 ports, one for the host and one for the jail.

The host is working fine, I bring up igb0 and give it a dhcp address and all is well on the host side.
But for the jail side, I'm not sure if I should bring up igb1 on the host or in the jail. I assume the vlan bridge should be created in the jail config file, but I've not 100% on that either.

No matter what I try, I can't ping the gateway. I've managed to ping another pc inside the vlan once(I think with igb1 and the bridge only in the ifconfig output from inside the jail) but never the gateway, it either says the network is down, network is unreachable or the ping hangs forever.

Basically, I bring igb1 up wherever and do this for the vlan(# is whatever number I'm testing with):
Code: 
/sbin/ifconfig igb1.20 create vlan 20 vlandev igb1 inet 10.10.20.#/24 up

Knowing what goes where would be really helpful in figuring out where the problem is(I've tripled checked I didn't forget to change anything when I undid the lagg stuff but you never know) or if that above is even correct.
 
Is the switch port configured to accept the VLAN tags? Your switch needs to be aware of that. The basic run-of-the-mill unmanaged switch typically does not support VLANs.
 
SamKook said:
But for the jail side, I'm not sure if I should bring up igb1 on the host or in the jail. I assume the vlan bridge should be created in the jail config file, but I've not 100% on that either.
Click to expand...
depends - if only this jail is using that interface, you can pass-through the raw interface via vnet.interface.
If you also want to attach other jails, you usually configure the bridge (and vlan interface) on the host, then attach the epair interface that is connected to the jail to that bridge.

regarding your connection issues: apart from what SirDice already mentioned (the switch obviously needs to support vlans); is the gateway also available in that vlan? i.e. it has the relevant vlan configuration or is attached to an access port for that vlan?
post the output of ifconfig and the relevant parts of rc.conf from the jailhost and gateway.
 
Yes, the switch is configured to accept the vlan tag on that port(that is the only thing that changed from the previously kinda working configuration since it's not lagg anymore so I checked that a lot) and also the router port and the lagg interface on my proxmox server that I use for testing stuff.

Only that jail will be using the interface and I was apparently already passing it through with vnet.interface without really knowing what it did exactly.
The problem is that there is no vlan tag if all I do is passthrough the interface so what is the proper way to configure that?
What I was trying was executing the line from the code tag from my first post once the interface is passed through, but I had the issue previously mentioned hence why I'm trying to get a clear picture of exactly how it should be configured before trying to diagnose the whole network. I'm assuming I don't need epairs if I pass it through but I don't know.

The gateway(on the router) is configured for that vlan and icmp stuff is allowed to go from anywhere to everywhere for testing so ping should work on anything(works fine in a linux VM from my proxmox server).
Right now all that's configured is the passthrough so the ifconfig output wouldn't be much help. Once the jail is running, the host has igb0 with a dhcp ip and the jail has igb1 with a static ip(dhclient doesn't work inside the jail, is that the right way to do dhcp or can that even be done from inside?) set to the vlan address range but no vlan tag so I wasn't expecting that to work(does it need an ip if I also create a vlan interface or should only that vlan interface have an ip and igb1 just be up).
 
SamKook said:
Only that jail will be using the interface and I was apparently already passing it through with vnet.interface without really knowing what it did exactly.
The problem is that there is no vlan tag if all I do is passthrough the interface so what is the proper way to configure that?
Click to expand...
with vnet.interface you can pass through the physical interface (e.g. igb0). anything else (vlan) is configured within the jail as you would do on a host - i.e. via rc.conf
I'm not sure if you could also use a vlan interface with vnet.interface - never tried it as it doesn't make much sense (just use normal vnet/epair interfaces for such configurations).

The easiest solution if that interface always and only needs to carry traffic for vlan 20 would be to configure the port on the switch as an access port.
 
Oh, I'm pretty new to vlans(started on that 4 days ago) so I did not think to make the port dedicated to that vlan since I was assuming it would stay lagg when I started and set everything up.
That did instantly fix the network issue. Hmm, still not working, I tested in the wrong window, weird.

Now I'm curious though, it sounds like what I was doing before was the correct way to configure things or am I missing something?
That is, passthrough the interface to the jail and then configure a vlan interface with something like this inside of it:
Code: 
ifconfig igb1.20 create vlan 20 vlandev igb1 inet 10.10.20.100/24 up
and with igb1 up but with no ip. Meaning the host has igb0 and the jail has igb1(no ip) and igb1.20(vlan ip).
Making a separate interface to configure it just seems odd to me but that seems to be the way to do it in freebsd so I just want to make sure I understand that correctly.
 
SamKook said:
with igb1 up but with no ip
Click to expand...
That's fine, it doesn't need one. The igb1.20 won't need one either. But it depends where on your network you're going to do the routing.

igb1 -> igb1.20 -> bridge0 -> epair0a <--> epair0b-[vnet0 VM]
Really only the VM itself needs an IP address in the VLAN20 range, but don't do the VLAN tagging there, the bridge0 has the untagged VLAN20 traffic.

The vlan(4) interfaces add or remove the VLAN tag in the frames, depending on the direction of the traffic.

[tagged 20] -> [igb1] -> [tagged 20] -> [igb1.20] -> [untagged!]
 
Are you saying I need a bridge even if I do a passthrough of the interface? It doesn't feel like you understood the setup I described or I need a more specific explanation of where each of those are supposed to be located in.

As I have things now, the host only has igb0 and the loopback interface and the VM(jail) has igb1, (potentially igb1.20) and the loopback.
 
SamKook said:
Are you saying I need a bridge even if I do a passthrough of the interface? It doesn't feel like you understood the setup I described or I need a more specific explanation of where each of those are supposed to be located in.
Click to expand...

Right. Then just do:
igb1 -> igb1.20 -> attach jail.

rc.conf
Code: 
vlans_igb1="20"
ifconfig_igb1="up"
ifconfig_igb1.20="inet 10.10.20.1 netmask 255.255.255.0"
Then in your jail.conf
Code: 
myjail:
  {
     interface = igb1.20;
     ip4.addr = 10.10.20.23/24;
     {...}
  }
 
I'm assuming "ifconfig_igb1.20" needs to be "ifconfig_igb1_20" in rc.conf since that's what I was seeing while researching this and the doc for rc.conf mentions this:
An ifconfig_<interface> variable is assumed to exist for
each value of interface. When an interface name contains
any of the characters ".-/+" they are translated to "_" be-
fore lookup. For example, the interface em0.102 would be
configured using the variable ifconfig_em0_102.
Click to expand...
As for the ips, 10.10.20.1 is my gateway address so I'm assuming I don't want to set igb1.20 to that, unless that configures the gateway, but that's not my understanding.
Do I need an ip for both outside and inside the jail? This seems weird for passthrough but I am pretty confused about how it's supposed to work.

Edit: I realized when trying to test with what you just said that I forgot to mention I'm doing a vnet jail since I got this error "vnet jails cannot have IP address restrictions" because of the ip4.addr line.
I assumed that was required since it's meant to run on a separate vlan from the host.
 
Disabling vnet, enabling ip4.addr and having 2 ips for igb1.20(shows the 2 on the host and 1 in the jail) does work for pinging the gateway and other pc in the same vlan, but I don't have internet access and still have the gateway problem with a webui I was having with my old lagg setup.

Edit: Same thing with vnet enabled and igb1.20 passed through with vnet.interface and assigning an ip to it.
igb1 is then left on the host(with no ip) and the jail only shows igb1.20(with ip).
 
SamKook said:
I'm assuming "ifconfig_igb1.20" needs to be "ifconfig_igb1_20" in rc.conf
Click to expand...
Oh, yes. Typed it mostly from the top of my head.

SamKook said:
As for the ips, 10.10.20.1 is my gateway address so I'm assuming I don't want to set igb1.20 to that
Click to expand...
No, that was just an example. You definitely don't want to create IP conflicts.

Need to have a think about the situation, don't normally use VNET jails. Do use lots of VMs on different VLANs. So far the easiest way I could come up with would be to leverage a bridge(4) and attach an epair(4) to the VNET jail. Then you can simply set the IP address, gateway and other settings within the jail itself.

The bridge also provides a point to attach other jails and/or VMs on. Bit more future proof.
 
I was using a bridge with epairs back when the 2 network ports were configured as lagg, but I had a gateway issue with a webui so I was hoping separating the 2 interface might help fix that.
I'm really confused about how to tell the host which interface it should be using and which should be ignored for itself when setting up both on the host like I would need to if I were to do a bride config now. Avoiding the bridge and using passthrough sounds like a better idea since it avoids both that problem(maybe) and it should be the best performance and simpler.

It is extremely unlikely that I would want other jails on that interface. If I make any more, they'll likely be on the host interface instead since I'd almost certainly want them on that network instead.
That's why I'm trying to have 1 port for the host and 1 for that vlan, it fits my needs perfectly and makes things less complex so easier to debug.
 
SamKook said:
I'm really confused about how to tell the host which interface it should be using
Click to expand...
The host itself will simply look at the routing table to see where it should send its packets to (for both outgoing connections, and responses to incoming packets). People new to networking always seem to overlook the fact routing happens based on the destination IP address in a packet. It does not 'remember' where it came from, or on what interface it came in on.

Unless you have a bunch of static routes, the host typically uses whatever address is set as default gateway. Based on that address it will look for the appropriate interface that has the 'directly' connected network.

For example:
Code: 
% netstat -rn4
Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            195.x.x.x          UGS             ng0
10.0.0.0/24        link#5             U             em1.1
10.0.0.1           link#3             UHS             lo0
10.0.1.0/24        link#8             U            em1.20
10.0.1.1           link#3             UHS             lo0
37.y.y.y           link#3             UHS             lo0
127.0.0.1          link#3             UH              lo0
192.168.10.0/24    link#6             U            em1.10
192.168.10.1       link#3             UHS             lo0
192.168.11.0/24    link#7             U            em1.11
192.168.11.1       link#3             UHS             lo0
195.x.x.x          link#10            UH              ng0
As you can see I have a bunch of 'directly connected' networks on em1.1, em1.20, etc. Those 'directly connected' networks have an implied route based on the interface's IP and subnet mask. When the host tries to make a connection to 10.0.0.123 for example, it will lookup the route, finds 10.0.0.0/24 attached to em1.1 and will send the packet out that interface.
 
This is driving me mad.

If I plug in a random windows pc on the switch vlan 20 port I use for igb1, it automatically gets an ip in the dhcp range for 10.10.20.1 so I can only conclude that the vlan stuff is configured properly.

If I plug in igb1 into the switch port for igb0 and leave igb0 unplugged, it gets an ip in the base network and if I pass that interface through to a jail, I lose connection on the host and the jail has network and internet connection so I can only conclude I now know how to configure a passthrough connection to a jail.

But no matter what I try, I can't get that freebsd server to get a dhcp ip on either igb0 or igb1 when I connect them to the switch vlan 20 port.
At least that narrows that the scope of the problem, but I'm at a lost for what the issue could be at this point, my rc.conf relevant line is only this by now:
Code: 
ifconfig_igb0="up dhcp"
And a static ip doesn't work any better.
 
Well, apparently the freebsd interface has to be tagged even if the switch port is setting the id. If I create a vlan interface and pass that through instead of the raw interface, it can see the gateway.
For some reason it didn't worked when I used epairs earlier today, not sure why, I must have messed up something somehow.
And if I passthrough igb1 and create the vlan interface inside the jail, it doesn't work, which is what I was always trying.

At least I learned a lot about how freebsd networking works, but I'm still getting that pesky gateway problem this way too and still don't know why some of this doesn't work.
 
Read through this thread. It’s been a while since I dealt with VNET, VLAN, and jails. So, I can’t quite pin point what was going on with your setup.

Just wanted to jump in and provide some hard won advice.

I found a few things to be true. Bridges don’t relay VLAN tags. Neither do epairs. A physical interface should be in Access or Trunk mode, but never Hybrid. I found I would experience packet leakage when an interface had vlan tags and also a default tag (untagged). The solution was to make sure the interface on the FreeBSD device and the switch was trunk mode only with no default (untagged) vlan. The implications of the above is that for any particular interface that carries vlan tagging, if you wish to distribute individual vlans to a jail, you need a bridge dedicated to that vlan and need to separate it out from the interface as a tagged vlan, connect it to the bridge and pass a connected epair from the bridge to the jail. Repeat for whatever pairs of jail x vlan tags you have.

Or, pass the entire interface into the jail.

I’m not quite sure what your dhcp issue was, but dhcp assumes an untagged interface as it’s only looking on the local network for the dhcp server. DHCP request messages are tagged with TTL 1 and won’t go beyond the local network. There’s no such thing as an all vlan broadcast message, so the dhcp client request message needs to go out onto the network correctly tagged for the targeted vlan where a dhcp server is listening. I’m guessing something in your setup wasn’t properly tagging the dhcp request message with the vlan where the dhcp server was listening.

There’s also dhcp forwarding technology available on some switches, however I wouldn’t mess with that at this time and instead work on making sure the client is on the correct network in the first place.

As a means of testing this, you can hand configure the jail with a static IP address and gateway and test the network connection with ping to various machines. If those work, then reconfigure for dhcp and give it a try. Also, dhcp can be run from the command line (see the man pages). Which means you don’t have to wait on the jail to start/stop to get dhcp working.

Oh, one last thing. DHCP requires access to a particular device. I don’t recall which one off the top of my head. This means you have to change the devfs rules for the jail you’re using to enable access to that device. There are guides out on the web that explain which device, how to change the devfs rule set the jail uses and how to create a new ruleset for this purpose. If you don’t enable device access for the jail, dhcp won’t work.

ETA: bpf is the device for dhcp.
 
You must log in or register to reply here.
Back
Top