Questions about Vnet Jng jail.conf setup

keafao

keafao

Hi all,

I am trying to know more about vnet vimage. It was suggested a link zfs and vnet when I was researching about jail for a server.

After testing few approaches why if I do use this code on jail.conf
Code: 
exec.prestart += "jng bridge nginx ena0";
exec.poststop += "jng shutdown nginx";

I have the interface on jail

Code: 
ng0_nginx: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=28<VLAN_MTU,JUMBO_MTU>
    ether 0a:00:dd:86:47:ba
    inet6 fe80::1427:e888:767c:dce1%ng0_nginx prefixlen 64 scopeid 0x4
    inet 172.18.0.5 netmask 0xffffff00 broadcast 172.18.0.255
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

On host I did not see the interface ng0_nginx.


But on my testings, if I go to command and
Code: 
# jng bridge nginx ena0
# ifconfig ng0_nginx 172.18.0.5/24

The result is the interface available in both, host and jail.

Host
Code: 
ng0_nginx: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=28<VLAN_MTU,JUMBO_MTU>
    ether 0a:00:dd:86:47:ba
    inet6 fe80::1427:e888:767c:dce1%ng0_nginx prefixlen 64 scopeid 0x4
    inet 172.18.0.5 netmask 0xffffff00 broadcast 172.18.0.255
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Jail
Code: 
ng0_nginx: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=28<VLAN_MTU,JUMBO_MTU>
    ether 0a:00:dd:86:47:ba
    inet 172.18.0.5 netmask 0xffffff00 broadcast 172.18.0.255
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active


I also tried
- virtual.lan script /usr/share/examples/netgraph
- vnet https://github.com/genneko/freebsd-vimage-jails
- ngctl commands

All of them give me same results, showing the ng0_nginx available in jail and host.

Is it wrong running from host? I use Nat/pf to get internet inside jail, is it right/normal way to do this?

Thanks


Edit: typo
 
You have to take into account that the jail communicates with the host via an epair(4). An epair is a pair of two network interfaces that are connected via a virtual network cable.
Normally you take one of the interfaces and place it in the jail and the other one stays on the host. Their names are usually similar, but one is XXXXa and the other one is XXXXb.
After moving one of them into the jail's VNET network stack, you give them both different IP addresses. You could also bridge one of them with the host's network card instead of an IP address and the jail gets its own connection to the LAN:
Code: 
ifconfig epairA 172.18.0.5/24 up
ifconfig epairB vnet MyJail
jexec MyJail ifconfig epairB 172.18.0.6/24 up

And then you can ping your jail from the host:
host # ping 172.18.0.6

Take a look at this thread where I showed an example of how I do the VNET interfaces in a jail: https://forums.freebsd.org/threads/jails-vnet-freebsd-mastery-multiple-interfaces.70356/
I don't use those automation tools (although they might do a great job, I prefer keeping the control over the network interfaces).

If you understand how an epair works, then you can go ahead an use the scripts like jib or jng because that's exactly what they do. But for learning, better do it manually and see how things work.

To understand better how epairs work, imagine how a real-life network works. You have a switch, a bunch of computers connected to it via network cables. The computers send packets and the switch connects them all and passes the packages.
Analogous to a physical network, a bridge(4) interface works like a software switch, an epair(4) works like a virtual network cable and a jail(8) acts as a virtual computer.
The network configurations then work almost identically to physical devices.
 
P.S. Also, I had stability issues with VNET interfaces on jail stop. Sometimes the kernel would panic and the whole host crashes.
I worked around that by using a "prestop" script and removed all network interfaces from a jail before stopping it.
Code: 
ifconfig $epairB -vnet $jail
ifconfig $epairB destroy

With this method everything is stable so far.
 
abishai said:
Strange, I didn't noticed panics. Are you using pf in jails ?
Click to expand...
No, I use VNET interfaces and some jails have an IPFW firewall.
The panics are caused by VNET trying to cleanup things on jail exit. I debugged the cores, they land in the VNET cleanup code.
If the interfaces gets removed from the VNET before exit everything's fine.
 
Hi roccobaroccoSC,

Thanks for your reply and jail/zfs/vnet link, I decided to go with zfs and jail. I was a little afraid of my ec2 would not have enough ram/cpu. From what I read I am at minimum limit, 4gb ram.

I will give a try and see how the zfs/jail goes compare to the older freebsd server without zfs and jail.

About my tests, I found strange getting different results with same commands so I posted.

I am newbie on vnet, I tried jng, virtual.lan, vnet and ngctl. From what I read all use ngctl.

The example you wrote looks similar from what I did with ngctl. I basically did a bridge to my host then hook jails to bridge. Two hooks (lower and upper) to host. Jails I use only one hook.

I could get internet inside jails using nat/pf.

Code: 
  Name: ena0             Type: ether           ID: 00000002   Num hooks: 2
  Local hook      Peer name       Peer type    Peer ID         Peer hook
  ----------      ---------       ---------    -------         ---------
  upper           ena0bridge      bridge       00000009        link1
  lower           ena0bridge      bridge       00000009        link0

    Name: ena0bridge      Type: bridge          ID: 00000009   Num hooks: 7
  Local hook      Peer name       Peer type    Peer ID         Peer hook
  ----------      ---------       ---------    -------         ---------
  link2           ng0_nginx       eiface       0000000e        ether
  link1           ena0            ether        00000002        upper
  link0           ena0            ether        00000002        lower

    Name: ng0_nginx       Type: eiface          ID: 0000000e   Num hooks: 1
  Local hook      Peer name       Peer type    Peer ID         Peer hook
  ----------      ---------       ---------    -------         ---------
  ether           ena0bridge      bridge       00000009        link2


I will give a try with jib and epairs.

About your script link, I remember reading a vnet post from you but was another thread. It was also talking about epairs.

I did a simple start script and put it on /usr/local/rc.d/ to try, it worked nice but I did not cover the stop.

Thank you
 
keafao: Hi, you don't need a custom script in rc.d. If you put your jail's configuration in /etc/jail.conf you can use the jail service, it will manage your jail:
Bash: 
# This will boot all jails at startup:
sysrc jail_enable=YES

service jail start MyJail
service jail restart MyJail
service jail stop MyJail
service jail console MyJail

Regarding RAM: You have to carefully plan what do you want to use your jails for.
- What apps are going to run inside them?
- How much data are you planing to store in the jails?
- How many users are going to use your jails' services?
- Do you need snapshots, rollbacks, etc.?

If it's for your own private use and you're not going to run very heavy applications, ZFS might do just fine even with less RAM.
An app in a jail has virtually the same resource consumption as if it would run outside the jail. The overhead should be quite small (an additional network stack, probably an extra logger or so).

If you plan to use the jails more heavily, make a lot of snapshots, clones etc. then you should probably look into expanding your RAM or choosing UFS (but you won't be able to make clones and snapshots are expensive from what I have read).

ZFS is very versatile and makes a really good first impression, however you should be careful that the more data and snapshots you put into your system, the more resources ZFS will consume. Make sure you don't run into a bottleneck down the road, when your apps are in production.
 
Hi roccobaroccoSC,

The script I put on rc.d had only the netgraph/vnet part. Actually I did not put any vnet code on jail.conf or rc.conf.

Yes I have enable jail and pf on rc.conf.

- What apps are going to run inside them?
3 webapps and dbs

- How much data are you planing to store in the jails?
I did not plan this it is not a big app, also read about quota but did not use it. I only have to resize the ssd once.

- How many users are going to use your jails' services?
You say admin? Just me.

- Do you need snapshots, rollbacks, etc.?
I do not think I will use too often after server setup, pretty much will only update os and pkgs/ports.

About snapshots, I only snapshot once (skeleton) at setup server. I am using hard way jail, in fact I read the hard way jail and adapt it to multiple jails handbook since they are similar.

Hard way jail https://clinta.github.io/freebsd-jails-the-hard-way/

And I noticed when I use zfs send/receive to create the jails seems that I making snapshots... when I use clone I did not see it.
Code: 
zroot/usr/home/js/nginx           1.23M   108G  1.18M  /usr/home/js/nginx
zroot/usr/home/js/nginx@skeleton    54K      -  1.16M  -

Yes I am going to see if it is good. If not it was nice test.

The whole setup (build from source) used 8.04G against the start instance, a 1.45G Colin Percival ami image zfs.

Thanks
 
pyret said:
VNET is still having that problem? I know it was an issue a few years ago when I was testing them and stuck with SmartOS because of the panics. But I thought when VIMAGE was default in GENERIC for 12.0 it was fixed. On Solaris I never remove the network interface before I destroy zones and everything works fine and doesn't panic.
Click to expand...
I can confirm having the issues with 12.0-RELEASE-p3 but I test on a virtual machine first. I have not yet tested on metal.
For me it is a minor issue as long as I have a reliable workaround, like removing the interfaces from the VNET first.
The impact is quite serious though! The whole box goes down, so you gotta be careful.
 
iocage works quite well but it's a bit constraining when it comes to networking options. If you need one interface and intend to bridge it to the host's network - it works easily. Many other network configurations and you suddenly find yourself browsing through iocage's source trying to understand how to work around things. I go manual myself.
 
You must log in or register to reply here.
Back
Top