How to add dev unhide rules to VNET jails?

M

max21

This may be uncommon to most but this is the way I apply dev rules for my manual build jails since FreeBSD-10.0. Here are the relevant parts:

My portable scripts for jails
Code: 
cd $CELL_1
mount -t devfs devfs /usr/local/SLAMMER/proxy/dev
devfs -m /usr/local/SLAMMER/proxy/dev rule -s 4 applyset
devfs -m /usr/local/SLAMMER/proxy/dev rule apply path tun0 unhide
devfs -m /usr/local/SLAMMER/proxy/dev rule apply path pf unhide
... and this is essentially all I have to do to tear it down at any point by using a script after exiting the shell.
Code: 
service $ CELL_1 sshguard stop   # stop this and others services then type exit.
umount -A -t nullfs           # kill this first just because.
umount -A -t devfs           # now tear it all down!
unsetenv CELL_1               # flush the shell env.
Since I could not find an inkling of a clue on the internet all week long, last night I tried many ways to modify what I have above inside jail.conf but I’m having no luck.

My question is how do I add a rule to below to apply path to unhide a device such as pf or tun inside a VNET jails?

Jail.conf for VNET
Code: 
proxy {
path = /usr/local/SLAMMER/proxy;
mount.devfs;
devfs_ruleset = 4;

add something? add unhide? How in the world do we do it?
Thanks in advance for even pondering this. FYI, I did not miss much while searching. I even read that developers are nearing a fix to VNET/PF so that a single PF can run the entire system host and jails; so it will be the future if not already. As of now, above all VNET is no longer a memory hog and it works out-the-box as if it was on real hardware. Easy to build, add options VIMAGE and options VNET_DEBUG if you want to get wise. So basically, it is approaching light-speed, with little to no complications. It definitely has me trying again since I never got it to work years back.
 
Shut down your jail

See /etc/devfs.rules. Theses rules apply dynamically to any device after the boot.
and add something like

Code: 
[proxy=15]                                                 # jailname followed by the ruleset number as set in /etc/jail.conf
add include $devfsrules_hide_all                           # first we hide all
add include $devfsrules_unhide_basic                   # Unhide basic rule
add include $devfsrules_unhide_login                    # Unhide basic rule
add path 'urandom' unhide                                  # unhide a specific device
add path 'da[0-9]*' unhide                                  # unhide usb storage devices using joker
add path 'da[0-9]*' mode 0660 groupe operator  # change ownership on usb storage device

etc

service devfs onerestart, and restart your jail

I see some people claiming to make work VNET, and I am a little puzzled because following instruction to recompile my kernel I am unable to make a kernel work properly since several years.

- First : if I am not wrong compiling VIMAGE clashes with packet filter. It seems that one should used IPFW, just tell me if I am wrong
- Second : it seems that VIMAGE kernel clashes with the linux compatibility layer, it generates some strange errors at boot

So I gave up... in any case , I don't want to drop packet filter unless someone would be gentle enough to explain clearly how he compiles the kernel with VIMAGE

I am perhaps missing some additional options which don't appear in the very few online documentation we find.
 
Thanks Wozzeck.Live,

For heavy operations sometimes only one out of a thousand searches actually works like it should. Now I realize it could be because it’s still experimental, it don’t meant the setup or hardware we already have, our kernel is all jack-up, or it may be about Linux and we just trying to convert, then forgot it was for Linux.

Anyway, I got lucky and found this link and it WORKS. Its the kind just by looking at it, you KNOW it will work.

http://wb-hk.blogspot.com/2016/03/freebsd-jails-3-vimage.html

I ended up not change a thing but to included VIMAGE + because of that link.
Code: 
#...................................................... for better read, or hiding it
#options       SC_KERNEL_CONS_ATTR=(FG_BLACK|BG_BLACK)
options   SC_KERNEL_CONS_ATTR=(FG_GREEN|BG_BLACK)
#...................................................... it goes under SC_PIXEL_MODE

# At the bottom I complied all of this and it WORKS!

options IPSTEALTH
device   pf
device   pflog
device   pfsync
options   ALTQ
options   ALTQ_CBQ
options   ALTQ_RED
options   ALTQ_RIO
options   ALTQ_HFSC
options   ALTQ_PRIQ
options   ALTQ_NOPCC

options       VIMAGE
options         VNET_DEBUG

device crypto
device cryptodev
device aesni
options       GEOM_ELI
options       NULLFS

options HZ=2000
options DEVICE_POLLING
Thanks to VNET_DEBUG I dug these out of the /var/log/all.log. I check and pf is running on host but not in jail. The jail is running but can not ping anything. That is why I’m here .. I’m trying to add more device into the jail to see what happen, from there its ON!
Code: 
#CURVNET_SET() recursion in pf_load_vnet() line 3714, prev in vnet_register_sysinit()
#CURVNET_SET() recursion in pf_load_vnet() line 3714, prev in vnet_alloc()
#CURVNET_SET() recursion in pf_check6_in() line 3601, prev in ether_input_internal()
I read curvnet are macros. Also I am doing all of this in VirtualBox with FreeBSD as host. Maybe PF can run the entire system. Can it already? This works too for my VNET-IPFW kernel.

https://www.vultr.com/docs/enhancing-security-for-freebsd-using-ipfw-and-sshguard

Now I'm going to try the method you suggested. My script idea might not work with VIMAGE. However, I got one more trick to try if all else fails.

In a minute.
 
Wozzeck.Live said:
Shut down your jail

See /etc/devfs.rules. Theses rules apply dynamically to any device after the boot.
and add something like

Code: 
[proxy=15]                                                 # jailname followed by the ruleset number as set in /etc/jail.conf
add include $devfsrules_hide_all                           # first we hide all
add include $devfsrules_unhide_basic                   # Unhide basic rule
add include $devfsrules_unhide_login                    # Unhide basic rule
add path 'urandom' unhide                                  # unhide a specific device
add path 'da[0-9]*' unhide                                  # unhide usb storage devices using joker
add path 'da[0-9]*' mode 0660 groupe operator  # change ownership on usb storage device

etc

service devfs onerestart, and restart your jail

I see some people claiming to make work VNET, and I am a little puzzled because following instruction to recompile my kernel I am unable to make a kernel work properly since several years.

It will compile with 11x these days but there are still limitations; ITS JUST THERE, but it works for the host only. Isn’t it ironic that after all these years those who claim that they use it never once saying where it works or how they got it to work for the host and jail. I google deep and read them ALL just like you done.

- First : if I am not wrong compiling VIMAGE clashes with packet filter. It seems that one should used IPFW, just tell me if I am wrong
If you already had a clean install of FreeBSD-11.0+ or SVN-11x and used that link, you already know it don’t crash anymore more at comply time and that PF will at least run great on host only.

- Second : it seems that VIMAGE kernel clashes with the linux compatibility layer, it generates some strange errors at boot
I bet you know why already. My guess is those errors are because it can’t see FreeBSD linux compatibility layer devices.

So I gave up... in any case , I don't want to drop packet filter unless someone would be gentle enough to explain clearly how he compiles the kernel with VIMAGE

Me too. There is nothing is worth giving up packet-filter for. I guest we all just have to wait. I got faith that the developer(s) will complete soon, if not already. In the mean time only the know knows and he have not spoken a word yet that I found.

I am perhaps missing some additional options which don't appear in the very few online documentation we find.

Well it’s time that they document this: After everything I tried, VIMAGE will not accept any devices other then add include $devfsrules_unhide_basic.

To prove that to myself, I commented out everything and change hide to unhide -> add include $devfsrules_unhide_all and all it did was to produce the same unhide_basic devices. Yep, my hunch was correct. Thanks a ton for that can-opener. Well it time to get back on it. I’m happy that you are with it. When you Win, let me know, I’ll do the same, right here.
Click to expand...

Fair enough?

It has been a real trip trying to do one simple thing with vnet-jails. It just not in there yet I guest. Like they say max21 did not build Rome in a day. Now I’m reading about other options such as netgrapth and such. Will it ever end? I had to open up the jail and allow all devices so that I can at least explore the process. From reading more about VIMAGE usage, I think my plain jails suits me just fine, but now hook. I see that PF is running fine on the host and my old jails are working. I don’t think I will have any problem doing battle with a single vnet-jail. It don’t seem to affect the rest of the system in any kind of way. All I want to be sure of is that sshguard got IPFW back, and to monitor memory usages.

Thank Again Wozzeck.Live


BTW: I complied: kernel.original – kernel.pf – kernel.pf-altq - kernel.ipfw …just-in-case
 
Last edited:
I’m throwing in the towel once again. If I had the time, I could beat it but then again, I might not. I can’t seem to complete anything since last winter. It going to be a while for VIMAGE. The problem I read is funding, and the handbook never said much about it since invented; and to make it worse, pf will only run on the host, so those who say otherwise only think it is, and all other firewalls can cause the system to crash. Still, I learn so much more about how FreeBSD works since trying so hard over the past few weeks. I’m going to try bhyve. The word is once its completed it will be the kind of virtualization that is going to knock your socks off. I like talk like that by experienced people and it was a Linux guru who said it.

http://www.a1poweruser.com/35.00-Jails_guide_article.php

Just wanted to post this link. Info at bottom of page about VIMAGE.
 
You must log in or register to reply here.
Back
Top