Solved Home server: networking, zfs dataset, several jails

Hello!

I want to use FreeBSD for my home server, I'm not quite familiar with system administration within FreeBSD but I really enjoyed it on my home desktop, so have couple questions to ask. My goal is to understand how the one should utilize jails feature. I started with raw jails(I read about ezjail and few other options, but I want to understand the basics, so expect to fulfill my needs with the raw jails).

1. I want to have the following services: [jail#1]plex media server, [jail#2]transmission-daemon, [jail#3(?)]nfs+smb server AND I want all of them share the same zfs data set(torrent and plex should share the same dataset 100% otherwise it has no use). Is it possible to share single dataset between several jails? From what I read (https://docs.freebsd.org/en/books/handbook/zfs/) it's possible to "jail" dataset to the jail, but only for one at the time. Does it mean that my understanding/use case for jails is incorrect? Will it be the correct way to utilize jails here the following way: [jail#1]plex media server+transmission-daemon+zfs dataset#1; [jail#2] smb+nfs server+zfs dataset#2 ? i remember reading some article comparing linux containers to bsd jails and they said with containers you can share disk space, but with jails - not, so probably that's how it works.
2. Networking inside jails is a tricky one. I created jail(ip4=inherit) for Plex media server and installed/started Plex media server. I use web interface with host ip BUT I can't reach the plex server itself(it says something like: "no server found" on the web interface). [Meanwhile, if I installed plex server on the host - I would be able to reach the plex server(not only it's web interface)] I thought it might be somehow related to the way I set up networking in the jail(ip4=inherit). I tried another way(ip4.addr="192.168.88.10";interface=re0), but the result is the same. If I used virtual machine for this case I would create bridge and get another IP for the guest, but I thought it will be more handy for me to have the same IP for every service. On the other hand: what is the reason to use jails in this case? :) Probably I should set up "vnet" here? I followed(partially: created jail by myself using the following article: https://rubenerd.com/starting-with-freebsd-jails/) the instructions set for plex set up from here: https://gist.github.com/dreamcat4/f19580cbd31d8f628aca and there are lots of comments in there describing the same problem I have.

Any help is appreciated!
 
Is it possible to share single dataset between several jails?
Use nullfs(5).

Networking inside jails is a tricky one.
It's not that difficult. Simplest is to not use VNET and simply bind the jail to the host's interface and assign it an IP address in your network. Just treat it as a separate host, as if it's just another machine on your network.
 
plexmediaserver won't work is you don't use VNET. A simple smb server will, though.
I use iocage for jails (static ip). It's configured with nullfs like this in /etc/fstab of the host:
Code:
# Device                Mountpoint                              FStype  Options         Dump    Pass
/zroot/share          /zroot/iocage/jails/smb/root/share    nullfs  rw,late         0    0
/zroot/share          /zroot/iocage/jails/plex/root/share    nullfs  rw,late         0       0
In this example, you have two jails, a plex and a smb servers sharing the same datatset I created with zfs create. In each jail, you have to "mkdir" the mountpoint (/share).
 
nullfs is just what I was looking for(but didn't manage to find).
On the vnet, I read tons of articles and forum topics and the more I get into it the more I reckon it's just doesn't fit my needs. Probably, if plex was able to work with ip alias, this would be much easier, but vnet needs too much attention to set it up. Will dig into it more to find the conclusion on the topic.
Thanks for the response!
 
plexmediaserver won't work is you don't use VNET. A simple smb server will, though.

Is this plex-specific? I'm running emby without VNET, attached to a loopback-interface and it works just fine. In fact I still don't use VNET on any jailhost...

I usually create a loopback interface for each VLAN I want to attach jails to, then use PF to handle the external-IP shenanigans for the jails. This way it's easy to swap out jails in the background (e.g. for major upgrades) while keeping the network-facing configuration untouched and you can also easily attach multiple jails (e.g. that serve different parts of a whole application) to the same external IP. That's what I do on public servers with single IPs, e.g. mailservers where every service runs in a dedicated jails, but everything is reachable via the same external IP.

E.g. for jails that I want to connect to VLAN4 (my 'DMZ' which is used for services that are also accessible from the outside world), I create 'lo4' and attach jails to it using the 127.4.0.0/24 subnet. The host is always '127.x.0.1' for each of those subnets and acts as the default gateway.

Depending on how tightly you want to tie down the jails, you could simply use one PF rule to allow outgoing connections from all jails on the lo4 interface:
Code:
nat pass on $dmz_if from 127.4.0.0/24 to any tag EGRESS -> ($dmz_if)
and incoming traffic:
Code:
rdr on $dmz_if inet proto tcp port $webserver_ports from any to $ext_webserver -> $webserver
where $ext_webserver is the network-facing IP of the webserver and $webserver its IP on the loopback interface

One caveat to keep in mind for rdr-rules: they are NOT 'last match wins' like other PF rules, so if you e.g. want to remap specific ports (e.g. incoming port 80 to some weird non-standard port a service on the jail uses for its web interface), that rdr has to come BEFORE the generic rdr for that host.

IMHO a nice bonus of this approach is, that jails and services on the same loopback interface can easily interact with each other without being exposed to the outside world. So you never have to expose a database that is only needed by the local services to the world and can easily serve dozens of web-interfaces (even from built-in webservers of dubious quality and security, some of which still don't support TLS) over a single nginx reverse-proxy that also handles TLS in a single location.
This way when redirecting only the required ports to the specific jails, the attack surface is kept at an absolute minimum.
 
Is this plex-specific? I'm running emby without VNET, attached to a loopback-interface and it works just fine. In fact I still don't use VNET on any jailhost...
I tried suggested approach(using additional reading: http://kbeezie.com/freebsd-jail-single-ip/ and https://jjba.dev/posts/freebsd-jails-loopback-ip/).
Unfortunately it still can't find my server, tho web interface on my server does work (on http://192.168.88.250:32400).
But the idea to have everything bind to servers(host) IP - is what I would like to have. Unfortunately I can't manage to make it working.
Will try to find something in logs and hope that helps.

rc.conf
cloned_interfaces="lo1"
ipv4_addrs_lo1="127.1.0.1/24"
gateway_enable="YES"
pf_enable="YES"

sysctl.conf
security.jail.allow_raw_sockets=1

jails.conf
interface="lo1";
ip4.addr="127.1.0.2";

pf.conf (list of ports I got from https://support.plex.tv/articles/20...ports-do-i-need-to-allow-through-my-firewall/)
ext_if="re0"
jail_if="lo1"

# Public IP address
ip_pub="192.168.88.250"

# Packet normalization
scrub in all

# Skip packet filtering on loopback interfaces
set skip on lo0
set skip on lo1

# Allow outbound connections from within the jails
nat pass on $ext_if from 127.1.0.0/24 to any -> $ip_pub
rdr on $ext_if proto tcp from any to $ip_pub port 32400 -> 127.1.0.2 port 32400
rdr on $ext_if proto tcp from any to $ip_pub port 32469 -> 127.1.0.2 port 32469
rdr on $ext_if proto udp from any to $ip_pub port 1900 -> 127.1.0.2 port 1900
rdr on $ext_if proto udp from any to $ip_pub port 5353 -> 127.1.0.2 port 5353
rdr on $ext_if proto udp from any to $ip_pub port 32410 -> 127.1.0.2 port 32410
rdr on $ext_if proto udp from any to $ip_pub port 32412 -> 127.1.0.2 port 32412
rdr on $ext_if proto udp from any to $ip_pub port 32414 -> 127.1.0.2 port 32414
rdr on $ext_if proto udp from any to $ip_pub port 32413 -> 127.1.0.2 port 32413

ifconfig on HOST
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
ether f0:b4:d2:2c:7d:51
inet 192.168.88.250 netmask 0xffffff00 broadcast 192.168.88.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.1.0.1 netmask 0xffffff00
inet 127.1.0.2 netmask 0xffffffff
inet6 fe80::1%lo1 prefixlen 64 scopeid 0x3
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

ifconfig on JAIL
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
ether f0:b4:d2:2c:7d:51
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
groups: lo
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.1.0.2 netmask 0xffffffff
groups: lo
 
Does the jail have a default route set? (127.1.0.1 in your case) and can reach a DNS server?
Try pinging a public IP and/or hostname from within the jail (e.g. ping 1.1.1.1 and ntp.org) and also try pinging the 'external' IP of the jail (you'd need rules for ICMP traffic in you pf.conf, which is always a good idea for troubleshooting)

It usually is one of those two things (default route or DNS) that causes connection problems. If it still doesn't work, have a look at what passes the loopback and external interface with tcpdump to see how far/in what direction traffic comes through.

On my home server I don't just forward specific ports. I have a separate IP on the host for emby and just forward all traffic to that IP to the jail (and NAT traffic from the jail to its own external IP):
Code:
nat pass on $dmz_if from $emby to any tag EGRESS -> ($dmz_emby)
[...]
rdr on $dmz_if  inet proto tcp  from any    to $dmz_emby -> $emby

And just as a good advice: You specified an RFC1918 IP as "public IP" - avoid such naming to prevent any confusion with a truly public IP. I usually call them <vlan>_<jail> (e.g. dmz_emby). anything "public" or "pub_xyz" is a truly public interface/IP and should raise several red flags when seen somewhere between local rules.
 
This Plex server is definitely the tough one!
First of all, I can ping wan,lan IP's and hostnames from the jail. This works fine.
And still no luck with the Plex server =\

I found some tips here: https://forums.plex.tv/t/freebsd-jail-plex-looking-for-servers/201102/14
I added values of my networks to Preferences.xml file and nothing changed.
Also I modified rc.conf like this to follow another tip:
ipv4_addrs_lo1="192.168.89.1/24"
They said, you need to have not \32 cidr passing to the jail. Something related to broadcasting. My understanding is that both client and server send broadcast messages(to all subnet) to find each other. And probably my jail network setup is not correct for broadcasting. That's my latest guess, still working on it.
I also changes jail ip to be close to my LAN subnet: 192.168.89.1/24
here is current ifconfig and routes from the host:
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
ether f0:b4:d2:2c:7d:51
inet 192.168.88.250 netmask 0xffffff00 broadcast 192.168.88.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 192.168.89.1 netmask 0xffffff00
inet 192.168.89.2 netmask 0xffffff00
inet6 fe80::1%lo1 prefixlen 64 scopeid 0x3
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Routing tables

Internet:
Destination Gateway Flags Netif Expire
default 192.168.88.1 UGS re0
127.0.0.1 link#2 UH lo0
192.168.88.0/24 link#1 U re0
192.168.88.250 link#1 UHS lo0
192.168.89.1 link#3 UH lo1
192.168.89.2 link#3 UH lo1

Internet6:
Destination Gateway Flags Netif Expire
::/96 ::1 UGRS lo0
::1 link#2 UHS lo0
::ffff:0.0.0.0/96 ::1 UGRS lo0
fe80::/10 ::1 UGRS lo0
fe80::%lo0/64 link#2 U lo0
fe80::1%lo0 link#2 UHS lo0
fe80::%lo1/64 link#3 U lo1
fe80::1%lo1 link#3 UHS lo0
ff02::/16 ::1 UGRS lo0

From the above output I can see that lo1 doesn't support broadcasting(assume it's because it was cloned from lo0), that's might be the rootcause, but I can't be sure in anything here, hehe
 
This Plex server is definitely the tough one!
First of all, I can ping wan,lan IP's and hostnames from the jail. This works fine.
And still no luck with the Plex server =\

Just in case you didn't know a well known FreeBSD based NAS ;) has plugins for plex and nextcloud among other things and a simple mangement user interface for jails. Won't say any more since mentioning it is not allowed.
 
Wow. We did it.
The resolution was simply stupid.
On my desktop linux machine I have line in hosts for my home server like this: 192.168.88.250 homeserver
and when I tried to use plex web with this hostname: http://homeserver:32400/web - I got web part of the server working while media server itself wasn't found.
when I used IP instead: http://192.168.88.250:32400/web - both web part and media server part work now.
I found in plex logs this hostname which makes no sense to me, why my locally assigned hostname was passed into the server's code.
I'm glad we make it working!

Thanks everyone!
 
Is this plex-specific? I'm running emby without VNET, attached to a loopback-interface and it works just fine. In fact I still don't use VNET on any jailhost...
After further investigation, it seems that plexmediaserver works without VNET. It's only the DNLA service which doesn't work. I think it's because a jail without VNET does not receive the UPnP broadcoast (SSDP at 239.255.255.250).

Perhaps you will argue you can do this with one or several additional pf rules. But, from my humble & amateur point of view, these rules are already a pain to understand and probably a torment to maintain. However, it's interesting. I read a very good article there: https://blog.grem.de/pages/ayvn.html

In comparison, VNET is very simple to set and it works without any complication. Times where it was experimental, where you had to compile a custom kernel are already far from here.
 
I never cared about UPnP, especially if it involves automatic firewall rules on behalf of some network device/service. This is just fundamentally a very bad idea...
As my emby server resides in the DMZ and the clients are in different VLANs, broadcasts won't work anyways (and forwarding broadcasts over subnet/vlan boundaries is an even worse idea...)

Yes, VNET has come a long way and is a vital addition to jails. I also tested it on a jailhost after it was considered stable (it unfortunately wasn't...) and it sure has its advantages, however - I still find the approach of using loopback interfaces and managing all jail connectivity through the hosts PF more convenient, easier and more flexible. This approach has served me well for several years and 'just works'™so I'll stick to it for now until I truly have an urge to use VNET...
 
I never cared about UPnP, especially if it involves automatic firewall rules on behalf of some network device/service. This is just fundamentally a very bad idea...
As my emby server resides in the DMZ and the clients are in different VLANs, broadcasts won't work anyways (and forwarding broadcasts over subnet/vlan boundaries is an even worse idea...)

Yes, VNET has come a long way and is a vital addition to jails. I also tested it on a jailhost after it was considered stable (it unfortunately wasn't...) and it sure has its advantages, however - I still find the approach of using loopback interfaces and managing all jail connectivity through the hosts PF more convenient, easier and more flexible. This approach has served me well for several years and 'just works'™so I'll stick to it for now until I truly have an urge to use VNET...
You are right: what is working have to be kept as is (until one has some time to test something else).

Concerning UPnP, I agree too but, it only concerns the gateways. I have disabled UPnP on my box since the day I understood what a software can do with it. A very bad idea, that's for sure (and it's often activated by default).
But in the context of a LAN, devices (other than gateways) can communicate with each other and, with some luck, work together. :)
 
Back
Top