PF Plex + PF

Hello,

Over the past few months I have posted several times regarding an issue I experienced when upgrading the version of Plex media server. I believe the issue is related to my firewall ruleset so was hoping to get some help. I will do my best to explain what I have experienced:

Several months ago I had a webserver configured with a few different jails. Apache on one jail, an irc server on another, and finally a jail dedicated for plex. My ruleset (below) seemed to be just fine for what I needed. After upgrading plex it would not longer respond. I posted on these forums and received a lot of different recommendations and similar on the plex forums. To make a long story short I abandoned my setup (10.1-Release) and switched to Stable so that I could use bhyve(8) since I have an AMD CPU and required the last version. I got everything set up with bhyve(8) now but am having the same exact issue (seems to be at least) where plex works only when I turn off my firewall. Once enabled, plex does not respond.

Ultimately I would prefer to revert back to 10.1-RELEASE and use sysutils/ezjail instead of use bhyve(8). I plan to spin up a Release install and test this out again but am fairly confident that it will work once the firewall is turned off so I do not think I NEED to use bhyve(8).

A few things to note:
  • I am not sure if this is related or not but I just noticed that my /var/unbound/unbound.conf cannot be read. The permissions are showing read by everyone and write for root, so not sure why that is an issue.
  • My firewall ruleset is not leveraging the bridge/tap interfaces required for bhyve(8) as shown in the handbook (https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html). I would imagine I should be using these if I stick with bhyve(8) which is the primary reason I would rather keep my system set up the way it was previously and stick with sysutils/ezjail.
  • My ruleset is pretty old, and probably has a lot of room for improvement. Any input would be greatly appreciated.
  • I prefer to use jails over bhyve(8) but just want plex to work regardless, so whichever is preferred please let me know. I think I just need help tweaking my ruleset but would expect that solution will differ depending on which environment plex is running inside.
  • This is the documentation for firewall port forwarding from plex: https://support.plex.tv/hc/en-us/ar...ports-do-I-need-to-allow-through-my-firewall-
Code:
brad@mercury:/home/brad$ cat /etc/pf.conf
set skip on lo0
interface="re1"
apacheJail="192.168.0.102"
ircJail="192.168.0.103"
virtualPlex="192.168.0.104"
scrub in all
rdr pass on $interface proto tcp from any to $interface port 80-> $apacheJail
rdr pass on $interface proto tcp from any to $interface port 6667-> $ircJail
rdr pass on $interface proto tcp from any to $interface port 32400-> $virtualPlex
rdr pass on $interface proto udp from any to $interface port 1900-> $virtualPlex
rdr pass on $interface proto tcp from any to $interface port 3005-> $virtualPlex
rdr pass on $interface proto udp from any to $interface port 5353-> $virtualPlex
rdr pass on $interface proto tcp from any to $interface port 8324-> $virtualPlex
rdr pass on $interface proto udp from any to $interface port 32410-> $virtualPlex
rdr pass on $interface proto udp from any to $interface port 32412-> $virtualPlex
rdr pass on $interface proto udp from any to $interface port 32413-> $virtualPlex
rdr pass on $interface proto udp from any to $interface port 32414-> $virtualPlex
rdr pass on $interface proto tcp from any to $interface port 32469-> $virtualPlex
block in on $interface
pass in on $interface proto tcp from any to any port 2662
passout on $interface proto {tcp,udp,icmp} all
Thanks in advance!
 
Last edited:
In regards to my unbound.conf it looks like it is complaining about two unknown keywords: 0.0.0.0 and "interface". I thought I had set that file up based on the advice given in these forums but perhaps I made a mistake. Here is my file below, any help on how to update that (I honestly am not sure I even need to use that functionality) would be great.
Code:
brad@mercury:/home/brad$ cat /var/unbound/unbound.conf
# Generated by local-unbound-setup
server:
  username: unbound
  directory: /var/unbound
  chroot: /var/unbound
  pidfile: /var/run/local_unbound.pid
  auto-trust-anchor-file: /var/unbound/root.key
interface 0.0.0.0
interface-automatic: yes
access-control: 192.168.0.0/16 allow
include: /var/unbound/forward.conf
include: /var/unbound/lan-zones.conf
include: /var/unbound/conf.d/*.conf
 
I am also getting a lot of these messages when using an ssh session. I have been using the same firewall ruleset for several years with no issues until switching to 10.1-STABLE. Not sure if I need to do an entire overhaul or if it just a few lines of deprecated functionality.

Any help is appreciated. Thanks

Code:
channel 2: open failed: connect failed: Connection refused

==> /var/log/messages <==
Feb 16 14:24:35 mercury sshd[49340]: error: connect_to 192.168.0.103 port 6667: failed.

I am using the ssh -L flag to forward ports for IRC and have allowed that through my sshd config file. Not sure if I need to focus more on that or my firewall.
 
Last edited by a moderator:
Hi z662, I suggest you insert some "log" commands into your pf ruleset like
Code:
block in log on $interface
You should have pflog enabled though, as shown in the FreeBSD Handbook at https://www.freebsd.org/doc/en/books/handbook/firewalls-pf.html. After that you can watch your packets hitting your block rule with tcpdump(1):
Code:
tcpdump -i pflog0 -n
... to see which packets or connection are blocked. Maybe this gives you some insight what is happening. I do not use bhyve(8) so no specific input on that. And focus on one problem at a time. ;-)
If no blocked packets show up try switching tcpdump(1) to your LAN interface and a specific port:
Code:
# show all traffic on interface re1 and to/from host 192.168.0.104 and port 1900
tcpdump -i re1 -n host 192.168.0.104 and port 1900
 
Is this line intentional? Or is this a typo in your /etc/pf.conf?
Code:
set skip on lo0interface="re1"
Are you missing a linebreak between "lo0" and "interface"?
 
I think that was a formatting issue when I copied/pasted. Thought I had resolved all of those but I must have missed that. Sorry about that I will edit/update.

Thanks for the log advice, I will have to do that. Aside from the issue, does that ruleset seem pretty good then? I was happy with it beforehand but knew it was pretty simple and basic. I basically just want to block everything except the services I am redirecting or allowing. I am not sure why I am having this issue now...
 
I ended up dropping Plex and focusing on a more lightweight DLNA server, net/minidlna. This service runs on port 8200 so I changed my rdr pass rules to:
Code:
rdr pass on $interface proto tcp from any to $interface port 8200 -> $minidlna
$minidlna is assigned to 192.168.0.104 and is its own jail.

Similar to Plex, I can only reach the server if I disable my firewall via pfctl -d. Does anyone know why?

When looking at my logged blocked traffic I get this:

Code:
[NOPARSE]
brad@mercury:/home/brad$ sudo tcpdump -i pflog0 
tcpdump: WARNING: pflog0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 65535 bytes
19:15:40.183318 IP6 fe80::c24a:ff:fe04:4b56 > ff02::1: ICMP6, router advertisement, length 56
19:15:41.296932 IP EARTH.milkyway.53304 > 192.168.0.104.8200: Flags [S], seq 157265891, win 14600, options [mss 1460,sackOK,TS val 2155789723 ecr 0,nop,wscale 7], length 0
19:15:41.547409 IP EARTH.milkyway.53305 > 192.168.0.104.8200: Flags [S], seq 3683369739, win 14600, options [mss 1460,sackOK,TS val 2155789973 ecr 0,nop,wscale 7], length 0
19:15:42.296801 IP EARTH.milkyway.53304 > 192.168.0.104.8200: Flags [S], seq 157265891, win 14600, options [mss 1460,sackOK,TS val 2155790723 ecr 0,nop,wscale 7], length 0
19:15:42.546756 IP EARTH.milkyway.53305 > 192.168.0.104.8200: Flags [S], seq 3683369739, win 14600, options [mss 1460,sackOK,TS val 2155790973 ecr 0,nop,wscale 7], length 0
19:15:44.296807 IP EARTH.milkyway.53304 > 192.168.0.104.8200: Flags [S], seq 157265891, win 14600, options [mss 1460,sackOK,TS val 2155792723 ecr 0,nop,wscale 7], length 0
19:15:44.546724 IP EARTH.milkyway.53305 > 192.168.0.104.8200: Flags [S], seq 3683369739, win 14600, options [mss 1460,sackOK,TS val 2155792973 ecr 0,nop,wscale 7], length 0
19:15:44.991317 IP6 fe80::c24a:ff:fe04:4b56 > ff02::1: ICMP6, router advertisement, length 56
19:15:48.296758 IP EARTH.milkyway.53304 > 192.168.0.104.8200: Flags [S], seq 157265891, win 14600, options [mss 1460,sackOK,TS val 2155796723 ecr 0,nop,wscale 7], length 0
19:15:48.546752 IP EARTH.milkyway.53305 > 192.168.0.104.8200: Flags [S], seq 3683369739, win 14600, options [mss 1460,sackOK,TS val 2155796973 ecr 0,nop,wscale 7], length 0
[/NOPARSE]
 
I also see a lot of broadcast traffic of some sort coming from my router (venus.milkyway). I am not sure if that is related or not but I see it is from port 1900 which seems to be required for the dlna jail. The following was run in the jail:
Code:
brad@mercury:/home/brad$ sudo jexec 2 sh
# sockstat -4
USER  COMMAND  PID  FD PROTO  LOCAL ADDRESS  FOREIGN ADDRESS  
root  sendmail  1295  3  tcp4  192.168.0.104:25  *:*
dlna  minidlnad  1287  5  udp4  192.168.0.104:1900  *:*
dlna  minidlnad  1287  6  tcp4  192.168.0.104:8200  *:*
dlna  minidlnad  1287  8  udp4  192.168.0.104:42697  *:*
root  syslogd  1238  6  udp4  192.168.0.104:514  *:*

While this is from the parent fw:
Code:
[NOPARSE]
brad@mercury:/home/brad$ sudo tcpdump -i pflog0
tcpdump: WARNING: pflog0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 65535 bytes
20:05:02.752647 IP EARTH.milkyway.53520 > 192.168.0.104.8200: Flags [S], seq 1881895052, win 14600, options [mss 1460,sackOK,TS val 2158751175 ecr 0,nop,wscale 7], length 0
20:05:03.003057 IP EARTH.milkyway.53521 > 192.168.0.104.8200: Flags [S], seq 179912690, win 14600, options [mss 1460,sackOK,TS val 2158751425 ecr 0,nop,wscale 7], length 0
20:05:03.631990 IP6 fe80::c24a:ff:fe04:4b56 > ff02::1: ICMP6, router advertisement, length 56
20:05:03.752139 IP EARTH.milkyway.53520 > 192.168.0.104.8200: Flags [S], seq 1881895052, win 14600, options [mss 1460,sackOK,TS val 2158752175 ecr 0,nop,wscale 7], length 0
20:05:04.002297 IP EARTH.milkyway.53521 > 192.168.0.104.8200: Flags [S], seq 179912690, win 14600, options [mss 1460,sackOK,TS val 2158752425 ecr 0,nop,wscale 7], length 0
20:05:05.628166 IP EARTH.milkyway.53518 > 192.168.0.104.8200: Flags [S], seq 1064540993, win 14600, options [mss 1460,sackOK,TS val 2158754051 ecr 0,nop,wscale 7], length 0
20:05:05.752172 IP EARTH.milkyway.53520 > 192.168.0.104.8200: Flags [S], seq 1881895052, win 14600, options [mss 1460,sackOK,TS val 2158754175 ecr 0,nop,wscale 7], length 0
20:05:05.879151 IP EARTH.milkyway.53519 > 192.168.0.104.8200: Flags [S], seq 1644303047, win 14600, options [mss 1460,sackOK,TS val 2158754302 ecr 0,nop,wscale 7], length 0
20:05:06.002200 IP EARTH.milkyway.53521 > 192.168.0.104.8200: Flags [S], seq 179912690, win 14600, options [mss 1460,sackOK,TS val 2158754425 ecr 0,nop,wscale 7], length 0
20:05:08.042022 IP6 fe80::c24a:ff:fe04:4b56 > ff02::1: ICMP6, router advertisement, length 56
20:05:09.752187 IP EARTH.milkyway.53520 > 192.168.0.104.8200: Flags [S], seq 1881895052, win 14600, options [mss 1460,sackOK,TS val 2158758175 ecr 0,nop,wscale 7], length 0
20:05:10.002166 IP EARTH.milkyway.53521 > 192.168.0.104.8200: Flags [S], seq 179912690, win 14600, options [mss 1460,sackOK,TS val 2158758425 ecr 0,nop,wscale 7], length 0
20:05:15.811002 IP6 fe80::c24a:ff:fe04:4b56 > ff02::1: ICMP6, router advertisement, length 56
20:05:17.752147 IP EARTH.milkyway.53520 > 192.168.0.104.8200: Flags [S], seq 1881895052, win 14600, options [mss 1460,sackOK,TS val 2158766175 ecr 0,nop,wscale 7], length 0
20:05:18.002187 IP EARTH.milkyway.53521 > 192.168.0.104.8200: Flags [S], seq 179912690, win 14600, options [mss 1460,sackOK,TS val 2158766425 ecr 0,nop,wscale 7], length 0
20:05:19.105028 IP6 fe80::c24a:ff:fe04:4b56 > ff02::1: ICMP6, router advertisement, length 56
20:05:19.364818 IP NP-1GH359100284.milkyway.57632 > 192.168.0.104.8200: Flags [S], seq 1977813699, win 14600, options [mss 1460,sackOK,TS val 9008 ecr 0,nop,wscale 4], length 0
20:05:21.628079 IP EARTH.milkyway.53518 > 192.168.0.104.8200: Flags [S], seq 1064540993, win 14600, options [mss 1460,sackOK,TS val 2158770051 ecr 0,nop,wscale 7], length 0
20:05:21.805005 IP venus.milkyway.27565 > 239.255.255.250.1900: UDP, length 380
20:05:21.805013 IP venus.milkyway.27565 > 239.255.255.250.1900: UDP, length 380
20:05:21.805019 IP venus.milkyway.27565 > 239.255.255.250.1900: UDP, length 452
20:05:21.805022 IP venus.milkyway.27565 > 239.255.255.250.1900: UDP, length 452
20:05:21.805028 IP venus.milkyway.27565 > 239.255.255.250.1900: UDP, length 388
[/NOPARSE]
 
I ended up dropping Plex and focusing on a more lightweight DLNA server minidlna. This service runs on port 8200 so I changed my rdr pass rules to:
Code:
rdr pass on $interface proto tcp from any to $interface port 8200 -> $minidlna
$minidlna is assigned to 192.168.0.104 and is its own jail.

"from any to $interface" intention conflicts with the packets smashing against your firewall, see "EARTH.milkyway.53304 > 192.168.0.104.8200". $interface is the HOST interface/IP.
So to me it looks like your EARTH machine is trying to talk to $minidlna directly and gets blocked - as defined by the ruleset. It should talk to the ezjail-HOST IP-address which gets redirected to your $minidlna host.
You have to decide if you want to
  1. be able to talk to your $minidlna machine directly, then add a "pass in" rule for that
  2. redirect your traffic to your ezjail-$minidlna but then you have to talk/send packets to your ezjail-host machine
As per DLNA it may be possible that $minidlna is advertising its real IP address so your earth host tries to contact this advertised IP address directly. So maybe only option 1. of the above is possible. I am not sure, you have to try that out.
Similar to Plex, I can only reach the server if I disable my firewall via pfctl -d. Does anyone know why?
When looking at my logged blocked traffic I get this:
Code:
[NOPARSE]
brad@mercury:/home/brad$ sudo tcpdump -i pflog0
tcpdump: WARNING: pflog0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 65535 bytes
19:15:40.183318 IP6 fe80::c24a:ff:fe04:4b56 > ff02::1: ICMP6, router advertisement, length 56
19:15:41.296932 IP EARTH.milkyway.53304 > 192.168.0.104.8200: Flags [S], seq 157265891, win 14600, options [mss 1460,sackOK,TS val 2155789723 ecr 0,nop,wscale 7], length 0
[...snip...]
[/NOPARSE]

Does this make sense to you?
 
It does, this is where I get confused due to the way this seems to have changed recently. Let me try to explain:

Several months ago I ran 10.1-Release and had basically the same setup (mercury was the host @ 192.168.0.101, then I had 3 jails, one for apache (192.168.0.102), irc (192.168.0.103), and plex (192.168.0.104). Long story short I was forced to upgrade plex to the latest version and unfortunately their latest version is doing some "weird" network stuff requiring a lot of ports to be open and they seem to be running a lot of services and maybe trying to do too much imo, I liked it when it wasnt bloated and more stable....Anyway, after I upgraded I couldn't get anything to connect to the plex services/jail. Up until this point I would always have to talk to a jail via mercury's host IP and not the jail directly.

THEN

After several posts and recommendations from plex, it seemed like they weren't sure how to figure out the jail portion (I am convinced it is my broken fw rules) after upgrading plex so I had to resort to using bhyve and because I have an AMD cpu I had to switch to the Stable branch. As soon as I did that I noticed that now all of my jails could ONLY be contacted through the jail IP and not the host. However in the case of plex I had to completely disable the fw (same with minidlna). Ultimately I would be very grateful if someone could help me square away my ruleset so that I could secure plex a bit and go back to it as it looks like minidlna is not very stable when streaming from my roku.

All I want pf to do is block all traffic except for three things, ssh, apache, and plex/dlna content. I do not care if I need to redirect traffic or connect directly to that jail, makes no difference to me as long as it is secure, stable, and works ;) Can you or someone help me write these rules in a better way so that I can get this setup working? I am most likely going to format and switch back to 10.1 Release for the time being as playing around in Stable is most likely not a good idea...however, I am wondering if I can get the plex jail to work in 10.1 Release again. My guess is that it can certainly be done but my fw rules need to be updated.

I am not sure why 10.1 Release -> Stable would cause the change in behaviour for the traffic redirect stuff but it tells me that my old fw rules were most likely incorrect.
 
Can you add log statements to all your rules? With that, you can use the extended syntax shown in pflog(4) with tcpdump -n -e -ttt -i pflog0. That plus a pfctl -vsr would be helpful. Yes, this will be a lot of info but this will get all the info right up front and be helpful when it comes to using troubleshooting time wisely. The things to look for is exact what rules packets are hitting in the tcpdump output and the extended output shows what rule it was on along with block or pass. The pfctl -vsr will show counters for all the rules. If some rules you expect to work have zeros then something isn't working as intended. The problem shouldn't be too difficult to find doing this.
 
Absolutely. I assume I should capture the traffic while trying to access plex/dlna? If so will it matter if I am using a jail vs bhyve? I would prefer to drop bhyve and revert back to the RELEASE branch and stick with ezjail, but I would have to format...so hoping for now the difference is moot. Also, just as an fyi I may have to split this up over several posts if there is that much noise, I think I hit the character limit on the forums a few times before while posting large log files. Let me know, especially about the bhyve vs ezjail and I will get that output uploaded shortly after. If not tonight definitely by tomorrow. This issue is driving me nuts! :)

Thanks again
 
Yes, it would be bunch of output so try to limit excess traffic by capturing just when you try to connect and it gets blocked. Use something like "not port 22" on the end of the tcpdump to filter out SSH traffic as that won't be helpful. The jail method should be alright. Toss your output on a site like pastebin.com and just post the links.
 
I believe this is everything. My roku (2nd link) has an IP of 192.168.0.112. I am not entirely sure how it finds a dlna service, not sure if it relies on the router at all for broadcasts; was surprised at the lack of traffic from it but guessing thats why the router (192.168.0.1) was using port 1900 to talk to the server.

Anyway...hope this helps - thanks in advance!

These are my current set of rules:

Code:
set skip on lo0
interface="re1"
apacheJail="192.168.0.102"
ircJail="192.168.0.103"
virtualPlex="192.168.0.104"
scrub in all
rdr pass on $interface proto tcp from any to $interface port 80 -> $apacheJail
rdr pass on $interface proto tcp from any to $interface port 6667 -> $ircJail

rdr pass log on $interface proto tcp from any to $interface port 8200 -> $virtualPlex
rdr pass log on $interface proto udp from any to $interface port 1900 -> $virtualPlex
rdr pass log on $interface proto udp from any to $interface port 42697 -> $virtualPlex

#rdr pass on $interface proto udp from any to $interface port 33358 -> $virtualPlex
#rdr pass on $interface proto udp from any to $interface port 14068 -> $virtualPlex
#rdr pass on $interface proto tcp from any to $interface port 3005 -> $virtualPlex
#rdr pass on $interface proto udp from any to $interface port 5353 -> $virtualPlex
#rdr pass on $interface proto tcp from any to $interface port 8324 -> $virtualPlex
#rdr pass on $interface proto udp from any to $interface port 32410 -> $virtualPlex
#rdr pass on $interface proto udp from any to $interface port 32412 -> $virtualPlex
#rdr pass on $interface proto udp from any to $interface port 32413 -> $virtualPlex
#rdr pass on $interface proto udp from any to $interface port 32414 -> $virtualPlex
#rdr pass on $interface proto udp from any to $interface port 57783 -> $virtualPlex
#rdr pass on $interface proto tcp from any to $interface port 57783 -> $virtualPlex

block in log on $interface
pass in on $interface proto tcp from any to any port 2662
pass out on $interface proto {tcp,udp,icmp} all
I did not use my bhyve plex instance for the firewall output, I am hoping that whatever the solution is for minidlna can be applied in the same way for the plex rules (will revert back to Release and put plex in a jail). Minidlna says in the config file that it listens on 8200 and that is what the web interface connects to (not sure about the roku so going to post both). I added logging to the rdr statements for the 192.168.0.104 interface. The active/non-commented lines are the ones that I see bound (to minidlna) via sockstat, the commented lines are ones that eventually I will need to work for plex (in case someone was curious).

Link for pc/Earth connecting to the web interface: http://pastebin.com/ZBqa6Wnt
Link for roku connecting to the dlna jail: http://pastebin.com/PgSPZqNj
 
For this packet:
Code:
00:00:02.434111 rule 0..16777216/0(match): block in on re1: 192.168.0.100.43020 > 192.168.0.104.8200

Based off everything, the "to $interface" in your NAT rule would translate to "to 192.168.0.101" assuming 192.168.0.101 is the main IP on that interface. So this line in your pf.conf:
Code:
rdr pass log on $interface proto tcp from any to $interface port 8200 -> $virtualPlex

Translates to this in your pfctl -vsn (show NAT rules), which I forgot to ask for. The counters on this rule is probably 0 right now if you show the NAT rules.
Code:
rdr pass log on re1 proto tcp from any to 192.168.0.101 port = 8200 -> 192.168.0.104

It looks like clients on your network just connect directly to the jail? Why not drop the "rdr" all together and go with this:
Code:
pass in on $interface proto tcp from any to any port 8200
# or
pass in on $interface proto tcp from any to $virtualPlex port 8200
# or keeping the rdr and allowing both direct connects to jail and rdr from host IP
rdr on $interface proto tcp from any to $interface port 8200 -> $virtualPlex
pass in on $interface proto tcp from any to any port 8200

Code:
00:00:00.000000 rule 0..16777216/0(match): block in on re1: 192.168.0.106.50726 > 239.255.255.250.1900
Same as above since "to 192.168.0.101" doesn't match 239.255.255.250.

Lastly, for the ICMPv6 blocks, here's some of my pf.conf. I believe these are exactly what pfSense uses as I copied a few things from them when I set up my pf.conf on my router for full IPv6 support.
Code:
# ICMPv6
icmp6_types="{ echoreq, unreach, toobig, neighbrsol, neighbradv }"
icmp6_types_in="{ echoreq, routersol, routeradv, neighbrsol, neighbradv }"
icmp6_types_out="{ echorep, routersol, routeradv, neighbrsol, neighbradv }"
pass inet6 proto ipv6-icmp all icmp6-type $icmp6_types
pass in inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type $icmp6_types_in
pass in inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type $icmp6_types_in
pass in inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type $icmp6_types_in
pass out inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type $icmp6_types_out
pass out inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type $icmp6_types_out
 
Thanks junovitch for the consistent help and replies.

I removed the redirect lines for dlna but still have the same issues. I have a few questions regarding this:

- Why would Apache and IRC work no problem with my current ruleset but anything I try to use as far as a media server fails?
- Why is it that when I had my jails setup in 10.1-Release I would hit the host IP (hence why I had those redirect rules) but now in Stable I have to hit them via their own IP? (IDC, just curious)
- Do I need to worry about ICMPv6? I don't use it for anything as far as I know, and block ping requests so unless there is a reason I should add those rules, I am fine dropping those packets. Unless maybe the root of the issue is that these media servers NEED ICMP for certain things?

As you predicted I had all 0's in the NAT states.
 
- Why would Apache and IRC work no problem with my current ruleset but anything I try to use as far as a media server fails?
I'm not 100% sure without really looking at what's going between systems. If I were to guess, I could see the way DLNA multicasts out its address and that address different from the address being connected to being an issue. For direct connects I would think it would work. Since DLNA isn't really a protocol designed for Internet use things might be odd when wierd networking things are being done.

- Why is it that when I had my jails setup in 10.1-Release I would hit the host IP (hence why I had those redirect rules) but now in Stable I have to hit them via their own IP? (IDC, just curious)
That seems odd. Are you sure it was the exact same rules?

- Do I need to worry about ICMPv6? I don't use it for anything as far as I know, and block ping requests so unless there is a reason I should add those rules, I am fine dropping those packets. Unless maybe the root of the issue is that these media servers NEED ICMP for certain things?
I was just pointing it out since I seen the blocked traffic. If you don't use IPv6 then don't worry about it.

Ultimately I'm just looking at what's getting blocked. Log everything, check what's dropped on pflog0, adjust your rules, check again, and repeat. Alternately, allow everything and log initially and go backwards by adding rules to explicitly cover everything you want and change it to a block afterwards. After some methodical trial an error you'll find the set of rules that covers everything.
 
I am positive the rule sets were the same. I was very surprised to see that the behavior changed. That ultimately resulted in me having to adjust my internet router for those services too (instead of allowing them for mercury, I had to allow them for direct jail access). Again, not a big deal, just very surprising since the only thing that changed was the change from Release to Stable. That being said there was an update to Pfsense too, but I thought I had done that after updating the firewall rules.

Would I be able to allow all traffic for the jail IP or specify a large port range and try to trim from there or would that not really work since the rules are based on forwarding specific ports? Not sure if that makes sense, but for instance I don't think I would be able to have port 80 allowed on multiple IPs within the same host or it wouldn't know which to pass to... so just wondering if I could try a large port range for both TCP and UDP and then keep trying to narrow down from there?

I am leaning towards formatting and reverting back to Release as I was before I started messing with these later versions of Plex. Then install in a fresh jail and use the pflog output to find which ports are causing the issues. Like you said before maybe the RDR rules were the root of the problem. I am not sure any more, and a bit out of my comfort zone when it comes to reading this traffic and finding what should and shouldn't be open.
 
Take a look at pf.conf(5). There's tons of good info.

For redirecting with port ranges:
Code:
     rdr   The packet is redirected to another destination and possibly a dif‐
           ferent port.  rdr rules can optionally specify port ranges instead
           of single ports.  rdr ... port 2000:2999 -> ... port 4000 redirects
           ports 2000 to 2999 (inclusive) to port 4000.  rdr ... port
           2000:2999 -> ... port 4000:* redirects port 2000 to 4000, 2001 to
           4001, ..., 2999 to 4999.

Instead of "to re0", consider using parenthesis so PF can follow IP address changes or modifers like below to allow the subnet. So "to (re0:network)" to allow the whole subnet or "to (re0:0)" to allow traffic only to the host and follow host IP changes.
Code:
           Interface names and interface group names can have modifiers
           appended:

           :network      Translates to the network(s) attached to the inter‐
                         face.
           :broadcast    Translates to the interface's broadcast address(es).
           :peer         Translates to the point-to-point interface's peer
                         address(es).
           :0            Do not include interface aliases.

           Host names may also have the :0 option appended to restrict the
           name resolution to the first of each v4 and v6 address found.

           Host name resolution and interface to address translation are done
           at ruleset load-time.  When the address of an interface (or host
           name) changes (under DHCP or PPP, for instance), the ruleset must
           be reloaded for the change to be reflected in the kernel.  Sur‐
           rounding the interface name (and optional modifiers) in parentheses
           changes this behaviour.  When the interface name is surrounded by
           parentheses, the rule is automatically updated whenever the inter‐
           face changes its address.  The ruleset does not need to be
           reloaded.  This is especially useful with nat.

For port ranges in general.
Code:
           Ports and ranges of ports are specified by using these operators:

                 =       (equal)
                 !=      (unequal)
                 <       (less than)
                 ≤       (less than or equal)
                 >       (greater than)
                 ≥       (greater than or equal)
                 :       (range including boundaries)
                 ><      (range excluding boundaries)
                 <>      (except range)

           ‘><’, ‘<>’ and ‘:’ are binary operators (they take two arguments).
           For instance:

           port 2000:2004
                       means ‘all ports ≥ 2000 and ≤ 2004’, hence ports 2000,
                       2001, 2002, 2003 and 2004.
 
Alright I will do some reading and playing around with this and eventually post back some more information on the subject. Work is busy so may be a few days/weeks but I plan to resolve this one way or another and will share my progress. Thanks
 
Figured it out...although I have one question regarding which IPs are tied to services. Below is the ruleset I needed to have. It was basically removing the RDR but also having it open for the host IP instead of the jail/bhyve host. I have also posted the results of sockstat. This is where my question comes into play: I have http and irc running in jails, while plex is running via bhyve. Why do the jail services require a RDR rule in pf while plex needs to have the host IP allowed? Taking that a step further does that mean that the plex ports are open for the jails too or JUST the host IP?

Code:
set skip on lo0
interface="re1"
apacheJail="192.168.0.102"
ircJail="192.168.0.103"
virtualPlex="192.168.0.104"
scrub in all

rdr pass on $interface proto tcp from any to $interface port 80 -> $apacheJail
rdr pass on $interface proto tcp from any to $interface port 6667 -> $ircJail

block in log on $interface
pass in on $interface proto udp from any to any port 1900
pass in on $interface proto tcp from any to any port 32400
pass in on $interface proto tcp from any to any port 32469
#pass in on $interface proto tcp from any to any port 8324
#pass in on $interface proto tcp from any to any port 3005
#pass in on $interface proto udp from any to any port 5353
pass in on $interface proto udp from any to any port 32410:32414
pass in on $interface proto tcp from any to any port 2662
pass out on $interface proto {tcp,udp,icmp} all

Code:
brad@mercury:/usr/jails$ sudo sockstat -4
Password:
USER  COMMAND  PID  FD PROTO  LOCAL ADDRESS  FOREIGN ADDRESS   
www  httpd  52156 3  tcp4  192.168.0.102:80  *:*
www  httpd  52155 3  tcp4  192.168.0.102:80  *:*
www  httpd  52154 3  tcp4  192.168.0.102:80  *:*
brad  sshd  42104 3  tcp4  192.168.0.101:2662  192.168.0.100:48182
brad  sshd  42104 8  tcp4  192.168.0.103:38482  192.168.0.103:6667
root  sshd  42102 3  tcp4  192.168.0.101:2662  192.168.0.100:48182
root  sendmail  1538  3  tcp4  127.0.0.1:25  *:*
root  sendmail  1482  4  tcp4  192.168.0.102:25  *:*
www  httpd  1478  3  tcp4  192.168.0.102:80  *:*
www  httpd  1477  3  tcp4  192.168.0.102:80  *:*
www  httpd  1476  3  tcp4  192.168.0.102:80  *:*
www  httpd  1475  3  tcp4  192.168.0.102:80  *:*
www  httpd  1474  3  tcp4  192.168.0.102:80  *:*
root  httpd  1472  3  tcp4  192.168.0.102:80  *:*
root  syslogd  1421  6  udp4  192.168.0.102:514  *:*
root  sendmail  1159  4  tcp4  192.168.0.103:25  *:*
72  inspircd  1132  6  udp4  192.168.0.103:26700  *:*
72  inspircd  1132  7  tcp4  192.168.0.103:6667  *:*
72  inspircd  1132  8  tcp4  192.168.0.103:6667  192.168.0.103:38482
72  inspircd  1132  9  tcp4  192.168.0.103:6667  192.168.0.103:22317
72  inspircd  1132  10 tcp4  192.168.0.103:6667  192.168.0.103:46651
root  syslogd  1098  6  udp4  192.168.0.103:514  *:*
root  sshd  994  4  tcp4  *:2662  *:*
root  ntpd  961  20 udp4  *:123  *:*
root  ntpd  961  22 udp4  192.168.0.101:123  *:*
root  ntpd  961  25 udp4  127.0.0.1:123  *:*
root  ntpd  961  27 udp4  192.168.0.103:123  *:*
root  ntpd  961  29 udp4  192.168.0.102:123  *:*
root  syslogd  811  7  udp4  *:514  *:*
?  ?  ?  ?  tcp4  192.168.0.101:732  192.168.0.105:2049
 
To anyone having an issue like this, it seems that Plex requires bhyve(8) (or VIMAGE) instead of a jail. Once in a true virtual environment and not a jail your fw rules must be set up similar to mine (above). Since using the later versions of Plex I am unable to get anything working when in a jail. Only once in bhyve(8) did it work. I have not used VIMAGE since it is still experimental at this time but have heard it is works.

HTH
 
I know that I am bring this up from the dead, but Plex needs to have raw sockets in the jail for DLNA to work. Hopefully, this will help others running Plex inside of a jail.

I added to my ezjail config file and restarted:

Code:
export jail_plex_parameters="allow.raw_sockets=1"

To test the setting change, I was able to ping out of the jail.

Magically, Plex showed up on my TV tuner.

I found this post helpful: https://forums.freebsd.org/threads/13272/
 
Back
Top