Solved PF rules don't load in a jail after upgrade to 15.1

S

Shakroglof

Since I upgraded host + jails from 14.4 to 15.1, pf rules don't load in a jail.

Inside the jail :
Code: 
root@sigil:~ # service pf start
Enabling pfpfctl: DIOCADDRULE: Operation not permitted
/etc/rc.d/pf: WARNING: Unable to load /etc/pf.conf.
pfctl: DIOCSTART: Operation not permitted

Rules on the host load fine.
I didn't see anything related to pf in jails in the 15.0 and 15.1 release notes.
I didn't see errors during the upgrade process.
Freebsd-update IDS tells me only /etc files differ on the host.

In the jail, Freebsd-update IDS tells me some files related to pfctl in /usr/tests differ, is that relevant ?
Code: 
/usr/tests/sbin/pfctl/files/pf0004.ok has SHA256 hash 00a214cee2643808ac73953f6fc04ce0b19a2079e76825489e4431e15c9608b8, but should have SHA256 hash 375a8e4969873c583c6dad09f9aa087bb1b077f2ca92b0112a4d0e2b352bcd67.
/usr/tests/sbin/pfctl/files/pf0016.in has SHA256 hash 31093d27b473a05921fa031c3bbaef060bd616618fb90d1b54bc3262bdc7856a, but should have SHA256 hash 4376227f5656bb1e6f8c4195cf6b5d2c286162012750b1402d55c2ca60be77c4.
/usr/tests/sbin/pfctl/files/pf0016.ok has SHA256 hash 7f9f073f0f140a49ee5f2e6866712cd9f273e70a610acff719b2a044d0e16b8e, but should have SHA256 hash 2b779b89c5f6d4cffb02cb39c217ce8011da0da8575f5c8113f3be072cc8363c.
/usr/tests/sbin/pfctl/files/pf0018.in has SHA256 hash d3fa700e1374e2faab5d048fddccbe2ead9f6479886b6dcb78d8b4c3a7771bd2, but should have SHA256 hash c2c6255469a7f2fd482cd30ee3cfe84cc369330ce1f6075ecc247cc478328cae.
/usr/tests/sbin/pfctl/files/pf0018.ok has SHA256 hash 6f06e046cb93d5c104e05a8d7574303094a3498e9c48a54ec3bd30a5ecb28698, but should have SHA256 hash 44cf5855b8cd291e6ab7adbc20301e61051ace4c0417a1ee9fefd0727f40a049.
/usr/tests/sbin/pfctl/files/pf0019.in has SHA256 hash 830bc4112d81d5873e49058247827b962e8939c8d84641360b94dc916fe0976f, but should have SHA256 hash 56925ec50b4bcc542911d30f8523b333f055617d66c5e33b3eaa4935440aa404.
/usr/tests/sbin/pfctl/files/pf0019.ok has SHA256 hash 23eacfd8066a93ac2a6caf59474df62b49f4dd986ec20305dd39e34fab7f434a, but should have SHA256 hash e52f0ea5560149a1fb8546614dd6375c1146aa3872b6383889428b698db83b30.
/usr/tests/sbin/pfctl/files/pf0020.in has SHA256 hash f29e2100bde658b27bec5b8a91d9d6fc327ddec87c26fe6cf9ee8e7fe9fc1582, but should have SHA256 hash ec63a3db5e085c9bd3f0da1a84fba453e1a0855e7dbd0fcf854d38297e30ea0e.
/usr/tests/sbin/pfctl/files/pf0020.ok has SHA256 hash 23eacfd8066a93ac2a6caf59474df62b49f4dd986ec20305dd39e34fab7f434a, but should have SHA256 hash 921eed16454361058fc0dcffcef7e9f9756dcda6fe626487eca049acb7fa07c0.
/usr/tests/sbin/pfctl/files/pf0048.in has SHA256 hash d40c3b0742f487f0cf92cade07993720d441f9e22ac5be77e51a09c9a7fcfe75, but should have SHA256 hash 5d56af3491056d830000da4e387775e3db9b7ececc8bc8b548648a00cc143774.
/usr/tests/sbin/pfctl/files/pf0048.ok has SHA256 hash b0c35e900bce08049f6924e0edd11364ff2c6d729b5a1512774d86d47e487767, but should have SHA256 hash 3b7d274b8bf60169a9e9b4b89abd6637bef779b56ceaad361d9ffcc7860093d4.
/usr/tests/sbin/pfctl/files/pf0069.in has SHA256 hash f5dd5aaa74eda1f556e9743185b7e7e1840b9c28571bb31ff5eccf53eac5a6f4, but should have SHA256 hash a8572ed6a21041663ece3284abcd5d73a82279e43de532ebff9b2e555f1f842b.
/usr/tests/sbin/pfctl/files/pf0069.ok has SHA256 hash d64b3972ea430b8c7802dd25ca65b8000d10b4b36b7d830a8c1d6f4f60aa8670, but should have SHA256 hash e4f6f1bb6097f2c93fc957a2be1a127ddab1bb48d93fd46f220c5c1fada873fb.
/usr/tests/sbin/pfctl/files/pf0070.in has SHA256 hash aceb3093a780847f39298e9a797ce49eb3a58f8a6d52ecf491be3909b9e517cd, but should have SHA256 hash 8513d872599ed971e768787e044f0b031ced803e3e3e7d58bfd4d505fb83cf8a.
/usr/tests/sbin/pfctl/files/pf0070.ok has SHA256 hash e73dbb5a987ff29355bcad9ba35f3d380be19d79ee0c7c8ba4a8c828b6be3071, but should have SHA256 hash 8cffe3282f07b9e1ae58d4b89ebe28ee0acc7129ca7d9ffc615e0e85610fc03b.
/usr/tests/sbin/pfctl/files/pf0071.in has SHA256 hash dd817010be0b4b8c0f13a97c6a2f9549d742d409988e763cded25502a6ece5a4, but should have SHA256 hash 606b714e7d38ed5d60fe96dae04b8597e926df8b3ecaa944e9951aea771a4842.
/usr/tests/sbin/pfctl/files/pf0071.ok has SHA256 hash e73dbb5a987ff29355bcad9ba35f3d380be19d79ee0c7c8ba4a8c828b6be3071, but should have SHA256 hash a08947b865c71068106138b4d4cd3d750f3a0c783a71d4f4c7b88d68d2244f6a.
/usr/tests/sbin/pfctl/files/pf0072.in has SHA256 hash 9dcb774cbf3e97de584887f03c32d0adc6ce4eebfd61073f66926e8636f185ad, but should have SHA256 hash b34dd28c1f777ac61e8e2f9c53cd38a7ef5a1e094f4cde5fddb9d6610739b850.
/usr/tests/sbin/pfctl/files/pf0072.ok has SHA256 hash e73dbb5a987ff29355bcad9ba35f3d380be19d79ee0c7c8ba4a8c828b6be3071, but should have SHA256 hash 2cda1446d676311313593507a3b7fb8c57df3450377bd39081d266b8efd9b380.
/usr/tests/sbin/pfctl/files/pf0084.in has SHA256 hash 3f6c9391645abcae036e7a46b8f3fc9e9c9ad7887b94bdf95d6a9671e9eb8d1c, but should have SHA256 hash d4f0e0a36c541408536ab6dde1b2d9be2414950f93e91e6ebc90550ddaa8c2b8.
/usr/tests/sbin/pfctl/files/pf0084.ok has SHA256 hash efd70af071f4bc2be30a1004f637b0d2ec4f2cae8e1bc05283f6623a73b9af7f, but should have SHA256 hash 73919ffac8b3a9a6a6dc33485f90e531173b8c861153c31259648b03a2fa9260.
/usr/tests/sbin/pfctl/files/pf0088.in has SHA256 hash bf40657987c3a0decc16980fb2d81693b1e8bb1544aaf197ffa831de75ec9ca7, but should have SHA256 hash 7cd0386a484366b043474c12c2b8288b54abdd49bfaa9ad8a74a3f7cb0ba97a3.
/usr/tests/sbin/pfctl/files/pf0088.ok has SHA256 hash 78dd1b2295d0677ad03cba7de9d246c91d0bee03f525e5e08edef1ae677f5f1b, but should have SHA256 hash ca266679afd0b86428added06f730f67ce4bb0d46c90e1b6179d5e074c0d7990.
/usr/tests/sbin/pfctl/files/pf0098.in has SHA256 hash 3d44dc87627874c003ca2b132ccecc3dc2e3d42e5ef2191cf831f5f4ab5ddf58, but should have SHA256 hash 83f6f81ad776f60eff39b86edc10aaad3638decd1429e42e93e891ffb3e60e3b.
/usr/tests/sbin/pfctl/files/pf0098.ok has SHA256 hash 780ecf08802ff932de721fb3862280d158bfaf111259ad64628ea3965681c987, but should have SHA256 hash d00f8dd7e05c04a2e9f34e37106cf01c30b9abd11b6d0a0f58a3dace84cf7f41.
 
@alelab@bsd.cafe has mentioned this on mastodon, and I've seen the same thing in my homelab too.

We're both using bastille, so Alexandre opened a problem report with bastillebsd, but it seems more likely to me to be a kernel issue than anything bastille is doing. I've seen the issue on aarch64 and amd64. I've wanted to create a test case using the jail commands directly but I haven't managed it yet.
 
How did you upgrade the bastille jail? I have a sneaky suspicion your jail is still running on 14.x.
 
SirDice said:
How did you upgrade the bastille jail? I have a sneaky suspicion your jail is still running on 14.x.
Click to expand...
Using bastille commands : bastille bootstrap and bastille upgrade

The base system of the jail points to the new release :
Code: 
imrryr# mount|grep sigil
/usr/local/bastille/releases/15.1-RELEASE on /usr/local/bastille/jails/sigil/root/.bastille (nullfs, local, read-only)
 
SirDice said:
How did you upgrade the bastille jail? I have a sneaky suspicion your jail is still running on 14.x.
Click to expand...
That's an interesting thought. I have about ten thin vnet jails on the aarch64 system; freebsd-version reports 15.1-RELEASE or 15.0-RELEASEp10 as expected. They were all (I believe!) running 15.0 when the host was 15.0; that's what they reported, and pf worked in the jails.

The upgrade this time (not all jails have been upgraded yet) was

Code: 
# bastille bootstrap 15.1-RELEASE
# bastille upgrade test 15.1-RELEASE

The fstab's look like this:

Code: 
# grep nullfs */fstab
acme/fstab:/usr/local/bastille/releases/15.0-RELEASE /usr/local/bastille/jails/acme/root/.bastille nullfs ro 0 0
caldav/fstab:/usr/local/bastille/releases/15.1-RELEASE /usr/local/bastille/jails/caldav/root/.bastille nullfs ro 0 0
(etc)

& the dates of the files in /usr/local/bastille/releases/15.1-RELEASE/bin are all recent (June 12th)...

I see that some of my jail.conf's are still using jib. I'll change that - but the bridge is created in rc.conf so it surely can't be a problem.
 
That looks good, but I've been bitten by that before. What does bastille cmd sigil freebsd-version output?
 
I created a 15.1 thick jail as per the handbook, changed the devfs ruleset to allow /dev/pf and created a simple pf.conf and i could start pf:
Code: 
imrryr# jexec classic service pf onestart             
Enabling pf.
So it might be related to bastille or a problem during the upgrade...
 
Creating a new account to mention this bit us at work today.

https://reviews.freebsd.org/D56390 seems to change pf add rules to
only be securelevel > 2 need securelevel < 2 (so securelevel=1)
while Bastille defaults to securelevel=2 for all jails.
EDIT: Wrong comparison operator.

Our normal setup uses vnet netgraph jails running their own pf. Host at 15.0-RELEASE would allow adding rules fine. Upgrading to 15.1-RELEASE gave the pfctl: DIOCSTART: Operation not permitted errors.

Our 14.X jails did not experience an issue adding pf rules at securelevel=2 on a 15.1-RELEASE host.
 
kiplings5 said:
https://reviews.freebsd.org/D56390 seems to change pf add rules to only be securelevel > 2 while Bastille defaults to securelevel=2 for all jails.
Click to expand...

That's an awfully big change to get no mention at all in the release notes. It would be very helpful to know how to run a vnet jail with pf, using securelevel 3.

edit: the above makes no sense. I don't want to run my jails at securelevel 1, and I _do_ want to have local firewalls. I guess I'm going to have to completely redesign the way I'm doing jails. Or is securelevel 1 fine for jails?
 
putney said:
That's an awfully big change to get no mention at all in the release notes. It would be very helpful to know how to run a vnet jail with pf, using securelevel 3.

edit: the above makes no sense. I don't want to run my jails at securelevel 1, and I _do_ want to have local firewalls. I guess I'm going to have to completely redesign the way I'm doing jails. Or is securelevel 1 fine for jails?
Click to expand...
I'll edit my above post. I realize I put the comparison operator the wrong way.

Hands on experience says 15.X VNET jails currently need to be securelevel 1 to add pf rules. That was the workaround done today at work.
 
putney said:
That's an awfully big change to get no mention at all in the release notes. It would be very helpful to know how to run a vnet jail with pf, using securelevel 3.

edit: the above makes no sense. I don't want to run my jails at securelevel 1, and I _do_ want to have local firewalls. I guess I'm going to have to completely redesign the way I'm doing jails. Or is securelevel 1 fine for jails?
Click to expand...
Yeah, having to lower the jail securitylevel to setup a firewall can't be good for security ?
 
Does anyone know when the securelevel is increased during jail startup? Is this documented? (I couldn't find it.) Perhaps there's a workaround for starting pf in the jail before this happens?
 
So: it doesn't help to make sure pf is started before the securelevel is increased (/etc/rc.d/pf and /etc/rc.d/securelevel, in the jail). Even if pf is enabled early, pfctl doesn't work.

However, there's this similar problem - <https://lists.freebsd.org/archives/freebsd-jail/2026-June/000777.html>.

In that case, a "FreeBSD 14 pfctl" in a jail worked on a 15.0-RELEASE host, because it's bypassing the new netlink code. Is it possible this also work on a 15.1-RELEASE host?
 
We can have the jail at securelevel 2 and pf rules loaded at startup : securelevel is set to 1 in jail.conf on the host, and kern_securevel=2 in rc.conf in the jail (and kern_securelevel_enable="YES").
We can even lower the securelevel of the jail from the host with "jail -m securelevel=1 name=myjail" and change the rules with pfctl
 
Thank you very much Shakroglof for your feedback! 🙏
I tried on my jails and it works like a charm!

rc.conf (inside jail)
kern_securelevel_enable="YES"
kern_securevel=2

I even commented in the line securelevel=2 in each jail.conf file. Then I verified with the command sysctl kern_securelevel inside jails, and jails start correctly. The command returns the result 2 (as expected).
 
You must log in or register to reply here.
Back
Top