PF rules won’t load on a XEN instance

[Veeny]

Folks,

I have created a new virtual FreeBSD machine on one of my hosting provider's servers, and upgraded it to 12.2-RELEASE-p7

I want to enforce very basic pf rules:

block in from any

pass inet proto tcp from any port {http, https, ssh}

# ANCHORS FOR BLACKLISTD
anchor "blacklistd" all {
        anchor "22" all
}

But when I try to disable/enable pf, I get the following message:

> sudo service pf stop
pfctl: DIOCGETSTATUS: Permission denied
> sudo service pf start
Enabling pfpfctl: DIOCADDRULENV: Operation not supported by device
pfctl: DIOCGETSTATUS: Permission denied
pfctl: pf already enabled
.

The network interface is xn0:

> dmesg | grep xn0
xn0: xbd0: <Virtual Network Interface>20480MB <Virtual Block Device> at device/vbd/51712 at device/vif/0 on xenbusb_front0
xn0: Ethernet address: 00:16:3e:1b:69:43
xn0: backend features: feature-sg feature-gso-tcp4
xn0: 2 link states coalesced
xn0: link state changed to UP

I'm not sure what happens here. Apparently, the hypervisor doesn't authorize the underlying OS any access to the peripheral (promiscuous mode)?
I have, however, other instances of FreeBSD running on the same infrastructure, and pf has always worked like a charm.

Does anyone have an idea what’s going on?

Thanks a bunch,
Vincent

 

[SirDice]

(promiscuous mode)

PF doesn't put the interface in promiscuous mode. So it's something else that's causing it.

 

[Kristof Provost]

> Enabling pfpfctl: DIOCADDRULENV: Operation not supported by device

That's very strange. DIOCADDRULENV is a new ioctl(). A 12.2 userspace should not know to call it. Did you somehow update your userspace without updating the kernel?

 

[Veeny]

PF doesn't put the interface in promiscuous mode. So it's something else that's causing it.

Well, makes sense, since pf works in pass-through mode.

I tried to google the error messages, but to no avail. DIOCADDRULENV is obviously some sort of ioctl, but why it fails is obscure.
Using the -i option doesn't make it work any better:


> sudo pfctl -f /etc/pf.conf -i /dev/xn0
pfctl: DIOCADDRULENV: Operation not supported by device


 

[Veeny]

> Enabling pfpfctl: DIOCADDRULENV: Operation not supported by device

That's very strange. DIOCADDRULENV is a new ioctl(). A 12.2 userspace should not know to call it. Did you somehow update your userspace without updating the kernel?

Ah no, I used the standard `freebsd-update -r 12.2` to upgrade the system from 12.1. Nothing fancy, pinkie promise!

 

[Kristof Provost]

Ah no, I used the standard `freebsd-update -r 12.2` to upgrade the system from 12.1. Nothing fancy, pinkie promise!

Check again, because that's basically the only way this could be happening.

 

[Veeny]

Check again, because that's basically the only way this could be happening.

> uname -K -U
1202000 1202508
> sudo freebsd-update fetch
src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching metadata signature for 12.2-RELEASE from update1.freebsd.org... done.
Fetching metadata index... done.
Inspecting system... done.
Preparing to download files... done.

No updates needed to update system to 12.2-RELEASE-p10.

 

[Kristof Provost]

I promise you, a 12.2 pctl does not call DIOCADDRULENV. It does not know it, as you can confirm by looking at the releng/12.2 branch. Somehow you wound up with a more recent pfctl on your system.

 

[Veeny]

> strings /sbin/pfctl | grep CADD
DIOCADDADDR
DIOCADDRULENV
DIOCADDALTQ

I thought there was a way to figure out from what version of the code my pfctl had been compiled from, but no dice.

I promise you, a 12.2 pctl does not call DIOCADDRULENV. It does not know it, as you can confirm by looking at the releng/12.2 branch. Somehow you wound up with a more recent pfctl on your system.

Oh, I don't call your word into question! I'll try and fetch pfctl directly from the ftp server and replace mine. I'll keep you posted.

 

[SirDice]

but no dice.

I had nothing to do with it!

Looking through the logs releng/13.0 doesn't have it either, stable/13 does though.

src - FreeBSD source tree

cgit.freebsd.org

Oh, I don't call your word into question!

Look at the author/committer

I'll try and fetch pfctl directly from the ftp server and replace mine. I'll keep you posted.

Is it possible somebody accidentally checked out -CURRENT in /usr/src and then built and installed sbin/pfctl.c?

 

[Veeny]

I had nothing to do with it!

Lol. I know, I'm not going to drag you into this

Looking through the logs releng/13.0 doesn't have it either, stable/13 does though.

src - FreeBSD source tree

cgit.freebsd.org

Look at the author/committer


Is it possible somebody accidentally checked out -CURRENT in /usr/src and then built and installed sbin/pfctl.c?

I don't think so, really. I mean, the first time I created this server, I upgraded to 13.0-RELEASE, but it failed ignominiously (it couldn't remount root, there were hash errors everywhere). So I deleted the server, and recreated it with a fresh 12.1 install. Is it possible I was given the same disk, simply overwritten? Might have some utilities survived from the crash/deletion/recreation? I wonder.

Oh, I don't call your word into question! I'll try and fetch pfctl directly from the ftp server and replace mine. I'll keep you posted.

With pfctl from the ftp server, it works fine. Thanks for the hint!

 

[SirDice]

. I mean, the first time I created this server, I upgraded to 13.0-RELEASE, but it failed ignominiously (it couldn't remount root, there were hash errors everywhere).

That could happen. But looking through the logs 13.0-RELEASE doesn't have that change either. So something else went awry.

So I deleted the server, and recreated it with a fresh 12.1 install. Is it possible I was given the same disk, simply overwritten? Might have some utilities survived from the crash/deletion/recreation? I wonder.

I doubt that. Even if you installed it on top of an existing install those files would have been overwritten.


If that pfctl(8) was modified on the system there's no telling what else may have been inadvertently updated though. It might be a good idea to run freebsd-update IDS.

 

[Veeny]

It might be a good idea to run freebsd-update IDS.

Almost every binary file turns out to be wrong.
I'm not sure how the system runs, but it does. PHP works, and so does NGINX. ZFS seems to work too, so…
I'm not sure what I shall do. For the time being, I'm going to left it as is. If anything goes wrong again, I will reinstall the system.

Thanks for your help!

 

[SirDice]

Almost every binary file turns out to be wrong.

Yeah, I was afraid of that.

That sort of confirms my idea of what may have happened. I suspect at some point -CURRENT was checked out in /usr/src and world was built and installed from that. It's quite easy to get the wrong version of the source code if you're not familiar with the process.

If anything goes wrong again, I will reinstall the system.

Expect the unexpected to happen. And plan ahead.

 

[Veeny]

Yeah, I was afraid of that.

That sort of confirms my idea of what may have happened. I suspect at some point -CURRENT was checked out in /usr/src and world was built and installed from that. It's quite easy to get the wrong version of the source code if you're not familiar with the process.


Expect the unexpected to happen. And plan ahead.

But I never updated from source! I used `freebsd-upgrade -r 12.2`. The update was purely made from binaries.


> cd /usr/src
> ls -la
total 8
drwxr-xr-x 2 root wheel 512 Sep 23 07:47 .
drwxr-xr-x 15 root wheel 512 Sep 29 09:55 ..