PF "cannot define table [...] Cannot allocate memory" since upgrade to 13.0

D

demonnodevil

Last weekend I upgraded my two VPS servers from 12.2 to 13.0. My primary (slightly more busy) server is fine while the secondary has an issue loading a table in the pf after a day of running.

If I do a pfctl -v -f /etc/pf.conf right after reboot it is fine. If I do it again after a day running, I get this error:
/etc/pf.conf:16: cannot define table green: Cannot allocate memory
Click to expand...

Line 16 contains: table <green> persist file "/etc/pf.green.table"

I could use a little assistance on what to check next.



What I already did:
  • I checked (with MD5 hashes) that /etc/pf.conf on both servers are the same. (They are)
  • I checked (with MD5 hashes) that the table files (/etc/pf.*.table) on both servers are the same. (They are)
  • I checked available memory with top
  • I checked the number of entries in pf.green.table ( wc -l /etc/pf.green.table returns 8308)
  • I checked the number of entries in all table files ( wc -l /etc/pf.*.table returns 8411)
  • I checked the limits from pfctl -sa they appear to be will within range

Top on Primary server:
Mem: 210M Active, 1875M Inact, 949M Laundry, 739M Wired, 394M Buf, 109M Free
Swap: 4096M Total, 598M Used, 3498M Free, 14% Inuse
Click to expand...

Top on Secondary server:
Mem: 276M Active, 1734M Inact, 1128M Laundry, 623M Wired, 283M Buf, 124M Free
Swap: 4096M Total, 405M Used, 3690M Free, 9% Inuse
Click to expand...

pfctl -sa:
[...]
LIMITS:
states hard limit 100000
src-nodes hard limit 10000
frags hard limit 5000
table-entries hard limit 200000
[...]
Click to expand...
 
Thank you. That is going to take a little bit of effort. I guess I will fire-up another virtual and see if I can build using the diffs provided.
 
Do wait for the update. It's coming in an hour or so. The current version has a bunch of unrelated changes that you don't work. And that possibly don't even work.
 
Hello,

It's strange, I'm also having the same problem right now on FreeBSD 13.0-RELEASE-p5. Never got it before.
My persisted file contains only 2 IPv4 addresses and 2 IPv6 prefixes and my pf setup hasn't changed in years.
 
OlivierW said:
Hello,

It's strange, I'm also having the same problem right now on FreeBSD 13.0-RELEASE-p5. Never got it before.
My persisted file contains only 2 IPv4 addresses and 2 IPv6 prefixes and my pf setup hasn't changed in years.
Click to expand...
I find that if pf crashes when loading tables, then you'll need to reboot the system. Seems parts of it? some of it? is left in RAM and blocks (in my case, large) tables from re-loading.

I'm still monitoring the situation. What I've done so far is to comment out automatic pf_load on boot, then load it manually. So far so good.

pfctl -sa

Code: 
LIMITS:
states        hard limit   100000
src-nodes     hard limit    10000
frags         hard limit     5000
table-entries hard limit 25400000
 
exist said:
I find that if pf crashes when loading tables, then you'll need to reboot the system
Click to expand...
Flushing everything seems to work for me. pfctl -Fa -f /etc/pf.conf but this does also kill your current ssh(1) session if you do this remotely, so be wary.
 
SirDice said:
Flushing everything seems to work for me. pfctl -Fa -f /etc/pf.conf but this does also kill your current ssh(1) session if you do this remotely, so be wary.
Click to expand...
Unfortunately this didn't work for me. I'm using pf-badhosts, but it seems the issue itself is with pf.

For example, as the OP posted, if the machine is left running for a few days, I'll see in the daily log that pf-badhosts has failed to update the database due to lack of memory. This error appears:

Code: 
pf-badhost 15202 - - Using experimental "aggy" aggregator...

pfctl: Cannot allocate memory.

pf-badhost 15256 - - ERROR: '/etc/pf-badhost.txt' contains invalid data! Reverting changes and bailing out...

If I log into the machine and flush everything, I'll get disconnected first, but am able to reconnect. If I try to manually run the pf-badhosts again, *all* tables fail to load like this:

Code: 
# pfctl -e -f /etc/pf.conf
/etc/pf.conf:18: cannot define table pfbadhost: Cannot allocate memory
/etc/pf.conf:23: cannot define table rfc6890: Cannot allocate memory
/etc/pf.conf:26: cannot define table gooDNS6: Cannot allocate memory
/etc/pf.conf:27: cannot define table friends: Cannot allocate memory
pfctl: Syntax error in config file: pf rules not loaded

it's like pf isn't letting go of memory it was allocated previously.

If the box is rebooted and pf-badhosts is run again, it all works:

Code: 
# doas -u _pfbadhost pf-badhost -O freebsd                                                                         
Password:

pf-badhost 1512 - - Using experimental "aggy" aggregator...

6105 addresses added.
6235 addresses deleted.

pf-badhost 1580 - -
IPv4 addresses in table:  619200750
 
had a look at resource limits

Code: 
# limits
Resource limits (current):
  cputime              infinity secs
  filesize             infinity kB
  datasize              1048576 kB
  stacksize             1048576 kB
  coredumpsize         infinity kB
  memoryuse            infinity kB
  memorylocked         infinity kB
  maxprocesses            12070
  openfiles              231147
  sbsize               infinity bytes
  vmemoryuse           infinity kB
  pseudo-terminals     infinity
  swapuse              infinity kB
  kqueues              infinity
  umtxp                infinity

maximum data size is only 1GB and some tables might get over that number temporarily, so looked here:
Code: 
# sysctl kern.maxdsiz
kern.maxdsiz: 1073741824

set it to 4GB
Code: 
# sysctl kern.maxdsiz=4294967296
kern.maxdsiz: 1073741824 -> 4294967296

surprised it could be changed on-the-fly. reloaded tables:

Code: 
# doas -u _pfbadhost pf-badhost -O freebsd
Password:

pf-badhost 2367 - - Using experimental "aggy" aggregator...

15 addresses added.

pf-badhost 2435 - -
IPv4 addresses in table:  619200765

I have another rpi4b same spec, used for testing. That one has geoip blocking functionality of pf-badhost turned on and so the tables are very much larger. Seems to have loaded fine:

Code: 
# doas -u _pfbadhost pf-badhost -O freebsd
Password:

pf-badhost 25210 - - Using experimental "aggy" aggregator...

17580 addresses added.
9092 addresses deleted.

pf-badhost 25286 - -
IPv4 addresses in table:  3192818881

will keep monitoring this to see if the error re-occurs
 
yeah it happens still. This is after about 48 hrs uptime.

Code: 
State Table                          Total             Rate
  current entries                       13               
  searches                         5758905       303100.3/s
  inserts                            42364         2229.7/s
  removals                           42351         2229.0/s
Counters
  match                              70314         3700.7/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                              0            0.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                         0            0.0/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                              0            0.0/s
  synproxy                               0            0.0/s
  map-failed                             0            0.0/s

LABEL COUNTERS:

TIMEOUTS:
tcp.first                    30s
tcp.opening                   5s
tcp.established           18000s
tcp.closing                  60s
tcp.finwait                  30s
tcp.closed                   30s
tcp.tsdiff                   10s
udp.first                    60s
udp.single                   30s
udp.multiple                 60s
icmp.first                   20s
icmp.error                   10s
other.first                  60s
other.single                 30s
other.multiple               60s
frag                         30s
interval                     10s
adaptive.start            60000 states
adaptive.end             120000 states
src.track                     0s

LIMITS:
states        hard limit   100000
src-nodes     hard limit    10000
frags         hard limit     5000
table-entries hard limit 25400000
Code: 
# doas -u _pfbadhost pf-badhost -O freebsd
Password:

pf-badhost 8462 - - Using experimental "aggy" aggregator...

pfctl: Cannot allocate memory.

pf-badhost 8516 - - ERROR: '/etc/pf-badhost.txt' contains invalid data! Reverting changes and bailing out...

this is with kern.maxdsiz=4294967296

I think the problem is with pf not liberating resources when it's done with them.
 
You must log in or register to reply here.
Back
Top