ipfw and tables

danbi · Sep 12, 2011

I am using ipfw with tables of IP addresses. Tables get updated few times a hour with a sequence like this:

Code:

ipfw table 1 flush
cat file | while read line;do
ipfw table 1 add $line 1
done

This.. sort of works, but once in a while some IP address hangs in the table and cannot be cleared. I have this behavior on systems like

Code:

# uname -a
FreeBSD xxx1 7.1-STABLE FreeBSD 7.1-STABLE #0: Tue Feb  3 11:36:55 EET 2009     root@xxx1:/usr/obj/usr/src/sys/ROUTER3  amd64

Code:

# uname -a
FreeBSD xxx2 8.2-STABLE FreeBSD 8.2-STABLE #0: Mon May  9 15:29:46 EEST 2011     root@xxx2:/usr/obj/usr/src/sys/GENERIC  amd64

The first one has IPFW in kernel, the second one loads it as a module.

Addresses get struck in IPFW, because the sequence

# ipfw table 1 flush
# ipfw table 1 list

produces non-empty output.

On the system, where IPFW is loaded as module, upon unload of the module, this is produced:

Code:

Sep 12 10:49:04 xxx2 kernel: Warning: memory type ipfw_tbl leaked memory on destroy (9 allocations, 2304 bytes leaked).

(usually 1 allocation is leaked)

Also, today on that system I noticed that each time above reloading of the table happens, an message is produced:

Code:

Sep 12 10:49:04 xxx2 kernel: rn_delete: couldn't find our annotation

I was expecting this table bug to be fixed sometimes after 7.1, as probably a lot of people are using this functionality, but perhaps are not reloading it that often? Or is there a better method to manipulate/update tables?

quintessence · Sep 15, 2011

Hello,

Did you try instead of flushing the whole table to delete the address(es)?

Code:

ipfw delete IP_addr

danbi · Sep 17, 2011

This does not work either. Any IP address listed after a flush is not modifiable --- as would be expected with a memory leak that messed up IPFW internal structures.

quintessence · Sep 17, 2011

Hello,

Not after flushing, before it.
Btw, how many rows does contain your table 1?

EDIT: for the last message you provide, you can search at net mailing list from April this year "Routing enhancement - reduce routing table locking".

danbi · Sep 18, 2011

I looked at the thread you mentioned in -net, but cannot correlate it with IPFW and tables.

This particular table has between 4000 and 5000 entries. I could of course iterate over the table content and remove entries one by one, but this would might end up with the same problem. A 'flush' should be guaranteed to make the table empty and should be an efficient atomic operation.

For the time being I "solve" the issue by periodically unload/reload the ipfw module in the kernel and reload the tables, but this is rather a dirty workaround to keep things going.

It is interesting that these events occur much more frequently at the first load/unload of the module --- after that there are only

Code:

Sep 17 12:00:00 xxx kernel: ipfw2 (+ipv6) initialized, divert loadable, nat 
loadable, rule-based forwarding disabled, default to accept, logging disabled
Sep 17 12:14:38 xxx kernel: rn_addmask: mask impossibly already in treern_ad
dmask: mask impossibly already in treeNon-unique normal route, mask not entered
Sep 17 12:29:37 xxx kernel: rn_delete: couldn't find our annotation
Sep 17 13:00:00 xxx kernel: IP firewall unloaded

quintessence · Sep 19, 2011

Hello,

4000-5000 entries is not so much.

The correlation was the ipfw's radix tree, there are some improvements made in RADIX part, but seems they are mostly in routing part.

Code:

diff -u -r ../src_org_8.2_20110329/sys/net/radix.c ./sys/net/radix.c


diff -u -r ../src_org_8.2_20110329/sys/net/radix.h ./sys/net/radix.h

I can make some test on 8.2-STABLE #0: Thu Jul 28 11:04:34 EEST 2011 with 10000 entries in IPFW table, and I'll edit my post.

Update:

Code:

# ipfw table 1 list | wc -l
   10200
# ipfw table 1 flush
# ipfw table 1 list | wc -l
       0

with no error on the version above.

danbi · Sep 27, 2011

Have you tried to flush/reload your table many times?

quintessence · Sep 27, 2011

Hello,

About 10 times, I'll setup cron job on 5 minutes to flushing and updating the table.

---
Update: ~20 hours with no errors.

Code:

ipfw2 initialized, divert enabled, nat loadable, rule-based forwarding disabled, default to accept, logging disabled

I also tries with 20k entries in the table with no errors (or lagging) on flushing/reloading.

How many memory does your server have? Does it swap?
My test PC is with 4GB ram, no sysctl tuning.

abraxis · Jun 18, 2012

Hello,
I have the same problem

Code:

root@[******] /var/log/ipfw# ipfw table 123 flush
root@[******] /var/log/ipfw# ipfw table 123 list
*.72.124.140/32 0

Code:

root@[******] /var/log/ipfw# uname -a
FreeBSD ******  8.2-RELEASE-p2 FreeBSD 8.2-RELEASE-p2 #2: Fri Jul 15 03:01:58 CEST 2011     
root@[******] /var/log/ipfw#

before flush table 123 has about 100 entries

Code:

CPU 0:  0.0% user,  0.0% nice, 25.5% system,  4.9% interrupt, 69.6% idle
CPU 1:  0.0% user,  0.0% nice, 40.8% system, 26.2% interrupt, 33.0% idle
Mem: 392M Active, 136M Inact, 1024M Wired, 77M Cache, 213M Buf, 342M Free
Swap: 2200M Total, 52K Used, 2200M Free

Code:

hw.machine: amd64
hw.model: Dual-Core AMD Opteron(tm) Processor 2214 HE
hw.ncpu: 2
hw.machine_arch: amd64

asterisks for anonymity