I want to optimize my FreeBSD pf NAT firewall and my questions are:
What exactly controls how much RAM pf will use for state tables and other data it may use in RAM? Also, are there any must-have configurations settings for traffic on the ISP level?
I have two FreeBSD 9.1, i5, 16 GB of RAM machines with 6 NICs each. Two NICs in one LAGG group (inside) and two more in another LAGG group (outside), using carp and one interface for pfsync and the last one for the management interface. I have the current stats below for the active firewall but I want to optimize for 700,000 table entries. Right now I'm using half that with 60% idle CPU and around 1.1 TB of data inbound daily.
What exactly controls how much RAM pf will use for state tables and other data it may use in RAM? Also, are there any must-have configurations settings for traffic on the ISP level?
I have two FreeBSD 9.1, i5, 16 GB of RAM machines with 6 NICs each. Two NICs in one LAGG group (inside) and two more in another LAGG group (outside), using carp and one interface for pfsync and the last one for the management interface. I have the current stats below for the active firewall but I want to optimize for 700,000 table entries. Right now I'm using half that with 60% idle CPU and around 1.1 TB of data inbound daily.
Code:
Mem: 52M Active, 34M Inact, 1134M Wired, 1226M Buf, 14G Free
load averages: 2.87, 2.67, 2.63
58.5% idle
Code:
root@ # pfctl -si
Status: Enabled for 3 days 18:29:48 Debug: Urgent
Interface Stats for lagg2 IPv4 IPv6
Bytes In 15352428125696 0
Bytes Out 1939415056495 0
Packets In
Passed 12771296874 0
Blocked 6968223 0
Packets Out
Passed 7750839635 0
Blocked 527616 0
State Table Total Rate
current entries 339464
searches 51784113403 158950.3/s
inserts 381900325 1172.2/s
removals 381660861 1171.5/s
Counters
match 366022632 1123.5/s
bad-offset 0 0.0/s
fragment 33642 0.1/s
short 6501 0.0/s
normalize 9315 0.0/s
memory 0 0.0/s
bad-timestamp 1154571 3.5/s
congestion 0 0.0/s
ip-option 210 0.0/s
proto-cksum 0 0.0/s
state-mismatch 2343282 7.2/s
state-insert 525918 1.6/s
state-limit 0 0.0/s
src-limit 0 0.0/s
synproxy 0 0.0/s
Code:
root@ # vmstat -z
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
UMA Kegs: 208, 0, 105, 14, 105, 0, 0
UMA Zones: 896, 0, 105, 3, 105, 0, 0
UMA Slabs: 568, 0, 5301, 5, 5722, 0, 0
UMA RCntSlabs: 568, 0, 11573, 5, 11573, 0, 0
UMA Hash: 256, 0, 2, 13, 3, 0, 0
16 Bucket: 152, 0, 153, 22, 153, 0, 0
32 Bucket: 280, 0, 169, 13, 169, 0, 0
64 Bucket: 536, 0, 145, 2, 145, 57, 0
128 Bucket: 1048, 0, 12311, 1, 12311, 763, 0
VM OBJECT: 232, 0, 37569, 287, 354587, 0, 0
MAP: 232, 0, 7, 25, 7, 0, 0
KMAP ENTRY: 120, 531185, 41, 486, 11045, 0, 0
MAP ENTRY: 120, 0, 1793, 377, 872259, 0, 0
fakepg: 120, 0, 0, 0, 0, 0, 0
mt_zone: 4112, 0, 302, 9, 302, 0, 0
16: 16, 0, 2658, 534, 515209, 0, 0
32: 32, 0, 3491, 650, 1373615, 0, 0
64: 64, 0, 12422, 1634,40462387557, 0, 0
128: 128, 0, 8287, 616, 71550, 0, 0
256: 256, 0, 913, 647, 245440, 0, 0
512: 512, 0, 1279, 303, 111756, 0, 0
1024: 1024, 0, 77, 179, 56581, 0, 0
2048: 2048, 0, 5146, 364, 39633, 0, 0
4096: 4096, 0, 308, 133, 28234, 0, 0
Files: 80, 0, 139, 311, 2637289, 0, 0
TURNSTILE: 136, 0, 286, 74, 286, 0, 0
umtx pi: 96, 0, 0, 0, 0, 0, 0
MAC labels: 40, 0, 0, 0, 0, 0, 0
PROC: 1184, 0, 73, 89, 19934, 0, 0
THREAD: 1128, 0, 242, 43, 275, 0, 0
SLEEPQUEUE: 80, 0, 286, 120, 286, 0, 0
VMSPACE: 392, 0, 54, 76, 19898, 0, 0
cpuset: 72, 0, 95, 55, 111, 0, 0
audit_record: 960, 0, 0, 0, 0, 0, 0
mbuf_packet: 256, 0, 20453, 1691,23430797474, 0, 0
mbuf: 256, 0, 1025, 1161,19543754432, 0, 0
mbuf_cluster: 2048, 128768, 22145, 821,3283952055, 0, 0
mbuf_jumbo_page: 4096, 262144, 0, 90, 5908, 0, 0
mbuf_jumbo_9k: 9216, 64000, 0, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 32000, 0, 0, 0, 0, 0
mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0
g_bio: 232, 0, 0, 4512, 295500, 0, 0
ttyinq: 160, 0, 300, 156, 735, 0, 0
ttyoutq: 256, 0, 157, 128, 384, 0, 0
ata_request: 328, 0, 0, 0, 0, 0, 0
ata_composite: 336, 0, 0, 0, 0, 0, 0
VNODE: 480, 0, 119594, 214, 2262197, 0, 0
VNODEPOLL: 112, 0, 0, 0, 0, 0, 0
NAMEI: 1024, 0, 0, 96, 7696461, 0, 0
S VFS Cache: 108, 0, 100127, 24151, 1174240, 0, 0
STS VFS Cache: 148, 0, 0, 0, 0, 0, 0
L VFS Cache: 328, 0, 25648, 58136, 1144443, 0, 0
LTS VFS Cache: 368, 0, 0, 0, 0, 0, 0
NCLNODE: 568, 0, 0, 0, 0, 0, 0
DIRHASH: 1024, 0, 5398, 54, 5400, 0, 0
pipe: 728, 0, 4, 86, 9183, 0, 0
Mountpoints: 792, 0, 5, 10, 5, 0, 0
ksiginfo: 112, 0, 184, 872, 17776, 0, 0
itimer: 344, 0, 0, 22, 1, 0, 0
pfsrctrpl: 152, 400000, 0, 0, 0, 0, 0
pfrulepl: 936, 0, 167, 173, 462, 0, 0
pfstatepl: 288, 600002, 237842, 128498,393787401, 0, 0
pfstatekeypl: 288, 0, 330002, 174268,570051213, 0, 0
pfstateitempl: 288, 0, 330005, 174291,567485893, 0, 0
pfaltqpl: 240, 0, 0, 0, 0, 0, 0
pfpooladdrpl: 88, 0, 142, 236, 384, 0, 0
pfrktable: 1296, 10002, 33, 99, 296, 0, 0
pfrkentry: 160, 600000, 61, 179, 459, 0, 0
pfrkcounters: 64, 0, 0, 0, 0, 0, 0
pffrent: 32, 10100, 0, 909, 7457768, 0, 0
pffrag: 80, 0, 0, 540, 3321766, 0, 0
pffrcache: 80, 10035, 0, 0, 0, 0, 0
pffrcent: 24, 50022, 0, 0, 0, 0, 0
pfstatescrub: 40, 0, 400147, 200453,412251111, 0, 0
pfiaddrpl: 120, 0, 0, 0, 0, 0, 0
pfospfen: 112, 0, 700, 125, 5600, 0, 0
pfosfp: 40, 0, 410, 430, 3280, 0, 0
KNOTE: 128, 0, 8, 195, 5940, 0, 0
socket: 680, 25602, 34, 86, 10562, 0, 0
ipq: 56, 1638, 0, 0, 0, 0, 0
udp_inpcb: 392, 25600, 6, 84, 8193, 0, 0
udpcb: 16, 25704, 6, 666, 8193, 0, 0
tcp_inpcb: 392, 25600, 9, 61, 152, 0, 0
tcpcb: 976, 25600, 9, 39, 152, 0, 0
tcptw: 72, 5150, 0, 250, 28, 0, 0
syncache: 152, 1048600, 0, 125, 20, 0, 0
hostcache: 136, 15372, 0, 112, 13, 0, 0
tcpreass: 40, 8064, 0, 252, 4, 0, 0
sackhole: 32, 0, 0, 606, 1886, 0, 0
sctp_ep: 1376, 25600, 0, 0, 0, 0, 0
sctp_asoc: 2288, 40000, 0, 0, 0, 0, 0
sctp_laddr: 48, 80064, 0, 360, 175, 0, 0
sctp_raddr: 704, 80000, 0, 0, 0, 0, 0
sctp_chunk: 136, 400008, 0, 0, 0, 0, 0
sctp_readq: 104, 400032, 0, 0, 0, 0, 0
sctp_stream_msg_out: 112, 400026, 0, 0, 0, 0, 0
sctp_asconf: 40, 400008, 0, 0, 0, 0, 0
sctp_asconf_ack: 48, 400032, 0, 0, 0, 0, 0
ripcb: 392, 25600, 0, 50, 8, 0, 0
unpcb: 240, 25600, 18, 78, 2196, 0, 0
rtentry: 200, 0, 108, 63, 179, 0, 0
pfsync: 88, 0, 0, 462, 817608, 0, 0
selfd: 56, 0, 145, 737, 6054404, 0, 0
SWAPMETA: 288, 116519, 0, 0, 0, 0, 0
FFS inode: 168, 0, 119556, 212, 2262109, 0, 0
FFS1 dinode: 128, 0, 0, 0, 0, 0, 0
FFS2 dinode: 256, 0, 119556, 219, 2262109, 0, 0