PF; traffic accounting

_martin

_martin

Hi,

Due to some issues with my ISP I've decided to do a traffic accounting on my server. Here's what I have trouble with:

Server is multihomed, I need to do a traffic accounting per each IP and then, in some cases, I need to do a per protocol statistics.

Basic skeleton is as follows:

Code: 
ext_if="em0"

IP-PUB-1="192.0.2.1"
IP-PUB-2="192.0.2.2"

# some normalization rules

# some nat/rdr rules 

# here I just want to account the traffic and let it be decided later what to do with it
pass in on $ext_if from any to $IP-PUB-1 label ip-pub-1-in
pass in on $ext_if from any to $IP-PUB-2 label ip-pub-2-in 

block in all

#  --< additional rules here >--

# per proto per IP 
pass in quick proto tcp from <ssh_trusted> to $IP-PUB-1 port 22 keep state label pub1-ssh-in

I understand that rules evaluated through ip-pub-1-in/ip-pub-2-in will be processed later down trough other filtering rules (unless quick keyword specified), but I thought that traffic passed trough it would be accounted. But it's not. I see evaluations, but all 0's afterward:

# pfctl -vsl
Code: 
ip-pub-1-in 635 0 0 0 0 0 0 0

Obviously, I'm missing something. :/
 
SirDice said:
Use something like net/pfflowd and an application like net-mgmt/nfsen.
Click to expand...

Hmm .. it seems that pfflowd requires

Code: 
options	pfsync

in kernel -- something I don't have in my currently running kernel (reboot of the server might not be possible right away).

Not sure if I get it correctly but it seems it tracks pakets/traffic based on rules it (PF) tracks. That's something I've problem with in the first place.
 
OK, so I went trough OpenBSD docs and found a way that suits me best. I made this configuration in LAB first, please do comment if you think it ain't the way to go.

First, my new /etc/pf.conf (LAB config):

Code: 
ext_if="em0"

IP_PUB_1="172.31.1.114"
IP_PUB_2="172.31.1.214"

anchor "pub1_in" from any to $IP_PUB_1 {
        pass in quick proto tcp from any to $IP_PUB_1 port 22 keep state label pub1_ssh_in
        pass in quick proto tcp from any to $IP_PUB_1 port 80 keep state label pub1_web_in
}

anchor "pub2_in" from any to $IP_PUB_2 {
        pass in quick proto tcp from any to $IP_PUB_2 port 22 keep state label pub2_ssh_in
        pass in quick proto tcp from any to $IP_PUB_2 port 80 keep state label pub2_web_in
}

block in all
pass out all

Check the overall traffic on the server:

# pfctl -vsr
Code: 
anchor "pub1_in" inet from any to 172.31.1.114
  [ Evaluations: 33        Packets: 937       Bytes: 95148       States: 2     ]
  [ Inserted: uid 0 pid 453 ]
anchor "pub2_in" inet from any to 172.31.1.214
  [ Evaluations: 27        Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 453 ]
block drop in all
  [ Evaluations: 27        Packets: 25        Bytes: 1252        States: 0     ]
  [ Inserted: uid 0 pid 453 ]
pass out all flags S/SA keep state
  [ Evaluations: 27        Packets: 4         Bytes: 364         States: 0     ]
  [ Inserted: uid 0 pid 453 ]

Check the per host traffic:

# pfctl -vsl -a pub1_in
Code: 
pub1_ssh_in 21 984 100418 564 50242 420 50176
pub1_web_in 19 19 1010 10 570 9 440

So this does basically what I need.
 
It's been a while since I played with pfflowd, details are a bit blurry. But I can't remember I had to do something 'special' with my rule-set to get it to collect the correct data.

Netflow is more or less a standard way to get traffic flow information.
 
Those tools seem good and all, but I don't need (or at least not necessarily) any graphs - I just need to see those numbers from shell to compare when something happens.
Well those rules help to 'catch' what's needed - on-fly processing of each and every packet would be (I guess) just too expensive.

But I celebrated too early. Though these rules help me when IP address is activated on the interface (webs, jails), it doesn't work when I want to catch VirtualBox traffic.

To elaborate: one egress interface - em0 - is bridged with vboxnet0. I just need to count traffic that comes in/out from this IP address. I can see this traffic with tcpdump on em0, I thought it is possible to count it with PF too.
 
Try setting one of those 'counting rules' on the vboxnet0 interface. It's just an interface like all the others.
 
SirDice said:
Try setting one of those 'counting rules' on the vboxnet0 interface. It's just an interface like all the others.
Click to expand...

Sorry, I forgot to mention - did that already. I've tried both variants, either:

Code: 
anchor "VM1-in" on $vbx_if from any to $IP_PUB_1

or

Code: 
anchor "VM1-in" on $ext_if from any to $IP_PUB_1

and tried one without specifying interface too. Unfortunately I don't see any traffic passing this rule.
 
Well I did couple more tests and found out that PF doesn't filter any traffic to IPs which are assigned to VirtualBox. Strange - PF has no problem filtering on bridged interfaces.

It seems that vboxnetflt kernel module might have something to do with this.
 
You must log in or register to reply here.
Back
Top