Solved [Solved] PF queue rule problem

Hi, I have a FreeBSD 8.2 four-ethernet gateway with PF but I have problem with rules to send download traffic at a destination to a queue.

em0 and em1 are for internet connection called $ext1 and $ext2. em2 is a $server and is connected to the switch with all my servers. em3 is for hosts using the internet and called $lan. I want to prioritise some hosts on $ext1 and unprioritise some others (the ones that download too much), on $ext2, I only want to unprioritise some hosts (the ones that download too much).

I don't use round-robin for the two internet connections, I simply use em0 as default gw and use policy routing for a subnet to route them to em1.

pi = "pass in"
po = "pass out"

Here my queues:

Code:
altq on $lan1 hfsc bandwidth 950Mb qlimit 750 queue { voip, serveur, complus, def_ext1, depri_ext1, def_ext2, depri_ext$
   # serveur
   queue pbx        on $lan1 bandwidth 15Mb qlimit 100 priority 6 hfsc(realtime 10Mb upperlimit 98Mb red)
   queue serveur    on $lan1 bandwidth 99Mb qlimit 100 priority 2 hfsc(upperlimit 500Mb red)
   # INTERNET1
   queue pri_ext1   on $lan1 bandwidth 1Mb qlimit 100 priority 4 hfsc(upperlimit 96Mb red)
   queue def_ext1   on $lan1 bandwidth 1Mb qlimit 100 priority 2 hfsc(upperlimit 96Mb red default)
   queue unpri_ext1 on $lan1 bandwidth 1Mb qlimit 250 priority 0 hfsc(upperlimit 96Mb red)
   # INTERNET2
   queue def_ext2   on $lan1 bandwidth 1Mb qlimit 100 priority 2 hfsc(upperlimit 96Mb red)
   queue unpri_ext2 on $lan1 bandwidth 1Mb qlimit 250 priority 0 hfsc(upperlimit 96Mb red)
Here my NAT rules:

Code:
nat on $ext1 from <ipnat> to any -> $ext1
nat on $ext2 from <ipnat> to any -> $ext2

Here my policy routing rule:

Code:
$pi on $lan1 route-to ($ext2 $ext2_gw) from <internet2>

allow rule for server that come after this one will let those user acces server without matching this rule.

Now my rule to send traffic on each queue:
Code:
$po on $lan1 from any         to <unpri_ext1> queue unpri_ext1
$po on $lan1 from any         to <internet2>  queue def_ext2
$po on $lan1 from any         to <unpri_ext2> queue unpri_ext2
$po on $lan1 from any         to <pri_ext1>   queue pri_ext1
$po on $lan1 from <serveur>   to any          queue serveur
$po on $lan1 from $pbx        to any          queue pbx

The problem is that those last rules to send traffic in each queue don't seem to work. If I use pftcl -vvsq, I only see taffic in def_ext1 and a just a little traffic in def_ext2.

Code:
queue root_em3 on em3 bandwidth 950Mb priority 0 qlimit 750 {pbx, serveur, pri_ext1, def_ext1, unpri_ext1, def_ext2, unpri_ext2}
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/750 ]
  [ measured:     0.0 packets/s, 0 b/s ]
queue  pbx on em3 bandwidth 15Mb priority 6 qlimit 100 hfsc( red realtime 10Mb upperlimit 98Mb ) 
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/100 ]
  [ measured:     0.0 packets/s, 0 b/s ]
queue  serveur on em3 bandwidth 99Mb priority 2 qlimit 100 hfsc( red upperlimit 500Mb ) 
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/100 ]
  [ measured:     0.0 packets/s, 0 b/s ]
queue  pri_ext1 on em3 bandwidth 1Mb priority 4 qlimit 100 hfsc( red upperlimit 96Mb ) 
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/100 ]
  [ measured:     0.0 packets/s, 0 b/s ]
queue  def_ext1 on em3 bandwidth 1Mb priority 2 qlimit 100 hfsc( red default upperlimit 96Mb ) 
  [ pkts:     106025  bytes:   81894475  dropped pkts:      6 bytes:   3653 ]
  [ qlength:   0/100 ]
  [ measured:  6287.7 packets/s, 38.69Mb/s ]
queue  unpri_ext1 on em3 bandwidth 1Mb priority 0 qlimit 250 hfsc( red upperlimit 96Mb ) 
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/250 ]
  [ measured:     0.0 packets/s, 0 b/s ]
queue  def_ext2 on em3 bandwidth 1Mb priority 2 qlimit 100 hfsc( red upperlimit 96Mb ) 
  [ pkts:      40879  bytes:    5296149  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/100 ]
  [ measured:  2439.9 packets/s, 2.53Mb/s ]
queue  unpri_ext2 on em3 bandwidth 1Mb priority 0 qlimit 250 hfsc( red upperlimit 96Mb ) 
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/250 ]
  [ measured:     0.0 packets/s, 0 b/s ]

Even when the are around 50 to 60 Mpbs on internet2 when I look at our ISP web panel, it still shows the same result. Also, if I disable all $po rules, it's still the same result so those rules are not working but every example I have seen is using similar rules to send taffic into queues so I really don't understand where the problem is.
 
If nobody can't help me here, is there a better place to get help for PF on FreeBSD ? Maybe the mailing list? Or should I write to the OpenBSD mailing list?

Thanks
 
mike99 said:
If nobody can't help me here, is there a better place to get help for PF on FreeBSD ? Maybe the mailing list? Or should I write to the OpenBSD mailing list?

Thanks

Check what your tables really contain
[cmd=]pfctl -t yourtable -T show[/cmd]

I recommend you use the pftop tool to check in which rule the packets are falling. Consider [cmd=]pftop -v rules[/cmd]

You can add log to the pass rules, so you can see packet within the rules:
[cmd=]tcpdump -vvteni pflog0[/cmd]

I hope that helps you analyze. Don't worry. I also was struggling with PF over 2 months and still got some issues (my topic http://forums.freebsd.org/showthread.php?t=24180).
 
Hello,

Please, insert some network scheme.

If the traffic leaves your server from a different interface than where it came from, you should change the queue direction on the external interface with the opposite one.
 
Network scheme:

fiber1 = em0, fiber2 = em1, server switch = em2 and lan switch = em3.

I'm only working on shaping rules for download right now, so all my queues are on em3.
 
I finally could work on this again, read again some docs and looking at this : http://www.openbsd.org/faq/pf/fr/queueing.html, my rules are like the bob_in queue exemple for residential.

I've done some test and I think it's NAT related.

On my test unit, I have re0 as wan with 10.5.14.20/24 as ip address and re1 as lan with 10.8.1.1/24 and a laptop with 10.8.1.2/24 on eth0. If I do a simple rule like :

Code:
block out on re1 to 10.8.1.2

it will not match and I'm still able to download.

same for :

Code:
block in on re0 to 10.8.1.2

I understand why it's impossible to block in on a external if to a address behind NAT but NAT is done on the LAN so I think that LAN should be able to block out to this IP. Is this possible that it's NAT related. Is their any workaround if it's true ?
 
Now it's really strange, rule never match to my laptop IP but I can see transfer for my IP with tcpdump. I was doing test without NAT on a test unit to check which rules will match while downloading a debian iso. Here the output of pftop taken from the FreeBSD test unit:
[cmd=]pftop -v rules[/cmd]
Code:
pfTop: Up Rule 1-23/23, View: rules, Cache: 10000                                                                             15:14:51

RULE  ACTION   DIR LOG Q IF     PR        K     PKTS    BYTES   STATES   MAX INFO
   0  Pass     In      Q lo0              K        0        0        0       inet6 from any to fe80::1/128  flags S/SA
   1  Pass     In      Q lo0              K        0        0        0       inet6 from any to ::1/128  flags S/SA
   2  Pass     In      Q lo0              K        0        0        0       inet from any to 127.0.0.1/32  flags S/SA
   3  Pass     Out     Q lo0              K        0        0        0       inet6 from any to fe80::1/128  flags S/SA
   4  Pass     Out     Q lo0              K        0        0        0       inet6 from any to ::1/128  flags S/SA
   5  Pass     Out     Q lo0              K        0        0        0       inet from any to 127.0.0.1/32  flags S/SA
   6  Pass     In      Q                  K        0        0        0       inet from 192.168.3.135/32 to any  flags S/SA
   7  Pass     Out     Q                  K        0        0        0       inet from 192.168.3.135/32 to any  flags S/SA
   8  Pass     In      Q re0    tcp       K        0        0        0       from any to any port = XXXX  flags S/SA
   9  Pass     In      Q re0    udp       K     6262  8415104        2       from any to any port = olsr
  10  Pass     In      Q re1    udp       K     3230  3639104        2       from any to any port = olsr
  11  Pass     In        re0              K        0        0        0       inet from any to 10.254.200.0/24  flags S/SA
  12  Pass     Out       re0              K        0        0        0       inet from 10.254.200.0/24 to any  flags S/SA
  13  Pass     In        re1              K        0        0        0       inet from any to 10.254.200.0/24  flags S/SA
  14  Pass     Out       re1              K     2678  3597388        1       inet from 10.254.200.0/24 to any  flags S/SA
  15  Pass     In        re0              K      314    26376        0       inet from any to 10.254.201.1/32  flags S/SA
  16  Pass     Out       re0              K        0        0        0       inet from 10.254.201.1/32 to any  flags S/SA
  17  Pass     In        re1              K        0        0        0       inet from any to 10.254.201.1/32  flags S/SA
  18  Pass     Out       re1              K        0        0        0       inet from 10.254.201.1/32 to any  flags S/SA
  19  Pass     In        re0              K        0        0        0       inet from any to 10.254.201.19/32  flags S/SA
  20  Pass     Out       re0              K        0        0        0       inet from 10.254.201.19/32 to any  flags S/SA
  21  Pass     In        re1              K        0        0        0       inet from any to 10.254.201.19/32  flags S/SA
  22  Pass     Out       re1              K        0        0        0       inet from 10.254.201.19/32 to any  flags S/SA

Here a sample of the output of tcpdump taken from my FreeBSD test unit at the same time the pftop result above was running
[cmd=]tcpdump -ni re1 | grep 10.254.201.1[/cmd]
Code:
15:18:07.377870 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:07.508902 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:07.510264 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:07.510434 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:07.514277 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:07.514522 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:07.645347 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:07.655027 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:07.655253 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:07.779650 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:07.779923 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:07.789259 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:07.793156 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:07.914564 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:07.928901 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:08.055979 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:08.199142 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:08.199303 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:08.199623 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:08.333927 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:08.334227 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:08.477852 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:08.614136 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:08.614305 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:08.750711 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:08.750857 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:08.885963 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:09.022417 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:09.356163 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:09.357005 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:09.491089 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:09.491595 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:09.491991 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:09.631022 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:09.631705 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:09.774264 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:09.774509 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:09.916474 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:09.921980 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:10.051830 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:10.053716 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:10.185703 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:10.186142 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:10.325508 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:10.325801 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
15:18:10.460095 IP 130.239.18.165.80 > 10.254.201.19.49549: Flags [.], ack 1, win 54, length 1460
 
I tried with OpenBSD 4.9 and it was the same. After lot of test, I found that the logic is reverse, upload rule will shape download (looks like a behavior to me but as been the same for several years).

Exemple :

Code:
altq on $int_if hfsc bandwidth 10Mb qlimit 500 queue { main, second }
   queue main  on $int_if bandwidth 6Mb qlimit 250 priority 4 hfsc(upperlimit 10Mb default)
   queue second on $int_if bandwidth 4Mb qlimit 250 priority 0 hfsc(upperlimit 7Mb)

Code:
$pi
$po
$po on $ext_if from <second> queue second

So looks like there's a mistake on PF FAQ that I posted above if we look at bob shaping rules.
 
Back
Top