PF + ALTQ + FTP = fail? (a bug?)

Hi there,

As we all know, there is only one way to shape the bandwidth the right way: the outgoing traffic (egress). Today I got in situation with PF in which it seems to not be working the right way.
Here are my rules:
Code:
altq on em0 hfsc bandwidth 10Mb queue {std_up ftp_up sig_up}
    queue std_up bandwidth 8% hfsc (default realtime 2% ecn )
    queue ftp_up bandwidth 46% hfsc (realtime 39% ecn)
    queue sig_up bandwidth 46% hfsc (realtime 39% ecn)

altq on em1 hfsc bandwidth 10Mb queue { std_down ftp_down sig_down}
    queue std_down bandwidth 8% hfsc (default realtime 2% ecn )
    queue ftp_down bandwidth 46% hfsc (realtime 39% ecn)
    queue sig_down bandwidth 46% hfsc (realtime 39% ecn)


#FTP
nat on $ext_if from $ftp_lan to any -> $ftp_wan

#RDR

#FTP
rdr pass on $ext_if proto tcp from any to $ftp_wan port 21 -> $ftp_lan port 21
rdr pass on $ext_if proto tcp from any to $ftp_wan port 51000:65534 -> ftp_lan port 51000:65534


#RULES
block in log all
pass out
################### WAN RULES ##########################
#discard traffic from or to bogon nets on external interfaces
block drop in log quick on $ext_if from $priv_nets to any
block drop out log quick on $ext_if from any to $priv_nets
#drop any ipv6 as not implemented in our net
block drop in log quick proto ipv6


################## LAN RULES #########################
pass in on vlan1112 from vlan1112:network no state

############### Shaping

#Upload shaping pipe (external interface)
pass out on vlan1041 from $ftp_wan queue ftp_up
pass out on vlan1041 from $sig_lin_wan queue sig_up
pass out on vlan1041 from $sig_win_wan queue sig_up

#Download shaping pipe (outgoing to lan)
pass out on vlan1112 to $ftp_lan queue ftp_down no state
pass out on vlan1112 to $sig_lin_lan queue sig_down no state
pass out on vlan1112 to $sig_win_lan queue sig_down no state

Rules are as simple, as they can be (I didn't show the macros definitions, but it seems obvious, that it is not necessary).

So the problem is with the FTP rule: As soon as I try these rules, FTP outgoing traffic goes to the std_up queue. Why? Download traffic goes into the right one. If I test the sig_lin_wan rule, it goes into the right queue.

Code:
QUEUE                             BW SCH  PRIO     PKTS    BYTES   DROP_P   DROP_B QLEN BORROW SUSPEN     P/S     B/S
root_em0                         10M hfsc    0        0        0        0        0    0                     0       0
 std_up                         800K hfsc          2680  3970890       11    16654    0                    17   25235
 ftp_up                        4600K hfsc             0        0        0        0    0                     0       0
 sig_up                        4600K hfsc             0        0        0        0    0                     0       0
root_em1                         10M hfsc    0        0        0        0        0    0                     0       0
 std_down                       800K hfsc             0        0        0        0    0                     0       0
 ftp_down                      4600K hfsc          1424    79468        0        0    0                     7     431
 sig_down                      4600K hfsc             0        0        0        0    0                     0       0

And it keeps going there whatever I do.

So I decided to try to cheat it and added this rule right after my first LAN rule:
Code:
pass in on vlan1112 from $ftp_lan queue ftp_up no state
From my network experience it is the wrong way of shaping (as traffic first gets in, then it is shaped), but it is the only way I can get it working.

So is this a bug or some other specific PF and FTP problem? My FTP uses passive mode, all ports are redirected and matching passive ports defined in proftpd.conf. Also have the MasqueradeAddres in proftpd.conf matching my $ftp_wan in pf.conf

Any ideas? The OS is FreeBSD 9.0.
 
And now I marked this in pftop:

Code:
  [B]28  Pass     In        vlan11                 8242 12358775        0       inet from 192.168.20.0/24 to any[/B]
 [I] 29  Pass     Out       vlan10           K        0        0        0       inet from "ftp_wan"/32 to any  flags S/SA queue ftp_up[/I]
  30  Pass     Out       vlan10           K        0        0        0       inet from "sig_lin_wan"/32 to any  flags S/SA queue sig_up
  31  Pass     Out       vlan11                 4234   174971        0       inet from any to "ftp_lan"/32  queue ftp_down


The 29'th rule counter is not moving, like the network connection was directly established between my client and the FTP server behind this router. Weird? If I test the 30th rule, the counter moves.
 
Freebsd FreeBSD 7.x through 9.0 is using the OpenBSD 4.5 version of PF firewall. The current PF OpenBSD 5.0 version has different syntax for ftp-proxy. You have to use the Freebsd FreeBSD pf manpages for pf syntax and sample rules.
 
romeor said:
From my network experience it is the wrong way of shaping (as traffic first gets in, then it is shaped), but it is the only way I can get it working.

Shaping is always done egress, as ingress interface can't hold large buffer queue, among other things. Your rule of matching queue ingress is only marking traffic that will be shaped egress later, through that same interface, and is the right way to do it here. The actual shaping will only occur egress.
 
romeor said:
Code:
altq on em0 hfsc bandwidth 10Mb queue {std_up ftp_up sig_up}
    queue std_up bandwidth 8% hfsc (default realtime 2% ecn )
altq on em1 hfsc bandwidth 10Mb queue { std_down ftp_down sig_down}
    queue std_down bandwidth 8% hfsc (default realtime 2% ecn)

#RULES
block in log all

#Upload shaping pipe (external interface)
pass out on vlan1041 from $ftp_wan queue ftp_up

#Download shaping pipe (outgoing to lan)
pass out on vlan1112 to $ftp_lan queue ftp_down no state

Any ideas?
1. NAT is messing your rules. Using $ftp_lan instead of $ftp_wan for upload may help.
2. Match packets with ipfw and shape them with ALTQ.
3. Use packet marks/tags.
4. Use ipfw queues/pipes, any IPv4 packet that can match an ipfw rule can be sent to queues/pipes. A HFSC without "childs" can be simulated fair enough with WF2Q+ weights.
 
bbzz
Thanks for explanation about applying rules to incoming traffic. Very useful.
ecazamir
I don't think it is a NAT-related problem, as it happens only with FTP traffic, as I did mention before. Anyway, ipfw is not an option for me at the moment. Had problems with its kernel-NAT in 8.0-8.2 with port redirection (could not redirect more than 19 ports, as it started to complain about memory - the only way to fix it was source hacking - not my approach to production things, so I don't even remember the ipfw syntax anymore :( )
 
IPFW can be used together with pf. It's possible to filter/classify with ipfw while NATing and queueing with pf.

Did you try adding a queue on the wan side using $ftp_lan as source?
 
Back
Top