altq only working for bulk traffic queue (rules problem? )

wonslung · Jul 18, 2010

I've been using pf for a bit and it works great...i recently decided to try to learn to use altq.

I've got my rules set up and it seems to be working but when i check the queues, it seems only the bulk queue has any packages in it.....what am i doing wrong here?

here is my pf.conf

Code:

#------------------------------------------------------------------------
# macros
#------------------------------------------------------------------------
# interfaces
ext_if  = "em1"
int_if  = "em0"
wifi_if = "wlan0"
#protocol
icmp_types = "{ echoreq, unreach }"
#hosts
rtor = "192.168.1.60"
scp_j = "192.168.1.53"
Xbox360 = "192.168.1.22"
ssh_zone = "192.168.1.55"
#ports
Xlive_tcp = "{ http, https, 3074 }"
Xlive_udp = "{ 88, 3074 }"
rtor_ports = "{http, https }"
bittorrent = "{ 51000, 51001, 51002 }"
#nets
lan_net = "{ 192.168.1.0/24, 192.168.2.0/24 }"
priv_nets = "{ 127.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8 }"
# config
#------------------------------------------------------------------------
# options
#------------------------------------------------------------------------
# config
set block-policy drop
set loginterface $ext_if
set skip on lo0
set skip on lo1
set optimization conservative
# scrub
#scrub all reassemble tcp no-df
#scrub in all fragment reassemble
scrub out all random-id
#--------------------------------------------------
#queue
#---------------------------------------------------
### FIOS Upload = 15Mb/s (queue at 97%)
 altq on $ext_if bandwidth 14.55Mb hfsc queue { ack, dns, ssh, games, web, mail, bulk, bittor }
  queue ack        bandwidth 30% priority 8 qlimit 500 hfsc (realtime   20%)
  queue dns        bandwidth  5% priority 7 qlimit 500 hfsc (realtime    5%)
  queue ssh        bandwidth 10% priority 6 qlimit 500 hfsc (realtime   15%) {ssh_login, ssh_bulk}
   queue ssh_login bandwidth 50% priority 6 qlimit 500 hfsc
   queue ssh_bulk  bandwidth 50% priority 5 qlimit 500 hfsc
  queue games      bandwidth 10% priority 5 qlimit 500 hfsc (realtime   5%)
  queue bulk       bandwidth 20% priority 4 qlimit 500 hfsc (realtime   20% default)
  queue web        bandwidth  5% priority 3 qlimit 500 hfsc (realtime  (10%, 10000, 5%) )
  queue mail       bandwidth  5% priority 2 qlimit 500 hfsc (realtime    5%)
  queue bittor     bandwidth  1% priority 1 qlimit 500 hfsc (upperlimit 95%)

#------------------------------------------------------------------------
# redirection (and nat, too!)
#------------------------------------------------------------------------
# network address translation
#nat on egress from (self)         to any tag EGRESS -> ($ext_if:0) port 1024:65535
nat on egress from $int_if:network to any tag EGRESS -> ($ext_if:0) port 1024:65535
nat on $ext_if from $Xbox360 to any -> ($ext_if:0) static-port
nat on $ext_if from $lan_net to any -> ($ext_if)
nat on $ext_if from 10.0.0.0/24 to any -> ($ext_if)
nat-anchor "ftp-proxy/*"
rdr-anchor "ftp-proxy/*"
rdr on $int_if proto tcp from any to any port 21 -> 127.0.0.1 port 8021
rdr on $wifi_if proto tcp from any to any port 21 -> 127.0.0.1 port 8021
rdr on $ext_if proto tcp from any to ($ext_if) port $rtor_ports -> $rtor

rdr on $ext_if proto tcp from any to ($ext_if) port 50022 -> $ssh_zone port ssh

rdr on $ext_if proto tcp from any to ($ext_if) port 51000:51002 -> $rtor
rdr on $int_if proto tcp from $lan_net to ($ext_if) port $rtor_ports -> $rtor
rdr on $wifi_if proto tcp from $lan_net to ($ext_if) port $rtor_ports -> $rtor
rdr on $ext_if inet proto udp from any to ($ext_if) port $Xlive_udp tag XBOX360 -> $Xbox360
rdr on $ext_if inet proto tcp from any to ($ext_if) port $Xlive_tcp tag XBOX360 -> $Xbox360
no nat on $int_if proto tcp from $int_if to $lan_net
nat on $int_if proto tcp from $lan_net to $rtor port $rtor_ports -> $int_if 
no nat on $wifi_if proto tcp from $wifi_if to $lan_net
nat on $wifi_if proto tcp from $lan_net to $rtor port $rtor_ports -> $wifi_if
#rdr on $int_if inet proto tcp from any to any port www -> 10.0.0.1 port 3128
#rdr on $int_if inet proto tcp from any to any port 8080 -> 10.0.0.2 port www
#rdr on $wifi_if inet proto tcp from any to any port www -> 10.0.0.1 port 3128



#------------------------------------------------------------------------
# firewall policy
#------------------------------------------------------------------------
# restrictive default rules
block log all
block drop quick log on $ext_if from $priv_nets to any
anchor "ftp-proxy/*"
# anti spoofing
antispoof for { $int_if, $wifi_if, $ext_if }

pass out quick on $ext_if inet proto tcp from $rtor to any keep state queue (bittor)
pass out quick on $ext_if inet proto tcp from any to any port $bittorrent keep state queue (bittor)
pass log proto tcp from any to $rtor port $rtor_ports synproxy state queue (web, ack)
pass log proto tcp from any to $rtor port ssh synproxy state queue (ssh_bulk, ssh_login)
pass log proto tcp from any to $ssh_zone port ssh synproxy state queue (ssh_bulk, ssh_login)
pass log proto tcp from any to $rtor port 10000:10040 synproxy state queue (bittor, ack)
pass in log on $ext_if inet proto udp from any to $Xbox360 port $Xlive_udp keep state queue (games, ack) tagged XBOX360
pass in log on $ext_if inet proto tcp from any to $Xbox360 port $Xlive_tcp flags S/SAFR synproxy state queue (games, ack) tagged XBOX360
pass out log on $int_if inet proto udp from any to $Xbox360 port $Xlive_udp keep state queue (games, ack) tagged XBOX360
pass out log on $int_if inet proto tcp from any to $Xbox360 port $Xlive_tcp flags S/SAFR synproxy state queue (games, ack) tagged XBOX360
pass in log on $int_if inet proto udp  from $Xbox360 to any port $Xlive_udp keep state queue (games, ack)
pass in log on $int_if inet proto tcp  from $Xbox360 to any port $Xlive_tcp flags S/SAFR synproxy state queue (games, ack)
pass in log on $int_if inet proto tcp  from $Xbox360 to any port $Xlive_tcp flags S/SAFR synproxy state queue (games, ack)


pass quick on $int_if keep state
pass quick on $wifi_if keep state

pass inet proto icmp all icmp-type $icmp_types keep state
pass out on $ext_if inet proto udp from any to any port 33433 >< 33626 keep state

pass out on $ext_if proto tcp all modulate state flags S/SA queue (bulk, ack)
pass out on $ext_if inet proto tcp from ($ext_if) to any flags S/SA modulate state queue (bulk, ack)
#pass out on $ext_if proto { udp,icmp } all keep state

pass in on $ext_if inet proto tcp from any to any port 50022 keep state queue (ssh_bulk, ssh_login)
pass in on $int_if inet proto tcp from any to 10.0.0.1 port 3128 keep state 
pass out on $ext_if inet proto tcp from any to any port www keep state queue (web, ack)


pass out from any to any keep state

thanks for any help

(i'm especially interested in getting the queues working for torrents and ack packets....)

adri · Jul 19, 2010

If you are using FreeBSD 8.0, then altq doesn't work using the em? interfaces, due to a bug the em drivers.
Upgrade to 8.1 and the queues should work.

Adri.

DutchDaemon · Jul 19, 2010

The problem was also addresses here: http://forums.freebsd.org/showthread.php?t=6656

wonslung · Jul 19, 2010

I wasn't aware that this was the same issue.....My queues seemed to work but everything was only showing up in bulk...i thought this was due to user error, so this is due to a bug in FreeBSD (specifically the em driver)

Is there a fix for this? I'd rather not switch to FreeBSD 7.x

wonslung · Jul 19, 2010

see, i don't think my issue is the same as everyone elses....wehn i check my queues, they show up, and ev erything is hitting the bulk queue

I'm sure this is do to me not understanding how to set up altq, this si why i posted my rules to begin with.....

I'm not sure why my torrent traffic isn't hitting the bittor queue....

I have one specific ip which runs nothing but bit torrent traffic...

rtorrent is running on port 51001 and 51002 at ip 192.168.1.60

though all my traffic is showing up as the bulk queue....what did i do wrong in my rules?

here is what i see when i do pfct -vsq

Code:

queue root_em1 on em1 bandwidth 14.55Mb priority 0 {ack, dns, ssh, games, bulk, web, mail, bittor}
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
queue  ack on em1 bandwidth 4.37Mb priority 8 qlimit 500 hfsc( realtime 2.91Mb ) 
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/500 ]
queue  dns on em1 bandwidth 727.50Kb priority 7 qlimit 500 hfsc( realtime 727.50Kb ) 
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/500 ]
queue  ssh on em1 bandwidth 1.46Mb priority 6 qlimit 500 hfsc( realtime 2.18Mb ) {ssh_login, ssh_bulk}
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/500 ]
queue   ssh_login on em1 bandwidth 727.50Kb priority 6 qlimit 500 
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/500 ]
queue   ssh_bulk on em1 bandwidth 727.50Kb priority 5 qlimit 500 
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/500 ]
queue  games on em1 bandwidth 1.46Mb priority 5 qlimit 500 hfsc( realtime 727.50Kb ) 
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/500 ]
queue  bulk on em1 bandwidth 2.91Mb priority 4 qlimit 500 hfsc( default realtime 2.91Mb ) 
  [ pkts:     436525  bytes:  436163552  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/500 ]
queue  web on em1 bandwidth 727.50Kb priority 3 qlimit 500 hfsc( realtime(1.46Mb 10000 727.50Kb) ) 
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/500 ]
queue  mail on em1 bandwidth 727.50Kb priority 2 qlimit 500 hfsc( realtime 727.50Kb ) 
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/500 ]
queue  bittor on em1 bandwidth 145.50Kb qlimit 500 hfsc( upperlimit 13.82Mb ) 
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/500 ]

wonslung · Jul 21, 2010

i figured out my issue and how to fix it.

It WAS user error, i just needed to change the order of my rules.....

my new pf.conf looks like this:

Code:

#------------------------------------------------------------------------
# macros
#------------------------------------------------------------------------
# interfaces
ext_if  = "em1"
int_if  = "em0"
wifi_if = "wlan0"
#protocol
icmp_types = "{ echoreq, unreach }"
#hosts
rtor = "192.168.1.60"
scp_j = "192.168.1.53"
Xbox360 = "192.168.1.22"
ssh_zone = "192.168.1.55"
#ports
Xlive_tcp = "{ http, https, 3074 }"
Xlive_udp = "{ 88, 3074 }"
rtor_ports = "{http, https }"
bittorrent = "{ 51000, 51001, 51002 }"
#nets
lan_net = "{ 192.168.1.0/24, 192.168.2.0/24 }"
priv_nets = "{ 127.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8 }"
# config
#------------------------------------------------------------------------
# options
#------------------------------------------------------------------------
# config
set block-policy drop
set loginterface $ext_if
set skip on lo0
set skip on lo1
set optimization conservative
# scrub
#scrub all reassemble tcp no-df
#scrub in all fragment reassemble
scrub out all random-id
#--------------------------------------------------
#queue
#---------------------------------------------------
### FIOS Upload = 15Mb/s (queue at 97%)
 altq on $ext_if bandwidth 14.55Mb hfsc queue { ack, dns, ssh, games, web, mail, bulk, bittor }
  queue ack        bandwidth 30% priority 8 qlimit 500 hfsc (realtime   20%)
  queue dns        bandwidth  5% priority 7 qlimit 500 hfsc (realtime    5%)
  queue ssh        bandwidth 10% priority 6 qlimit 500 hfsc (realtime   15%) {ssh_login, ssh_bulk}
   queue ssh_login bandwidth 50% priority 6 qlimit 500 hfsc
   queue ssh_bulk  bandwidth 50% priority 5 qlimit 500 hfsc
  queue games      bandwidth 10% priority 5 qlimit 500 hfsc (realtime   5%)
  queue bulk       bandwidth 20% priority 4 qlimit 500 hfsc (realtime   20% default)
  queue web        bandwidth  5% priority 3 qlimit 500 hfsc (realtime  (10%, 10000, 5%) )
  queue mail       bandwidth  5% priority 2 qlimit 500 hfsc (realtime    5%)
  queue bittor     bandwidth  1% priority 1 qlimit 500 hfsc (upperlimit 95%)

#------------------------------------------------------------------------
# redirection (and nat, too!)
#------------------------------------------------------------------------
# network address translation
#nat on egress from (self)         to any tag EGRESS -> ($ext_if:0) port 1024:65535
nat on egress from $int_if:network to any tag EGRESS -> ($ext_if:0) port 1024:65535
nat on $ext_if from $Xbox360 to any -> ($ext_if:0) static-port
nat on $ext_if from $lan_net to any -> ($ext_if)
nat on $ext_if from 10.0.0.0/24 to any -> ($ext_if)
nat-anchor "ftp-proxy/*"
rdr-anchor "ftp-proxy/*"
rdr on $int_if proto tcp from any to any port 21 -> 127.0.0.1 port 8021
rdr on $wifi_if proto tcp from any to any port 21 -> 127.0.0.1 port 8021
rdr on $ext_if proto tcp from any to ($ext_if) port $rtor_ports -> $rtor

rdr on $ext_if proto tcp from any to ($ext_if) port 50022 -> $ssh_zone port ssh

rdr on $ext_if proto tcp from any to ($ext_if) port 51000:51002 -> $rtor
rdr on $int_if proto tcp from $lan_net to ($ext_if) port $rtor_ports -> $rtor
rdr on $wifi_if proto tcp from $lan_net to ($ext_if) port $rtor_ports -> $rtor
rdr on $ext_if inet proto udp from any to ($ext_if) port $Xlive_udp tag XBOX360 -> $Xbox360
rdr on $ext_if inet proto tcp from any to ($ext_if) port $Xlive_tcp tag XBOX360 -> $Xbox360
no nat on $int_if proto tcp from $int_if to $lan_net
nat on $int_if proto tcp from $lan_net to $rtor port $rtor_ports -> $int_if 
no nat on $wifi_if proto tcp from $wifi_if to $lan_net
nat on $wifi_if proto tcp from $lan_net to $rtor port $rtor_ports -> $wifi_if
#rdr on $int_if inet proto tcp from any to any port www -> 10.0.0.1 port 3128
#rdr on $int_if inet proto tcp from any to any port 8080 -> 10.0.0.2 port www
#rdr on $wifi_if inet proto tcp from any to any port www -> 10.0.0.1 port 3128



#------------------------------------------------------------------------
# firewall policy
#------------------------------------------------------------------------
# restrictive default rules
block log all
block drop quick log on $ext_if from $priv_nets to any
anchor "ftp-proxy/*"
# anti spoofing
antispoof for { $int_if, $wifi_if, $ext_if }
#-----------------------
#pass any to any
#------------------------
pass inet proto icmp all icmp-type $icmp_types keep state
pass quick on $int_if keep state
pass quick on $wifi_if keep state
pass log proto tcp from any to $rtor port $rtor_ports synproxy state queue (web, ack)
pass log proto tcp from any to $rtor port ssh synproxy state queue (ssh_bulk, ssh_login)
pass log proto tcp from any to $ssh_zone port ssh synproxy state queue (ssh_bulk, ssh_login)
pass log proto tcp from any to $rtor port 10000:10040 synproxy state queue (bittor, ack)
pass out from any to any keep state queue (bulk, ack)
#-------------------------
#pass ext_if in
#--------------------------
pass in on $ext_if inet proto tcp from any to any port 50022 keep state queue (ssh_bulk, ssh_login)
pass in log on $ext_if inet proto udp from any to $Xbox360 port $Xlive_udp keep state queue (games, ack) tagged XBOX360
pass in log on $ext_if inet proto tcp from any to $Xbox360 port $Xlive_tcp flags S/SAFR synproxy state queue (games, ack) tagged XBOX360
#---------------------------
#pass int_if in
#------------------------
pass in on $int_if inet proto tcp from any to 10.0.0.1 port 3128 keep state
pass in log on $int_if inet proto udp  from $Xbox360 to any port $Xlive_udp keep state queue (games, ack)
pass in log on $int_if inet proto tcp  from $Xbox360 to any port $Xlive_tcp flags S/SAFR synproxy state queue (games, ack)
pass in log on $int_if inet proto tcp  from $Xbox360 to any port $Xlive_tcp flags S/SAFR synproxy state queue (games, ack)
#----------------------------------------
#pass int_if out
#--------------------------------------
pass out log on $int_if inet proto udp from any to $Xbox360 port $Xlive_udp keep state queue (games, ack) tagged XBOX360
pass out log on $int_if inet proto tcp from any to $Xbox360 port $Xlive_tcp flags S/SAFR synproxy state queue (games, ack) tagged XBOX360

#--------------------------------------
#pass ext_if out
#---------------------------------------
pass out on $ext_if inet proto tcp from any to any port www keep state queue (web, ack)
pass out on $ext_if inet proto udp from any to any port 33433 >< 33626 keep state
pass out on $ext_if proto tcp all modulate state flags S/SA queue (bulk, ack)
pass out on $ext_if inet proto tcp from ($ext_if) to any flags S/SA modulate state queue (bulk, ack)
pass out on $ext_if proto { udp,icmp } all keep state
pass out quick on $ext_if inet proto tcp from $rtor to any keep state queue (bittor)
pass out quick on $ext_if inet proto tcp from any to any port $bittorrent keep state queue (bittor)

and even after only running for a minute or so i can see items in my ack queue FINALLY

Code:

queue root_em1 on em1 bandwidth 14.55Mb priority 0 {ack, dns, ssh, games, bulk, web, mail, bittor}
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
queue  ack on em1 bandwidth 4.37Mb priority 8 qlimit 500 hfsc( realtime 2.91Mb ) 
  [ pkts:       4184  bytes:     236088  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/500 ]
queue  dns on em1 bandwidth 727.50Kb priority 7 qlimit 500 hfsc( realtime 727.50Kb ) 
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/500 ]
queue  ssh on em1 bandwidth 1.46Mb priority 6 qlimit 500 hfsc( realtime 2.18Mb ) {ssh_login, ssh_bulk}
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/500 ]
queue   ssh_login on em1 bandwidth 727.50Kb priority 6 qlimit 500 
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/500 ]
queue   ssh_bulk on em1 bandwidth 727.50Kb priority 5 qlimit 500 
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/500 ]
queue  games on em1 bandwidth 1.46Mb priority 5 qlimit 500 hfsc( realtime 727.50Kb ) 
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/500 ]
queue  bulk on em1 bandwidth 2.91Mb priority 4 qlimit 500 hfsc( default realtime 2.91Mb ) 
  [ pkts:       2336  bytes:     398743  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/500 ]
queue  web on em1 bandwidth 727.50Kb priority 3 qlimit 500 hfsc( realtime(1.46Mb 10000 727.50Kb) ) 
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/500 ]
queue  mail on em1 bandwidth 727.50Kb priority 2 qlimit 500 hfsc( realtime 727.50Kb ) 
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/500 ]
queue  bittor on em1 bandwidth 145.50Kb qlimit 500 hfsc( upperlimit 13.82Mb ) 
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/500 ]

edit:

At second look, this seems to be working better than before, but it's still not 100%....for instance i have torrents running and the bitorrent queue isn't hitting anything....i'm guessing it's to do with my pass rules which doesn't specify an interface....i'm going to have to sit down and think of EXACTLY how to do this...

DutchDaemon · Jul 21, 2010

I'm assuming you're running bittorrent under its own user? Write rules for that user.

Code:

## rtorrent only
pass  out quick inet proto tcp all user rtorrent modulate state queue( torrent, toracks )
pass  out quick inet proto udp all user rtorrent keep state queue( torrent, toracks )
pass  in quick inet proto tcp all user rtorrent synproxy state (max-src-conn 10, max-src-conn-rate 10/5, overload <bruteforce> flush global) 
queue( torrent, toracks )
pass  in quick inet proto udp all user rtorrent keep state (max-src-conn 10, max-src-conn-rate 10/5, overload <bruteforce> flush global) 
queue( torrent, toracks )

wonslung · Jul 21, 2010

no, bit torrent is running on another machine on the network.

this is my network firewall.

altq only working for bulk traffic queue (rules problem? )

wonslung

adri

DutchDaemon

Administrator

wonslung

wonslung

wonslung

DutchDaemon

Administrator

wonslung