natd very high cpu, stalls connection? please help!

I

idownes

I'm attempting to create a private network of jails running on a machine which is itself on a 10.x.x.x network (if it helps, it's an Amazon EC2 machine). I don't control anything beyond my machine. I'd like the jails to have access to the 10.x.x.x network and beyond to the internet so I'm working with nat and ipfw.

Code: 
jails                 <-- vnet -->  bridge            <...nat....>  xn0    <----------------- Amazon's network
192.168.0.x                         192.168.0.1                       dhcp 10.17.212.x

I have this working on my own 10.x.x.x test network just fine but when I try to get it working on the ec2 machine/network it is mostly broken. Natd generally has excessive cpu usage and when I try any large transfers (host) with scp or wget it jumps to 100% of a core and the connection stalls. Is this a misconfiguration or am I trying to do something silly?

I've read this recent post http://forums.freebsd.org/showthread.php?t=24019 but I don't have control of the network and I want the jails to be on a private network (I don't have 10.x.x.x addresses for them) so it doesn't really apply.

As a side note, because it's an ec2 instance it's running on Xen under HVM virtualization.

/etc/rc.conf
Code: 
ifconfig_xn0="DHCP"
gateway_enable="YES"
firewall_enable="YES"
firewall_logging="YES"
firewall_script="/etc/ipfw.rules"
natd_enable="YES"
natd_interface="xn0"
natd_flags=""

/etc/ipfw.rules
Code: 
#!/bin/sh
cmd="ipfw -q add"
pif="xn0"          # public facing nic
iif="jail_bridge" # internal private virtual network
skip="skipto 500"
good_tcp="22,43,53,80,443"
netbios_ports="137,138,139"

ipfw -q -f flush
$cmd 002 allow all from any to any via $iif  # exclude private network
$cmd 003 allow all from any to any via lo0  # exclude loopback interface

$cmd 100 divert natd ip from any to any in via $pif
$cmd 110 check-state

$cmd 200 $skip udp from any to any 53 out via $pif keep-state
#$cmd 210 $skip tcp from me to any out via $pif setup keep-state uid root
$cmd 220 $skip icmp from any to any out via $pif keep-state
$cmd 230 $skip tcp from any to any $good_tcp out via $pif setup keep-state
$cmd 240 $skip tcp from any to any 65223 out via $pif setup keep-state

# Deny all inbound traffic from non-routable reserved address spaces
$cmd 300 deny all from 192.168.0.0/16  to any in via $pif  #RFC 1918 private IP
$cmd 301 deny all from 172.16.0.0/12   to any in via $pif  #RFC 1918 private IP
#$cmd 302 deny all from 10.0.0.0/8      to any in via $pif  #RFC 1918 private IP
$cmd 303 deny all from 127.0.0.0/8     to any in via $pif  #loopback
$cmd 304 deny all from 0.0.0.0/8       to any in via $pif  #loopback
$cmd 305 deny all from 169.254.0.0/16  to any in via $pif  #DHCP auto-config
$cmd 306 deny all from 192.0.2.0/24    to any in via $pif  #reserved for docs
$cmd 307 deny all from 204.152.64.0/23 to any in via $pif  #Sun cluster
$cmd 308 deny all from 224.0.0.0/3     to any in via $pif  #Class D & E multicast

$cmd 320 deny tcp from any to any $netbios_ports in via $pif
$cmd 323 deny tcp from any to any 81  in via $pif

$cmd 360 allow udp from any to any 68 in keep-state

$cmd 370 allow tcp from any to me 80 in via $pif setup

$cmd 380 allow tcp from any to me 22 in via $pif setup limit src-addr 5

$cmd 450 deny log all from any to any via $pif

# This is the skip to location for outbound stateful rules
$cmd 500 divert natd ip from any to any out via $pif
$cmd 510 allow ip from any to any

[CMD="uname -a"][/CMD]
Code: 
 FreeBSD ip-X-X-X-X 8.2-RELEASE FreeBSD 8.2-RELEASE #1: Mon May 23 21:31:40 UTC 2011     root@ip-X-X-X-X:/usr/obj/usr/src/sys/XHVMVIMAGE  amd64

The kernel is the standard kernel from Colin Percival for Amazon HVM instances but with the following changes
Code: 
options         VIMAGE
nooptions       SCTP
 
Three things you can try:

1. ipfw's kernel NAT
2. pf
3. running multiple natd processes, and divert traffic from different jails to each.
 
aragon:
thanks for the suggestions.
1/ I'll try this one
2/ This one I've tried and it results in kernel panics since I'm using vnet/vimage. pf (and ipf) doesn't appear to have been virtualized yet (http://unix.derkeiler.com/Mailing-Lists/FreeBSD/net/2010-04/msg00048.html
3/ I'm not sure that I follow? Why do I need multiple natds?

What I'm working on now is figuring out rules so that the host's traffic isn't sent through nat, and only the 192.168.0.0/16 traffic is nat'ted. Does this make sense?
 
idownes said:
3/ I'm not sure that I follow? Why do I need multiple natds?
Click to expand...
You mentioned CPU usage is 100% on one core, so I assumed you had multiple cores at your disposal. Running multiple natd processes would allow you to take advantage of them.


idownes said:
What I'm working on now is figuring out rules so that the host's traffic isn't sent through nat, and only the 192.168.0.0/16 traffic is nat'ted. Does this make sense?
Click to expand...
You can try alter your divert rule so that only outgoing traffic from 192.168.0.0/16 gets diverted, but unfortunately all return traffic has to be diverted to natd. Unless, of course, you can get a second IP address from Amazon, and dedicate that IP to natd, and use another IP for your host.
 
Come to think of it, what happens if you try and skip natd all together and just have your jails request an address via DHCP too?
 
Ok, yes the machine has many cores but I figured that I was doing something wrong because it was at 100% and stalling.

I altered my outgoing divert rule to the following:
Code: 
$cmd 500 divert natd ip from 192.168.0.0/16 to any out via $pif
which helps immensely and gets me the expected network speed for outbound transfers But, as you mention it doesn't help incoming packets which all go through nat. Natd doesn't take any noticeable cpu but throughput inbound is only a few hundred kB/s.

Am I correct in assuming that a modern machine can do nat at LAN speeds?

I'll try requesting a DHCP address on a virtual interface in see if that works. I'd still need to fix the inbound slowness though as my jails will have lots of inbound traffic (downloads).
 
You must log in or register to reply here.
Back
Top