Using setfib with same gateway on both interfaces?

E

ebichu

I have 3 physical interfaces (msk0, sk0, xl0) in one FreeBSD 8.1-box which is used as my gateway with PF-NAT.
The thing I want is to isolate one of my jails to xl0. My ISP gives me 5 dynamic public IP's.
I've tried seting up multiple routes with setfib without success and I think the problem lies with the ISP giving me the same gateway on every physical interface I try with dhcp.

Is that a problem?

uname -a:
Code: 
FreeBSD xxx.xxx.xxx 8.1-RELEASE-p2 FreeBSD 8.1-RELEASE-p2 #0: Thu Jan 20 17:57:48 UTC 2011 root@xxx.xxx.xxx:/usr/src/sys/amd64/compile/lain  amd64
ifconfig:
Code: 
msk0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=c011a<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,TSO4,VLAN_HWTSO,LINKSTATE>
	ether X:X:X:X:X
	inet 10.0.1.1 netmask 0xffffff00 broadcast 10.0.1.255
	inet 10.0.1.10 netmask 0xffffffff broadcast 10.0.1.10
	inet 10.0.1.20 netmask 0xffffffff broadcast 10.0.1.20
	inet 10.0.1.30 netmask 0xffffffff broadcast 10.0.1.30
	media: Ethernet autoselect (100baseTX <full-duplex,flag0,flag1>)
	status: active
xl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=80009<RXCSUM,VLAN_MTU,LINKSTATE>
	ether X:X:X:X:X
	inet A.229.66.183 netmask 0xfffff000 broadcast A.229.79.255
	media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
sk0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE>
	ether X:X:X:X:X
	inet A.229.70.4 netmask 0xfffff000 broadcast A.229.79.255
	media: Ethernet autoselect (100baseTX <full-duplex,flag0,flag1>)
	status: active

setfib 0 netstat -rn:
Code: 
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            A.229.64.1        UGS         0  1941879    sk0
10.0.1.0/24        link#1             U          26  4392385   msk0
10.0.1.1           link#1             UHS         0        0    lo0
10.0.1.10          link#1             UHS         0        0    lo0 =>
10.0.1.10/32       link#1             U           0        0   msk0
10.0.1.20          link#1             UHS         0        0    lo0 =>
10.0.1.20/32       link#1             U           0        0   msk0
10.0.1.30          link#1             UHS         0       33    lo0 =>
10.0.1.30/32       link#1             U           0        0   msk0
A.229.64.0/20     link#3             U           0        0    sk0
A.229.66.183      link#2             UHS         0        3    lo0
A.229.70.4        link#3             UHS         0       16    lo0
127.0.0.1          link#4             UH          0     4828    lo0

setfib 1 netstat -rn
Code: 
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            A.229.64.1        UGS         0      165    xl0
10.0.1.0/24        link#1             U           0        0   msk0
10.0.1.10/32       link#1             U           0        0   msk0
10.0.1.20/32       link#1             U           0        0   msk0
10.0.1.30/32       link#1             U           0        0   msk0
A.229.64.0/20     link#3             U           0        1    sk0
127.0.0.1          link#4             UH          0        0    lo0

pf.conf
Code: 
# macros
ext_if="sk0"
int_if="msk0"
# options
set loginterface none
set block-policy drop
set state-policy if-bound
set debug none
set skip on lo0
scrub in

# Usual NAT
nat on $ext_if from !($ext_if) -> ($ext_if:0)

rdr on $ext_if proto tcp from any to ($ext_if) port smtp -> 10.0.1.1 port smtp

# filter rules
block in
pass out quick
pass in quick proto tcp from any to any port 25
pass in quick on $int_if

pass in quick on xl0 \
        reply-to ( xl0 A.229.64.1 ) \
        proto tcp from any to any port { 8010 } \
        keep state ( floating )
 
Both sk0 and xl0 are on the same subnet. Routing will not separate the traffic.
 
Oh. Do you have any clue as to why it behaves like this?
I forgot to tell how the gateway is connected to the internet.
Interface sk0 is connected directly to one of 4 ports of a VDSL2-modem (60/20Mbit)
and msk0 is connected to a 3com Officeconnect 100Mbit switch. Interface xl0 is connected directly to the modem.

Could that be the problem?


I mean if I have 5 public IP's at my disposal from my ISP the traffic ought to be isolated
if I'm connecting another computer directly to the VDSL2-modem. Otherwise it's no point having all those IP's. :)
 
ebichu said:
Oh. Do you have any clue as to why it behaves like this?
Click to expand...
Ehm, yes. It's basic TCP/IP actually :stud

But perhaps you and I have a different meaning to "isolating"? Perhaps you could explain more what you mean by that?

You normally bind a jail to a specific interface with a specific IP address. All network services on both the host and the jail are also configured to use one or more specific IP addresses.

I forgot to tell how the gateway is connected to the internet.
Interface sk0 is connected directly to one of 4 ports of a VDSL2-modem (60/20Mbit)
and msk0 is connected to a 3com Officeconnect 100Mbit switch. Interface xl0 is connected directly to the modem.

Could that be the problem?
Click to expand...
I don't really see a problem.

I mean if I have 5 public IP's at my disposal from my ISP the traffic ought to be isolated if I'm connecting another computer directly to the VDSL2-modem.
Click to expand...
No, not if all those IP addresses are on the same subnet. Certainly not if they all come from the same modem.

Otherwise it's no point having all those IP's. :)
Click to expand...
That depends on what you planned on doing with them.
 
SirDice said:
Ehm, yes. It's basic TCP/IP actually :stud

But perhaps you and I have a different meaning to "isolating"? Perhaps you could explain more what you mean by that?

You normally bind a jail to a specific interface with a specific IP address. All network services on both the host and the jail are also configured to use one or more specific IP addresses.
.
Click to expand...

Yes like that. For example if the jail is bound to xl0 and I start httpd in that specific jail both incoming and outgoing traffic should be isolated in that jail and on that interface.

I don't really see a problem.
Click to expand...
Thought so too but was unsure.


No, not if all those IP addresses are on the same subnet. Certainly not if they all come from the same modem.
Click to expand...

I just tested connecting one of my other boxes (OS X 10.6.6) to port number 2 of the VDSL2-modem and came up with the following information:

Code: 
[B]ifconfig[/B]
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
	inet 127.0.0.1 netmask 0xff000000 
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	ether X:X:X:X:X 
	inet A.226.168.184 netmask 0xfffffc00 broadcast A.226.171.255
	media: autoselect
	status: active
Code: 
[B]netstat -rn[/B]
Destination        Gateway            Flags        Refs      Use   Netif Expire
default            A.226.168.1        UGSc           46        0     en0
A.226.168/22       link#4             UCS             4        0     en0
A.226.168.1        0:d0:2:c8:7c:0     UHLWI          29        0     en0   1047
A.226.168.184      127.0.0.1          UHS             0        0     lo0
A.226.171.255      link#4             UHLWbI          3      134     en0
127                127.0.0.1          UCS             0        0     lo0
127.0.0.1          127.0.0.1          UH              0        0     lo0
169.254            link#4             UCS             0        0     en0
That depends on what you planned on doing with them.
 
ebichu said:
Yes like that. For example if the jail is bound to xl0 and I start httpd in that specific jail both incoming and outgoing traffic should be isolated in that jail and on that interface.
Click to expand...
The traffic isn't isolated. It only gets send to/from the jail's services. But you need to make sure those services are bound to the correct IP address and not to all (usually indicated by * or 0.0.0.0). That's the normal way of operating a jail, there's no need fiddle with multiple routing tables.
 
Maybe I'm using the wrong definition of "isolated"?
Just want the traffic to be seen as coming from xl0 and the dynamic public ip that I get on that interface.

Shouldn't it work with xl0 being configured to get a dynamic public ip,
putting the jail on a private ip and then putting some rdr pass rule and pass out in pf.conf?

Another possibility would be that xl0 doesn't get a public ip on a separate subnet when requested by dhclient
which perhaps prevents what I'm trying to do working...
My other box got an IP on another "correct" subnet when connected to my VDSL2-modem.

I'm sorry for my confused post...
 
You must log in or register to reply here.
Back
Top