Dual Homed Systems

Z

zealot

Situation: 2 upPlinks, 2 different ISPs and 1 machine.

Current Setup:
em0 uses mpd5 for pppoe
em1 uses static IP settings for the cable LAN; the em1 interface is DMZ'd from the router hosting the cable connection.

Problem:
When I pull up my pppoe connection I lose my route to the 10.0.0.0/24 network. Now accessing the box from the cable LAN works but if I try to say ssh the DMZ'd interface from a totaly different remote machine it does not seem to let me go, although via netstat I do see a SYNRCVD on the ssh port but the connection is never built.

Now I've tried several different ways of routing but none of it seems to work properly. It's almost as if I need to use one connection at a time or something along those lines.

I'm using 9-0RC3 with a GENERIC kernel.

I've read setfib could help but I'm a bit new to BSD's routing capabilities . (Former IPTables user)
 
In what range is the pppoe interface and what range is being used in the DMZ (I'm guessing it's 10.0.0.0/24 there)?
 
The problem is that returning IP for remote machine must be sourced from IP you are connecting to (DMZ interface). If you have default route pointing to pppoe connection that it fails.

(I think that's what you are getting at?)
 
Yes, I'm guessing the default gateway has something to do with it. But if all three interfaces are in different subnets it should still work regardless of what happens to the default gateway.

That's why I'm thinking the pppoe and DMZ interfaces share the same subnet which would cause problems.
 
The way its setup is like this

Cable Link - DMZ - em1
PPPoE Link - ng0 - em0

ifconfig

Code: 
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC>
        ether 00:15:17:30:a1:d4
        inet6 fe80::215:17ff:fe30:a1d4%em0 prefixlen 64 scopeid 0x1
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC>
        ether 00:15:17:30:a1:d5
        inet 10.0.0.100 netmask 0xffffff00 broadcast 10.0.0.255
        inet6 fe80::215:17ff:fe30:a1d5%em1 prefixlen 64 scopeid 0x2
        inet 10.0.0.110 netmask 0xffffffff broadcast 10.0.0.110
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
ng0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1492
        inet 206.248.190.* --> 206.248.154.101 netmask 0xffffffff
        inet6 fe80::215:17ff:fe30:a1d4%ng0 prefixlen 64 scopeid 0xa
        inet6 fe80::215:17ff:fe30:a1d5%ng0 prefixlen 64 scopeid 0xa
        inet6 2607:f2c0:a000:90:215:17ff:fe30:a1d4 prefixlen 64 autoconf
        inet6 2607:f2c0:a000:90:9963:53c1:d3af:874a prefixlen 64 autoconf temporary
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>


route get default

Code: 
   route to: default
destination: default
       mask: default
    gateway: gateway
  interface: ng0
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1492         1         0

netstat -nr
Code: 
Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            206.248.154.101    UGS         0      863    ng0
10.0.0.0/24        link#2             U           0       90    em1
10.0.0.100         link#2             UHS         0        2    lo0
10.0.0.110         link#2             UHS         0        0    lo0 =>
10.0.0.110/32      link#2             U           0        0    em1
127.0.0.1          link#9             UH          0        2    lo0
206.248.154.101    link#10            UH          0        0    ng0
206.248.190.181    link#10            UHS         0        0    lo0


Any other info needed let me know.
 
So let me be a bit more descriptive on this, I have a VPS I can ssh my pppoe connection properly but if I ssh my cable connection like I previously said it only shows (via netstat) SYNRCVD but as per last post no connection is built but it does seem to go threw the DMZ to em1, like I said em1 is accessible via the cable links LAN SBG6580 Modem-Router DMZ'ing em1 and doing no other port forwarding but does have a DHCP server running for other clients on the lan em1 is static
 
Your default route points to one internet connection (pppoe). You want to connect from internet (random IP) to machine's other public interface (DMZ). ssh destination is DMZ but your machine will source its packets from pppoe as that is where the default route is pointing to. In other words, the host on internet will get back SYN-ACK from local machine but they will be sourced from different IP, hence rejected.

I get this from glancing what you posted, I never tried pppoe on FreeBSD.
If this is the case than you need to set up machine to source back packets trough interface it received requests on.
 
Hmm, Sounds like I have some work ahead of me.
Any one have ideas on how to proceed with this kind of situation?
 
My friend has the same setup except his kernel locks and reboots when setting up both routes. He is using 8-2-STABLE
 
zealot said:
Hmm, Sounds like I have some work ahead of me.
Any one have ideas on how to proceed with this kind of situation?
Click to expand...

Simplest solution is to add IP range that you know you'll be connecting from internet pointing to DMZ router.

Like so (arbitrary example)

# route add A.B.0.0/16 [lan-ip-of-dmz-router]

Although I'm still not sure what is it that you are trying to do with setup.
 
Adding a route to 10.0.0.0/24 is not necessary, it's a 'directly connected' subnet and the route to it is implied (as can be seen from the netstat output).

Is there any kind of NAT running? How's that configured?
 
If I read this right the problem is that when the PPPoE link goes down there is no default route anymore and connecting to the machine from the outside using the cable connection does not work?

If that's the case you should look into the configuration of net/mpd5 and see if it's possible to run a script when the PPPoE link is torn down that would restore the default route back to what it was before the PPPoE connection was made.
 
SirDice said:
Adding a route to 10.0.0.0/24 is not necessary, it's a 'directly connected' subnet and the route to it is implied (as can be seen from the netstat output).
Click to expand...

No, not 10.0.0.0. Public route from internet used to connect to local ssh server needs to be pointed to DMZ interface on local server.
This solves reachability issues.
 
bbzz said:
Public route from internet used to connect to local ssh server needs to be pointed to DMZ interface on local server.
Click to expand...
You're not making any sense. And reading your previous post again didn't help either.

Routing tables only work for destination addresses, not source addresses.
 
Let's say IP from internet (200.200.200.200) wants to connect to his ssh server. Let's assume he has working default route pointing to pppoe interface.

For every packet going in to his DMZ interface from internet, route lookup for destination uses default route to pppoe interface. What he needs to do is point that destination to his dmz router.

route add 200.200.200.200 dmz


Can't make it any clearer than this.
 
bbzz said:
Let's say IP from internet (200.200.200.200) wants to connect to his ssh server. Let's assume he has working default route pointing to pppoe interface.

For every packet going in to his DMZ interface from internet, route lookup for destination uses default route to pppoe interface.
Click to expand...
No, it does not. It's destination is the DMZ, so it'll use the implied 10.0.0.0/24 route of the directly connected subnet.

What he needs to do is point that destination to his dmz router.

route add 200.200.200.200 dmz
Click to expand...
No, this is wrong. The 200.200.200.200 originates on the internet, it is therefor a source address, not a destination.
 
SirDice said:
No, it does not. It's destination is the DMZ, so it'll use the implied 10.0.0.0/24 route of the directly connected subnet.
Click to expand...

No not the router sitting between his machine on dmz and router...
His machine needs entry for dmz.

No, this is wrong. The 200.200.200.200 originates on the internet, it is therefor a source address, not a destination.
Click to expand...

It is a destination from the point of his ssh...

It's simple really, I have similar setup here. His ssh server needs to route back out of dmz interface, not pppoe interface.

Anyway, let's hear from OP what he did so far...
 
bbzz said:
His ssh server needs to route back out of dmz interface, not pppoe interface.
Click to expand...
Yes, but your route is not going to do that. Yours would route the reply back to the DMZ instead of the internet.
 
But, his ssh server is sitting in DMZ region, which is connected to Cable router, and then to internet. His pppoe interface is directly connected to internet.

So yeah, I don't understand you :e
 
Situation: 2 upPlinks, 2 different ISPs and 1 machine.
Click to expand...

He has only one machine that is the ssh server. If you take that into account it should be obvious why the route to 10.0.0.0/24 network is not the problem.
 
When did I say that 10.0.0.0/24 network is problem?

Yes, there is only one machine, two uplinks, and ONE CABLE LAN ROUTER sitting between his DMZ and internet.

It's the reachability to internet that is the problem. He uses default route to pppoe for all his routing on his ssh server.
 
Perhaps it's wiser to wait until the OP clarifies his setup.
Would be nice to see a network diagram because it's getting confusing really fast.
 
zealot said:
Situation: 2 upPlinks, 2 different ISPs and 1 machine.

Current Setup:
em0 uses mpd5 for pppoe
em1 uses static IP settings for the cable LAN; the em1 interface is DMZ'd from the router hosting the cable connection.

Problem:
When I pull up my pppoe connection I lose my route to the 10.0.0.0/24 network.
...
I've read setfib could help but I'm a bit new to BSD's routing capabilities . (Former IPTables user)
Click to expand...

I doubt you lose the route to 10.0.0.0/24, you are assuming that the ssh connection _reply_ packet is leaving the system through em0.

You want a multi-homed setup, so a a packet coming through em1 will be replied through the same interface where the connection was initiated, same for ng0.
I'm not sure setfib could be helpful, sshd probably can't be started easily on more than one FIB. Perhaps a two-sshd setup may work, using two different FIBs (each with a 'default route'), each sshd process listening to specific IP addresses. But this will probably work only for sshd, each extra process requiring 'multihoming' (apache, postfix, etc) will require config adjustment (listening IP address specifically).

Your problem can be solved easier with pf, which has the ability to bypass kernel's routing mechanism, look into pf man page. This method will work for any process and you won't need two or more FIBs.
Code: 
man pf.conf
   reply-to
           The reply-to option is similar to route-to, but routes packets that
           pass in the opposite direction (replies) to the specified inter-
           face.  Opposite direction is only defined in the context of a state
           entry, and reply-to is useful only in rules that create state.  It
           can be used on systems with multiple external connections to route
           all outgoing packets of a connection through the interface the
           incoming connection arrived through (symmetric routing enforce-
           ment).
 
ecazamir said:
I doubt you lose the route to 10.0.0.0/24, you are assuming that the ssh connection _reply_ packet is leaving the system through em0.

You want a multi-homed setup, so a a packet coming through em1 will be replied through the same interface where the connection was initiated, same for ng0.
I'm not sure setfib could be helpful, sshd probably can't be started easily on more than one FIB. Perhaps a two-sshd setup may work, using two different FIBs (each with a 'default route'), each sshd process listening to specific IP addresses. But this will probably work only for sshd, each extra process requiring 'multihoming' (apache, postfix, etc) will require config adjustment (listening IP address specifically).

Your problem can be solved easier with pf, which has the ability to bypass kernel's routing mechanism, look into pf man page. This method will work for any process and you won't need two or more FIBs.
Code: 
man pf.conf
   reply-to
           The reply-to option is similar to route-to, but routes packets that
           pass in the opposite direction (replies) to the specified inter-
           face.  Opposite direction is only defined in the context of a state
           entry, and reply-to is useful only in rules that create state.  It
           can be used on systems with multiple external connections to route
           all outgoing packets of a connection through the interface the
           incoming connection arrived through (symmetric routing enforce-
           ment).
Click to expand...

Yeah I was told to look into using pf for this, Since like you mentioned it has the ability to bypass the kernel routing. Thanks for the response. I'm also going to try setting the daemons to listen on specific IP addresses and see if that helps at all. For those of you who don't get the setup, it goes something like this:
Code: 
Main Up-link -[bridged/dsl modem]-[ng0 via mpd5]-> em0 
Backup Up-link -[cable modem/router]-[dmz 10.0.0.100]-> em1 Subnet 10.0.0.0/24
 
You must log in or register to reply here.
Back
Top