Multiple public network interfaces not accessable on Amazon EC2/VPC

arenoir

New Member


Messages: 5

Is it possible to have a Amazon EC2 instance attach to multiple VPC interfaces?

I currently have 3 VPC interfaces with public ip address attached to a FreeBSD 10.3-RELEASE instance. The configuration looks good and appears to be quite flexible. However I can only ping the public address of the default interface.

All interfaces are attached and the network security group is wide open. On closer inspection it looks like each interface that is magically created is setup to route through the loop back interface. I am unsure how this is intended to work but it seems to me that adding a removing interfaces should just work?

Any clues on how to make this work? I have tried adding routes to no avail.


Code:
#netstat -r
Internet:
Destination        Gateway            Flags      Netif Expire
default            172.30.0.1         UGS         xn0
localhost          link#1             UH          lo0
172.30.0.0         link#2             U           xn0
172.30.0.136       link#2             UHS         lo0
172.30.0.139       link#3             UHS         lo0
172.30.0.143       link#4             UHS         lo0

Internet6:
Destination        Gateway            Flags      Netif Expire
::                 localhost          UGRS        lo0
localhost          link#1             UH          lo0
::ffff:0.0.0.0     localhost          UGRS        lo0
fe80::             localhost          UGRS        lo0
fe80::%lo0         link#1             U           lo0
fe80::1%lo0        link#1             UHS         lo0
ff01::%lo0         localhost          U           lo0
ff02::             localhost          UGRS        lo0
ff02::%lo0         localhost          U           lo0
Code:
#ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=503<RXCSUM,TXCSUM,TSO4,LRO>
    ether 02:37:83:5f:33:13
    inet 172.30.0.136 netmask 0xffffff00 broadcast 172.30.0.255
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet manual
    status: active
xn1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=503<RXCSUM,TXCSUM,TSO4,LRO>
    ether 02:11:45:d7:f4:69
    inet 172.30.0.139 netmask 0xffffff00 broadcast 172.30.0.255
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet manual
    status: active
xn2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=503<RXCSUM,TXCSUM,TSO4,LRO>
    ether 02:93:b8:78:92:d9
    inet 172.30.0.143 netmask 0xffffff00 broadcast 172.30.0.255
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet manual
    status: active
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 7,770
Messages: 30,912

Having more than one interface in the same network is going to cause issues. If you need more than one interface for load-balancing or fail-over you should use LACP. But I'm not sure if Amazon supports that.

Handbook: 30.7. Link Aggregation and Failover
 

Murph

Well-Known Member

Reaction score: 181
Messages: 297

If you need the outgoing packets to route via a different interface based on their source address, that can't be done with a single traditional routing table, as it strictly routes based on destination address. You have to implement what Cisco would call "policy routing", or similar. Exactly how you do it depends on exactly what you are doing with those addresses and what type of firewall you use. With any firewall config (including no firewall), using multiple FIBs (routing tables) can enable per-process routing using setfib(1) (or FIB-aware code can even do it per socket). With IPFW, you can match the source address and add a setfib action. With PF, you can use route-to and reply-to options (and not bother with FIBs).

The interface addresses having a route on lo0 is perfectly normal and has nothing to do with outgoing traffic from those addresses.

If you explain the big picture, particularly why you feel you need for multiple addresses and what you intend to do with them, people may be able to offer better advice.
 
OP
OP
arenoir

arenoir

New Member


Messages: 5

Murph, I am thinking there is something missing from the ami image network configuration. If I attach 3 interfaces to an instance with the vanilla configuration I should be able to ping them.

Screen Shot 2016-07-28 at 12.40.07 PM.png


As you can see each public IP is routed to a internal ip. The instance doesn't have to concern its self with the public ip. As far as the instance is concerned it only deals with the private ips. The security groups are permissive the network devices are attached I am at a loss and think it is a bug with the ami configuration.
 

Murph

Well-Known Member

Reaction score: 181
Messages: 297

It depends on the configuration of the underlying network infrastructure. If they have a very strict implementation of BCP 38 (such as a default Cisco uRPF config), then only the first address in your type of config would be expected to work without fully configuring the host side of it. I.e. the network infrastructure config may require you to send packets through the correct interface and the network will drop sent packets which do not match the interface.

At this point, I'd probably do a quick check with tcpdump -nv -i xn1 and tcpdump -nv -i xn2 to see if the inbound packets are arriving as expected on those interfaces. If they are, then it is likely that you are simply not permitted to send those secondary addresses via xn0 (assuming that none of your security/firewall config is getting in the way).

N.B. It is largely a waste of an IP address to have a single system configured as both primary and secondary MX. It complicates mail configuration for little to no benefit.
 
OP
OP
arenoir

arenoir

New Member


Messages: 5

Murph, thanks. `tcpdump` shows packets coming in to `xn1` and out of `xn0`.

I have tried to add the following static routes.

Code:
route add 172.30.0.139/24 -iface xn1
only to return:

Code:
route: writing to routing socket: File exists
add net 172.30.0.139: gateway xn1 fib 0: route already in table
Neither pf or ipf are running. I am thinking this is a bug with the implementation of the ec2-net-utils package.

P.S I have my reasons (not related) to this issue for running the primary mx, secondary mx, imap, webmail, instances on the same machine.
 

Murph

Well-Known Member

Reaction score: 181
Messages: 297

You can't fix this through just adding routes, if my suspicion that the outbound traffic is being blocked by the infrastructure is correct. The standard/traditional Unix routing model does not allow for this type of configuration. You have to either use multiple FIBs or a firewall config which forcibly overrides the normal routing decisions. Without that special handling, all outbound traffic will always be through a single interface, and the source address of the outgoing packets will be completely ignored for routing decisions.
 

kpa

Beastie's Twin

Reaction score: 1,809
Messages: 6,318

To add to that, on FreeBSD you can't redirect locally originating outgoing traffic using a packet filter such as pf(4). You need FIBs for that. If your FreeBSD system was the router/firewall for a separate LAN network you could do what is known as "policy routing" by matching traffic incoming on the LAN interface and instructing PF with the route-to keyword to take a different routing decision for the traffic matched.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 7,770
Messages: 30,912

I am thinking this is a bug with the implementation of the ec2-net-utils package.
It's not. It's entirely expected behavior. You will get the exact same result using physical hardware.
 

Russell Rinaldi

New Member

Reaction score: 5
Messages: 3

I use the FreeBSD AMI everywhere I can on AWS and haven't seen any problems. If you're looking to run an application on different addresses, then aliasing the interface works. I think the problem here is that the address of the secondary nic is in the same subnet as the primary which is difficult routing since with both nics live on the same host i.e.:
Code:
xn0 inet 172.30.0.136 netmask 0xffffff00 broadcast 172.30.0.255
xn1 inet 172.30.0.139 netmask 0xffffff00 broadcast 172.30.0.255
The netmask/broadcast are identical. You can attempt to prioritize interfaces, but... If it's a case of nested vm or back-end database involving multiple hosts...etc, then provision the primary interfaces of the hosts in one subnet, and the secondary on a different one. Example: If using the default VPC, something like 172.31.0.0/16, sliced into 3 subnets (172.31.0.0/20, 172.31.16.0/20 and 172.31.32.0/20), put the all the primary interfaces in 172.31.16.0/20, and the secondary interfaces in 172.31.32.0/20. Or you could create a custom VPC and slice up the network as you like. In either case the primaries won't be able to talk to the secondaries and vise-versa. The only caveat being the need to tailor the security group(s) to the subnets.
 
Top