4 port nic?

Z

zader

Hi All,


Just wondering if anyone had some examples of using a multi port nic to create several or many separate networks between both bhyve vms and jails?

my problem is basically that I need several seperated networks with services on each .. but the caveat here is none of them can be directly attached .. ie I was thinking of using proxy servers between them.

the scenario looks something like this ..

port 1: (connected to a machine network, not computers more like boilers, chillers and air handlers - these relay amqp messages to a broker)
jail with pf firewall infront of
rabbitmq vm on bhyve (trying to convert this to the freebsd version in a jail) that delievres to
the database runs in a seperate jail with a mounted zvol for data

(also working on a way to add bro to filter port 1)

port 2: (connected to the user network)
apache in jail accepts db requests and queries the above db and returns the results to an engine running in another jail where its processed
then those results are turned into pretty charts, and delevered back to the apache instance.. (this is running in 3 jails)

(was thinking I could just share the /data zvol from above, but im not sure if thats wise, or create a new database jail and use zfs send/recieve to update the second copy) idk

port 3: (connected to internet gateway)
is actually a setting in the UI that allows the system access to the internet where it can update the host / jails and vms with updates, patches and container images.

port 4: (connected to a sdlan to the cloud)
runs a jail with snmp, im trying to get that to accept messages and logs from the rest of the systm and deliver them to a cloud instance.

so in total, theres a few vm's and several vms.

Q:
#1 what is the best way to use a combination of jails/vms to separate out all of this traffic?
#2 can you actually achieve physical separated traffic between port 1 and 2 with a multiple port nic? Or should I purchase 2 network cards.. or even a separate machine
#3 is it better to work towards dumping the vms and using all jails, or convert all the jails to vms's..?
#4 security and physical seperate of port 1 and the rest of the services is mission critcal.

Thanks!
 
i guess the next question is how to intergrate multiple vlans between a comibation of vms jails and the host?

is it as simple as
Code: 
ifconfig em0.10 create vlan 10 vlandev em0 inet 192.168.10.0/24
ifconfig em1.20 create vlan 20 vlandev em1 inet 192.168.20.0/24
ifconfig em2.30 create vlan 30 vlandev em2 inet 192.168.30.0/24
ifconfig em3.50 create vlan 30 vlandev em3 inet 192.168.40.0/24
then for example tap and create the vm switchs as needed on each? and mod the pf.conf to reflect routing as needed?
 
zader said:
i guess the next question is how to intergrate multiple vlans between a comibation of vms jails and the host?
Click to expand...
Probably the easiest way is to use bridge(4), each bridge is connected to a specific VLAN. Then you can bundle the VMs and/or Jails that do need to be on the same network together. I would recommend using VNET jails though, that will make it easier to tie the epair(4) interfaces of those jails to the bridge.

With sysutils/vm-bhyve this is easy to do for VMs.
Code: 
# vm switch list
NAME     TYPE      IFACE       ADDRESS  PRIVATE  MTU   VLAN  PORTS
servers  standard  vm-servers  -        no       9000  11    lagg0
public   standard  vm-public   -        no       9000  10    lagg0
The lagg0.10 and lagg0.11 interfaces don't need an IP address, they're just there to encapsulate the traffic in (or out) of a VLAN. That lagg0 interface is a LACP configuration of igb1 and igb2. Both are connected to a LACP bundle on my HP switch and the bundled port is set to pass the required VLAN tagged traffic. The actual routing between the VLANs happens on another machine I use to connect my network to the internet. It has a bundled (trunk) connection to the switch and has various vlan(4) interfaces for each of the VLANs I have. Those vlan(4) interfaces do have IP addresses, they're used as the gateway for each of the VLANs.

Code: 
 # ifconfig vm-servers
vm-servers: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        ether ba:dc:1e:aa:df:28
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap7 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 18 priority 128 path cost 2000000
        member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 13 priority 128 path cost 2000000
        member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 12 priority 128 path cost 2000000
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 11 priority 128 path cost 2000000
        member: lagg0.11 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 55
        groups: bridge vm-switch viid-d5539@
        nd6 options=1<PERFORMNUD>
The vm-public bridge looks similar but has different VMs attached to it and is connected to lagg0.10.

zader said:
and mod the pf.conf to reflect routing as needed?
Click to expand...
Firewalls are not routers. You use normal routing with everything.
 
that makes sense, thanks...

roger on the firewalls, my goal there is more for traffic shaping, filtering rules and monitoring.
 
zader said:
roger on the firewalls, my goal there is more for traffic shaping, filtering rules and monitoring.
Click to expand...
Right. Just wanted to make sure you understood the difference. On that host that connects everything I do the routing between the vlans and the internet. I have my firewall running there for NAT and to limit traffic between certain vlans. I also have a separate VLAN specifically for my guest Wifi for example. The firewall only allows it to go to the internet and you cannot connect to anything on my home network. For the firewall you just treat the vlan(4) interfaces as regular, individual, interfaces, setting rules to allow (or block) traffic in or out of them.
 
thanks, thats almost exactly what I need to do. In my case theres two issues, the first was the bridge was incorrect and the second is with my pf.conf .. Im seeing pcaps where traffic is going out one nic and and coming back in another.. where it should be all routed internally .. sounds like my real problem is that I dont have "On that host that connects everything I do the routing ... "
 
It sounds like you're trying to cram too much functionality on that single host. It seems to function as a router, a firewall and a host for various VMs and jails. I'd separate the hosting and router/firewall functionality. The router/firewall doesn't need to be a "big" box with lots of CPU power or memory. Just routing and firewall doesn't use a lot of resources.

What kind of switches does your network have? To do this properly you need at least managed layer 2 switches in order to separate the various vlans on the switches too. If you have layer 3 switches you may be able to let them handle the routing between the vlans.
 
yeah its very difficult to explain, and very complex..

the WAN part is actually a building network .. so there’s no traditional layer 2/3 switch.. in some cases it could be r232 or a number of other connector types .. at some point these are converted into some sort of ip / rj45 connector.. so at best I would get an output of that to plug into basically igb0.

so the firewall on igb0 is basically just dropping everything, except i wrote a page in a ui that allows someone to plunk in an ip of an edge device in the building network.. so the firewall matches rules for that ip .. and essentially port forwards 5672 to a vm (currently) running rabbitmq .. rabbitmq then deliveres to a jailed nginx reverse proxy.. in turn that’s sent to a jailed mongodb .. that has a zvol mounted as /data .. that lives on the host.

for this part there can be no way for mongo, rabbitmq, or the wan to be accessible .. it needs to be a 1 way door

igb2 is plugged into the normal corporate lan .. its given an ip as assigned by the client .
the users are able to go to this ip and pull up a web ui .. this ui essentially renders pretty charts and graphs ..

behind the scenes, the user is logging into a vm running nginx .. they enter a query string ... that is in turn sent to a rules processing engine (another vm) where its evaluated and checked for correctness.. assuming the query is good.. it connects to the mongodb and spits out a bunch of results back to the processing engine .. the results are processed and then returned to the user in the form of pretty charts and graphs.

so its
vm -> vm -> mongodb -> vm -> vm -> user

optional connections:
igb3 - is essentially a gateway for the host .. so the client can plug in to a port in say their dmz. .. push the update button from the ui .. and it will phone home and look for updates .. when they are done they can unplug it and be done with it.. in this case it has rules to only allow connections to the update servers

igb4 - logging network
this network is more complex, its running splunk, openvpn, snmp on a 172.16.100.0/24 network .. basically a few jails ..
the problem here is it needs access to reach into all of the networks .. it also needs to be able to get out to deliver logs to the internet .. or some other internal logging server.

I was also going to install bro in that network so that it could filter the other networks .. this part of the design is really the painful part ..


theres a bunch of questions,
like hammering out routing with the update wan and the 172.16 network .. is this the best way to do it? should i make an internal vm or use the hosts pf to route between the differant nics's ..

all that sort of stuff.

as for the actual mechanics .. the work load is very light .. at worst rabbitmq gets a pile of requests, or the db gets hammered.. ports 3 and 4 are rarely used.

hope that makes more sense..

thanks again!
 
I was thinking perhaps leaving igb2 and 3 as a rules set on the host .. then just jail everything and run the collection with cron .?

create a tap/bridge or what ever for igb0 and 1 ..but routing gets crazy when you start saying .. x can do y but not z .. and how do you aggregate everything etc
 
You must log in or register to reply here.
Back
Top