How to mitigate my BMC connection risk.

Phishfry

Son of Beastie

Thanks: 1,060
Messages: 3,102

#1
Now that I started a FreeBSD homelab I wonder how server people mitigate their BMC exposure.
I have seen people superglue an empty RJ45 connector to the BMC ethernet port but that doesn't seem common.
I want to tie all my BMC controllers together for local LAN usage and I am not sure which way to go.
Perhaps a VLAN or separate subnet.
How do you isolate your Broadband Managment Controller's?

OpenBMC sounds nice but that means Linux inside.
When I look at the technical steering committee it looks like the Axis of Evil.
https://github.com/openbmc/docs
IBM,Google,Facebook,Intel Microsoft.

So I understand why they are involved but you understand my immediate skepticism.
These are the last people I would trust with my security. I do understand this is a Facebook project. So expected them.

Interested to hear how you segregate your BMC.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,617
Messages: 28,158

#3
For a long time my home network was just a single, flat, network segment. I've now started to segment things more by splitting them up into different VLANs.

I'm using HP servers with dedicated ILO ports which i connect to separate management VLAN.
My SuperMicro server has its IPMI tied to a specific VLAN. That VLAN also has the management interfaces for my switch and my wireless AP. I've designated that VLAN to be my "management" network.

My firewall (small, low power PC with FreeBSD) is trunked to the switch. That allows me to do all the routing and firewalling between the different VLANs and the outside world on one device.
 

VladiBG

Well-Known Member

Thanks: 171
Messages: 435

#4
I have seen people superglue an empty RJ45 connector to the BMC ethernet port but that doesn't seem common.
If those people can't provide physical security to the server then the IPMI will be they smallest problem.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,617
Messages: 28,158

#5
As this is a homelab, how many untrusted persons would you let into your home?
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,617
Messages: 28,158

#7
My comment was too, Phishfry is setting up a homelab ;)

That's why I was wondering how many untrusted people would visit his home to warrant that much physical security (gluing ports, etc). If this was in a (shared) rack at some service provider's datacenter, then yes. I may go that far and would definitely try to prevent as much local access as possible.
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Thanks: 1,060
Messages: 3,102

#8
It looks like I have some reading to do about VLAN's. Thanks for the advice.
My firewall (small, low power PC with FreeBSD) is trunked to the switch.
I went ahead and ordered a 10GbE card for my firewall. I am going to reconfigure my network for a similar trunking arrangement.
I bought a Mellanox single port card MCX311A.

Right now I do >>Modem>>pfSense>>Cisco-SG300>>More -SG300's
Switching to:
Modem>>FreeBSD Firewall>>NetApp Switch>>Aruba
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,617
Messages: 28,158

#9
10GbE, nice. I'm still "stuck" on Gigabit. Well, stuck, it more than suffices for the things I do at home. I do have a nice HP (Aruba) 2530-48 switch. It was written off, replaced by new equipment, and given to me instead of thrown away. Sometimes I really, really love my job.

The SuperMicro (which was also a gift) is connected with 4 ports to the switch. One IPMI (default_vlan), one FreeBSD/bhyve 'management' interface (vlan10) and two, bundled (LACP) and trunked (all VLANs) for attaching VMs to.

The switch is mostly just managed by ssh(1). It does have a good looking web interface but I rarely use it, CLI is a lot faster once you know a few basic configuration settings, like setting the VLAN ID or showing some statistics.

Phones, tablets, media players, TV, game PC, Plex etc. are all on a "home" VLAN. VMs with stuff I'm playing with like Jenkins, Gitlab, Zabbix, ELK stack, and Puppet are on a "server" VLAN. I also have a "guest" VLAN tied to my guest wireless.

The FreeBSD firewall/router has several vlan(4) interfaces, one for each ID, using 10.0.X.1/24 as IP address, with X the same as the VLAN ID. Default gateway for each segment is therefor 10.0.X.1, i.e. 10.0.20.1 for VLAN 20. I'm also using that same address for DNS client settings. It's a fairly simple structure, easy enough to remember and it ties everything neatly together.
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Thanks: 1,060
Messages: 3,102

#10
I was afraid of fiber optic connections so I started with RJ45 10GbE cards but I have graduated to the big boys toys.
Sheepish to say I had never connected a fiber optic cable in my life.
I was searching all over on what the bailing latch does on the SFP module. I thought it was a retainer for the fiber connector, but alas it was a release catch for the sfp+ module itself. I can't believe how simple it was, but being new I was scared. Didn't want to hose up my new gear.
My Aruba(Pre-HP) switch was $70 dollars new-old stock. I did have to learn a new switch OS to upgrade the firmware.
My Netapp Cluster Switch is on the way and provides 16 10GbE SFP+ ports. That one is used and cost me around $110USD.
I am still using a few of the Cisco SG300's for their POE and some network cameras.

So I bought Intel X520-DA2, Chelsio T420 in both RJ45 and SFP+ and now I bought a Mellanox.
Originally I had planned on all Chelsio 10GbE but some good deals got in the way.

My rack is a Chatsworth 7' aluminum 2 post. I would like a proper cabinet but they are so bulky. Cheap but heavy.

I defiantly don't need all this gear, but I wanted to learn server gear so used seemed the best route to go.
My wallet is very light because of it. I did dabble in some SandyBridge LGA2011 boards. They seem just as fast as LGA2011-v3 but with less cores. My deluxe setup is a pair of 2650LV3 on a SM X10-DRi for 28cores/48T. Also bought X10-DRL using a pair of 2608L-v3. DDR4 killed me. I did 128GB on my deluxe setup and 64GB on the other SM.
 

Bobi B.

Well-Known Member

Thanks: 107
Messages: 253

#11
If you're paranoid about your BMC just use server without one. Basically it is a black box you're given, you have almost zero information and zero source code for. I don't think you can completely disable one by software and since it is capable of interacting with your motherboard "behind your back" (including some BMCs are capable of sharing NIC port with your host OS) you can never trust one. You told it to be in VLAN 123? You trust it to do so. My point is, that if your BMC is "properly" compromised you wouldn't know. Unless you catch it on your router. Unless it is compromised, as well. Ouch! Going this route is Conspiracy Theory.

Nevertheless, BMCs are pretty potent and handy when things go south in Friday afternoon =)
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,617
Messages: 28,158

#12
My rack is a Chatsworth 7' aluminum 2 post. I would like a proper cabinet but they are so bulky. Cheap but heavy.
My rack is custom built. A friend of mine built it for me, it fits nice underneath the desk. If you're looking for nice 19" cases, Chenbro has a lot, several of their models can fit a traditional ATX board and power supply. I have a 2U and two 4U cases. The 4U cases have 2x4 5.25" slots and I've fitted a couple of disk trays, that makes it easy to add/replace disks.

I defiantly don't need all this gear,
Yeah, I don't need all that stuff either. But it makes trying things a lot easier. So you can just muck about, learning how it works and what you can do with it.
 

ronaldlees

Aspiring Daemon

Thanks: 300
Messages: 708

#13
Phishfry : I laud your efforts to learn big server farm tech on your own time (perhaps you are eyeing new job definition at your company?) But, after the other Intel related scotches - I would be hesitant to use blob firmware driven management systems. I see that the source repos have *some* of the sources for the various BMC/IPMI stacks, but the compiler needs to extract and insert binaries at various points in the build process, from what I gleaned in a glance at one of the the repos. I imagine you've just used the pre-built images like everyone else.

It's too bad Google can't/won't force open source from vendors at the hardware level, for these particular bits ... Or would they even want to do that anymore? What is needed is an open hardware device for this, and it doesn't need to be very complicated. Someone should kickstart it.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,617
Messages: 28,158

#14
To configure IPMI on a FreeBSD server you can use sysutils/ipmitool. Make sure the ipmi(4) kernel module is loaded.

You can then configure IPMI using:
Code:
ipmitool lan set 1 ipaddr 10.0.0.20
ipmitool lan set 1 netmask 255.255.255.0
ipmitool lan set 1 defgw ipaddr 10.0.0.1
That's the most basic configuration. Your IPMI should then be accessible on http://10.0.0.20. You can check the current settings with ipmitool lan print.
 
Top