Some questions about FreeBSD-SA-19:07.mds

zirias@

Developer
I'm having a few doubts here. First about the microcode:
Ensure that the BIOS update or devcpu-data package is dated after 2014-05-14.
Is this a typo? Or does it mean there is no microcode update addressing this specific problem yet, but the mitigation needs some instruction introduced much earlier?

Mitigation Configuration

Systems with users, processes, or virtual machines in different trust
domains should disable Hyper-Threading
[...]

To activate the MDS mitigation set the hw.mds_disable sysctl.
[...]
What exactly does this mitigation do? Why is there the recommendation to still disable HT? Does that mean the mitigation is known not to address all possible problems or does it mean it's just unknown whether there's still a risk left with HT?
 
Does this help?
Not so much, unfortunately.

Without knowing what FreeBSD's mitigation does, it's hard to assess whether you should disable HT. Does FreeBSD's mitigation somehow classify processes in "security domains"? If so, how?

For example, I have a virtual machine with Windows 7 running on my server, it's for testing purposes and only has internet access -- no way to reach anything in my local network. So, if this machine catches some malware, it's not a big deal -- once noticed, just reset it to an earlier ZFS snapshot. Now, without disabling HT, do I have to worry that malware on this machine could somehow spy on processes running on my FreeBSD host? If for example the mitigation would make sure that bhyve processes will never share a core with processes of the host or of other bhyve VMs, I'd say it's fine for me to keep HT enabled. But is this the case?
 
Without knowing what FreeBSD's mitigation does, it's hard to assess whether you should disable HT. Does FreeBSD's mitigation somehow classify processes in "security domains"? If so, how?
That question is probably best asked on the freebsd-security mailing list. I'm not sure if we have forum members with that level of intimate knowledge.
 
Not so much, unfortunately.

Without knowing what FreeBSD's mitigation does

Looking at the code, it does bring in some assembler code differenciated for various CPU types from intel core gen2 upwards, and/or some use of an assember CPU instruction verw and a sysctl(8) switch hw.mds_disable.

Concerning that instruction, I could find this snippet:
The Intel CPU instruction called "VERW" is enhanced by the new CPU Microcode such that it flushes all buffers and ports. The VERW instruction will be called during task switch or VM switch by the patched kernels and hypervisors.
This explanation comes from some Linux, but one might assume it's the same here.

, it's hard to assess whether you should disable HT.

It is indeed, but that gets someway clear from the description of the thing in some of the papers floating around:
They say there are basically two different attack methods. One is to obtain unaccessible data when a thread transites from kernel to userland (probably difficult to implement such an attack), the other is that two threads are running on the same core in hyperthreading mode, and one obtains data owned by the other (the more common and demonstrated way).
It further says that cleaning the microarch structures does not fully help against the second scheme, because this cleaning happens within one thread, while the other could still obtain something.
Therefore they recommend to swich off HT in cases where a core could be shared by hostile threads.

Does FreeBSD's mitigation somehow classify processes in "security domains"? If so, how?

Sorry, I have no idea.

For example, I have a virtual machine with Windows 7 running on my server, it's for testing purposes and only has internet access -- no way to reach anything in my local network. So, if this machine catches some malware, it's not a big deal -- once noticed, just reset it to an earlier ZFS snapshot. Now, without disabling HT, do I have to worry that malware on this machine could somehow spy on processes running on my FreeBSD host? If for example the mitigation would make sure that bhyve processes will never share a core with processes of the host or of other bhyve VMs, I'd say it's fine for me to keep HT enabled. But is this the case?

Wouldn't it be possible to dedicate or reserve specific cores (and only complete cores) to each installation? On FreeBSD this could be done with cpuset(1) (I don't touch windows).
 
It further says that cleaning the microarch structures does not fully help against the second scheme, because this cleaning happens within one thread, while the other could still obtain something.
Therefore they recommend to swich off HT in cases where a core could be shared by hostile threads.
Thanks, this already explains a lot. So probably, FreeBSD's mitigation just makes sure to issue a VERW at appropriate times, but doesn't change scheduling so that bhyve and host processes don't share the same core...
That's bad news for me, and I guess I'll have to live with the risk here.
Wouldn't it be possible to dedicate or reserve specific cores (and only complete cores) to each installation?
Just pinning bhyve to a specific core wouldn't help, I'd also have to make sure any other process doesn't use that core. As my server only has 4 cores, doing that (possibly with several virtual machines) would be even worse than disabling HT. A good solution for my specific usecase would be if the scheduler would block the core as long as a bhyve process is active on it...
 
Just hit this problem system was really unstable random memory corruption random segfaults
because pkg repo does not have up-to-date devcpu-data (v1.21) had to build from ports (v1.22)
should have realised somthing was wrong when "sysctl hw.mds_disable_state" showed "Software ..." and not "VERW"
 
In mds advisory they say:

Code:
Perform one of the following:

Update CPU microcode, upgrade your vulnerable system to a supported FreeBSD
stable or release / security branch (releng) dated after the correction date,
evaluate mitigation and Hyper Threading controls, and reboot the system.

So we either need to use updated microcode or upgrade to 12.0-RELEASE-p5?

I just upgrade to p5 and I see:

Code:
sysctl hw.mds_disable_state
hw.mds_disable_state: inactive

So it's disabled by default and we have to set hw.mds_disable_state during boot.

They recommend to use hw.mds_disable=3 to autoselect if the microcode or the code will be used.

Is the microcode update better for performance?
 
In mds advisory they say:

Code:
Perform one of the following:

Update CPU microcode, upgrade your vulnerable system to a supported FreeBSD
stable or release / security branch (releng) dated after the correction date,
evaluate mitigation and Hyper Threading controls, and reboot the system.

So we either need to use updated microcode or upgrade to 12.0-RELEASE-p5?

NO! Both!

This advisory is horrible. It uses a terrible language, a creepy formatting, and needs a lawyer to understand what it says. :(

The stance "Perform one of the following" does not apply to the directly following paragraph!
It does apply to the numbered list that follows much later.

So it's disabled by default and we have to set hw.mds_disable_state during boot.

They recommend to use hw.mds_disable=3 to autoselect if the microcode or the code will be used.

Yepp.

Is the microcode update better for performance?

I might suppose so, but I don't know for certain.

I have set
Code:
pmc@disp:501:1~$ grep mds /etc/sysctl.conf 
hw.mds_disable=3
pmc@disp:502:1~$

and now it shows

Code:
pmc@disp:504:1~$ /sbin/sysctl -a | grep mds_
hw.mds_disable: 3
hw.mds_disable_state: software IvyBridge

I have not installed the microcode - if there is any for IvyBridge - because nobody did yet bother to put it into the 2019Q2 ports branch.
 
I'm still confused by the whole thing.

Are they saying we should try to get a firmware upgrade first, and then maybe we don't need to make these tweaks? How do I even go about figuring out which processor I've got in each machine? I've updated the BIOS on many computers before, but the vendor tools all required Windows.

Code:
Systems with users, processes, or virtual machines in different trust
domains should disable Hyper-Threading by setting the
machdep.hyperthreading_allowed tunable to 0:
What's a "different trust domain" in this context? If I don't have those sorts of things, does this mean I can ignore the mitigation? Does the mitigation significantly impact performance?

When did you guys start experiencing hardware problems? After the regular freebsd-update, or after applying the "Mitigation Configuration"?

Given that these instructions are usually extremely explicit, this one seems awfully muddy.
 
Are they saying we should try to get a firmware upgrade first, and then maybe we don't need to make these tweaks?
No. The firmware upgrade enables a "better" tweak using a new CPU instruction introduced in the microcode (VERW). So you should do both. Full protection means disabling HT altogether :(
How do I even go about figuring out which processor I've got in each machine?
sysctl hw.model
 
Is this a significant performance hit?
Depending on your workload, you'll loose almost 50%. With HT, you can have two jobs on each core.

My suggestion would be:
  • On a desktop, don't bother and don't enable any mitigation. There are attack vectors much more likely than this processor bug.
  • On a server, it really depends on the scenario. What's critical is running virtual machines with "untrusted" software*. These might spy on processes outside the vm. Setting hw.mds_disable to 1 or 3 is a partial mitigation and costs performance (I have no figures here, might be around 30%). Disabling HT altogether is the full mitigation, but -- see above.
* of course, just having processes with vastly different trust levels is already a risk, you don't want some malware to spy on your web browser, for example... how likely and how grave this is is up to you to assess.
 
When did you guys start experiencing hardware problems? After the regular freebsd-update, or after applying the "Mitigation Configuration"?

There aren't; that was a troll.

Concerning performance hits - my CPU cannot do hyperthreading, and I didn't measure/compare speed. In any case, the thing can be switched off again.
 
OK, I've read through this thread and some links a few more times and I feel like I understand a tiny bit more.

The root of the problem is processes being able to see what's going on in other processes (I think). And specifically that processes inside a virtual machine would be able to see what's going on outside the sandbox, which is very bad.

But what about if all activity on the server is taking place in the same space (ie, no virtual machines)? Is that better or worse or makes no difference?

All of this seems to require access to an account that can run processes, and if a bad guy's got that, we're already in trouble. I suppose we could have someone who should have access spying on stuff he shouldn't be able to see, but I'm fairly comfortable that none of our users would be able to figure something like that out, even if they were willing.

But let's say the server is running apache. Are we talking about outside web users being able to exploit this bug without some other ability to exploit apache? That is, can anyone who can't otherwise gain privileges exploit this bug?

If I'm going to cut efficiency in half or even by a third, I need a pretty good reason.

It does say you should do the freebsd-update before mitigation, so does that on its own actually improve the situation, or does it only give you the tools to "mitigate" and without doing so, you're no safer than before the update?

Sorry to be so newbish. I think my brain's gotten gotten lazy after years of "to get your system updated, just type in these exact magic words and you'll be fine".
 
I've been seeing segmentation faults (error 11) on cloud-based systems where I've enabled the mitigations. I'll say that it happens "under load" because it does not seem to happen on systems with light activity. I've not seen any definitive guidance from the FreeBSD Project, Intel, or cloud vendors as to whether one can rely on the host system to implement the mitigations (meaning that they can be left disabled on guest systems). Because crashes of mission-critical processes (I'm focused on the nginx, mariadb/mysql, php stack) are unacceptable, it seems like there's not much choice but to run without the mitigations on production cloud systems.

I'm sure someone knows better than I do, and I'm eager to hear from them.
 
Back
Top