Other NVMe drive consumes more idle power compared to Linux

K

kimminss0

I'm experiencing high power consumption from NVMe SSDs on FreeBSD.

On Linux, attaching my Solidigm P44 PRO 2TB SSD consumes less than 1W in addition. However, on the same machine, FreeBSD consumes about 4W.

I've tested with APST both enabled and disabled, along with some manual power state changes, but none of these helped. What could be causing this difference?
 
First of all in general: Don't blindly trust every value shown by some tool.
If you really need to know exact values you can actually fully trust, that needs a bit effort.

However:
If the tool you measured is identical in both cases, so, presupposing the values are comparable (which doesn't neccessarily means they are true), several ideas jump into my mind:
- different OS, different filesystem, different expense of maintenence (Which FS do you run under Linux? ext4? Which under FreeBSD? ZFS? Journaling enabled?)
- different installed software runs different loads; what software packages are running under Linux, which under FreeBSD, while you measure that difference? And which of those have what access load to the FS?
- different backup jobs running? Maybe under FreeBSD on ZFS doing periodic snapshots, while not under Linux?
- on unix[like] there always is /var/ in continous use. You may get the write-read-access to it for each of your systems.
- you may check your drives read-write-access anyway on each system, and figure out what software you run on each causes how much of that.
 
I measured power consumption with a power meter (something like a Kill A Watt). I first measured power consumption without the NVMe SSD, then measured again with the SSD attached.

I used ZFS on both Linux (Proxmox) and FreeBSD. To test power usage, I installed a clean image of each OS. I also applied some tuning options for optimizing power consumption on FreeBSD according to this article.
 
Okay. If I get it right that means:
There really seems to be a difference of 3W between both systems.
The NVMe in question does not contain the system, nor is it used for /var/.
The difference in power consumption seems to be independent of installed software.
As far as I understand it: That leaves one point left to be examined:
As far as I know FreeBSD and Linux having two different kinds of ZFS resp. the according kernel modules are from different trees, dealing with it. That was the trace I would follow next.
 
I don’t think ZFS is the source of the problem. I wiped the (ZFS) filesystem on my drive to see if it would make a difference, but the power consumption did not change.
 
The OS is not on this NVMe device, correct?
The Linux and FreeBSD systems, are they the same physical device or two different devices (laptops, desktops, whatever)?
 
The OS is installed on different drive. Even I wiped all filesystem on the NVMe drive and it did not make difference. The Linux and FreeBSD systems are the same physical machine except their OS drive.
 
I tested multiple power states with APST disabled. I also manually enabled APST, but neither made any difference in power consumption. Yes, it's strange.
 
This device is external drive via USB?
If so, perhaps a difference in how USB is being handled, like Linux defaulting to a low power/port idle mode until device is accessed and FreeBSD is not?
 
Are you're sure that switching power states doesn't make any difference? I assume yes since you've seen the thread about APST, but maybe it's still worth it to double check that the device doesn't do any IO while measuring.

Linux also has another power management feature that we still don't: PCIE ASPM. Have you tried to turn it off there just to compare? It can be done by passing pcie_aspm=off to the kernel.
 
Is it device-specific or does it apply to NVMe SSDs in general? Are there any NVMe SSDs with well-supported power management in FreeBSD? If so, I'd like to purchase them.
 
I think for now there is no well-supported NVMe device in terms of power management because those power management mechanisms themselves are mostly not supported by FreeBSD. That said, some of the drives may work well OOTB nonetheless, others require a bit of a diving into the specification details and/or some patching to be able to configure them properly. No idea how to choose the right ones beforehand.

P.S. I mentioned PCIe ASPM as a possible explanation of why the situation may look normal under Linux, not as a root cause for issues with NVMe interface.
 
Again I like to throw in some other point of view into the discussion.

As exemplary (!) for sure it is to watch energy consumption, measure things, snoop for energy devourers, keep the footprint small, and also to point out FreeBSD's powermanagment could do better - no question - don't oversee to put the numbers in perspective to each other.
If I understood right we are talking here some desktop machine, right?
Let's assume there is some >=4-core CPU @ >=2.7GHz, maybe some nice 3D graphics adapter, most likely WLAN, maybe wireless keyboard, wireless mouse, maybe some Bluetooth device in use, maybe a switch, for sure at least a router, and one monitor... so, we are talking here about 3W within a system with an overall consumption of at least 150W at running idle.

Don't get me wrong. kimminss0, you found that difference, and were absolutely right to ask for its source. That is right, and for sure a good thing. Out of question. Good engineering! 👍

But as long as we're not talking some embedded, battery driven system where really every single W counts, and so the whole system would have been specified by scratch by complete other specifications, I think this discussion is more academic, then to really find actual potential for energy savings. (Well, if there was potential to improve FreeBSD's powermanagement... - looking at the sum of all machines, that really was a large cake. But that's not the point here so far.)
Maybe I'm completely mistaken (again), and kimminss0 already squeezes every single watt that can be spared. But if not, you could easily save app. 20W by just using wires instead of wireless, which mostly aims at WLAN, and bluetooth (since most users use such I simply insinuated that was the case for to bring an example.)

All I'm saying is:
kimminss0 said:
If so, I'd like to purchase them.
Click to expand...
Watch out not to lose yourself counting beans at the wrong edge, while there could be found way more saving potential at other places.
 
Actually, I’m running my home lab on a Beelink SER8 mini PC with a Ryzen 8745HS and 48 GB of RAM. It has excellent power efficiency. On Linux, it idles at 5 watts, and on FreeBSD, it idles at 6.6 watts with one NVMe SSD. If I attach another one (a different model), Linux idles around 5.x watts, and FreeBSD idles around 10.x watts—the difference is substantial!
 
kimminss0 said:
Actually, I’m running my home lab on a Beelink SER8 mini PC with a Ryzen 8745HS and 48 GB of RAM. It has excellent power efficiency. On Linux, it idles at 5 watts, and on FreeBSD, it idles at 6.6 watts with one NVMe SSD. If I attach another one (a different model), Linux idles around 5.x watts, and FreeBSD idles around 10.x watts—the difference is substantial!
Click to expand...

Well, then obviously the first SSD is a model that idles efficiently with FreeBSD.
 
For those who wonder, the first SSD is SK hynix Gold P31 1TB, and the other one is Solidigm P44 PRO 2TB. Also, I’ve heard that the P31 is known as one of the most power-efficient NVMe SSDs. Unfortunately, it is out of stock in my country.
 
You must log in or register to reply here.
Back
Top