FreeBSD 50% higher power consumption than Debian

[weberjn]

That said, in the UK, running a Pi 3 (around 2-3 watts) 24/7, costs me under £5 a year. So in many ways it isn't worth the effort if there is only ~3 watts difference between FreeBSD and Linux. Even though for laptops it can make the difference between feeling warm.

For the i3 M710Q tiny, the 2.2 watts difference between FreeBSD and Debian make 19.2 kWh p.a., or 5.6€ here in Germany, nothing to fret about for a hobbyist.

But if there is indeed a similar 30% difference in a server farm, it makes all the difference for corporate bean counters.

And for a laptop it is not about a little more warmth, but about time away from the cord.

 

[cracauer@]

No, I have got no access to any, only user accounts at work.

I'd hope somebody has and can measure.

My dual Xeon v4 system takes 100 W idle under Linux and 150 W under FreeBSD.

 

[smithi]

For the i3 M710Q tiny, the 2.2 watts difference between FreeBSD and Debian make 19.2 kWh p.a., or 5.6€ here in Germany, nothing to fret about for a hobbyist.

Indeed ... how many takeaway coffees does 5.6€ buy?

But if there is indeed a similar 30% difference in a server farm, it makes all the difference for corporate bean counters.

If your server farm isn't running at least moderate load averages or servers are really idle for long, you're running too many servers.

And for a laptop it is not about a little more warmth, but about time away from the cord.

Exactly, which is why I really only care about power use on battery, with data from acpiconf -i0

In the end, comparing energy use of small and mobile home systems with larger business equipment makes little sense.

 

[Crivens]

Don't forget that powerd/powerdxx has different profiles for AC/BAT. Measuring the power on AC does little for battery runtime.
Also, my home server when idle consumes exactly 0W, because it is not plugged in then.

 

[weberjn]

How can it happen, that an OS needs more energy than another?

1. it does more useful work and therefore needs more CPU cycles
2. it does more unneccessary work, like busy waiting, and therefore needs more CPU cycles
   1. an implementation is inefficient and needs more CPU cycles. This leaves less CPU for applications.
3. it doesn't order the CPU to go to sleep enough or keeps the CPU from going to sleep
4. it doesn't switch off other hardware, that is currently not needed, like the GPU
5. ???

 

[cracauer@]

How can it happen, that an OS needs more energy than another?

1. it does more useful work and therefore needs more CPU cycles
2. it does more unneccessary work, like busy waiting, and therefore needs more CPU cycles
   1. an implementation is inefficient and needs more CPU cycles. This leaves less CPU for applications.
3. it doesn't order the CPU to go to sleep enough or keeps the CPU from going to sleep
4. it doesn't switch off other hardware, that is currently not needed, like the GPU
5. ???

In my case the server just doesn't reach the same C-states as under Linux.

It doesn't help that i7z apparently misdisplays C-states. (and that there is no i7z for AMD)

 

[weberjn]

In my case the server just doesn't reach the same C-states as under Linux.

It doesn't help that i7z apparently misdisplays C-states. (and that there is no i7z for AMD)

Thanks for the i7z hint. On the idling i3 M710Q tiny under FreeBSD it is

Socket [0] - [physical cores=2, logical cores=4, max online cores ever=2]
  TURBO ENABLED on 2 Cores, Hyper Threading ON
  Max Frequency without considering Turbo 3290.72 MHz (99.72 x [33])
  Max TURBO Multiplier (if Enabled) with 1/2/3/4 Cores is  32x/32x/32x/32x
  Real Current Frequency 1338.51 MHz [99.72 x 13.42] (Max of below)
        Core [core-id]  :Actual Freq (Mult.)      C0%   Halt(C1)%  C3 %   C6 %  Temp      VCore
        Core 1 [0]:       1226.56 (12.30x)         1     100       0       0    37      0.8335
        Core 2 [1]:       1338.51 (13.42x)         1     100       0       0    37      0.8333

Edit: Debian output. Freq is lower.

Socket [0] - [physical cores=2, logical cores=4, max online cores ever=2]
  TURBO ENABLED on 2 Cores, Hyper Threading ON
  Max Frequency without considering Turbo 3290.72 MHz (99.72 x [33])
  Max TURBO Multiplier (if Enabled) with 1/2/3/4 Cores is  32x/32x/32x/32x
  Real Current Frequency 870.39 MHz [99.72 x 8.73] (Max of below)
        Core [core-id]  :Actual Freq (Mult.)      C0%   Halt(C1)%  C3 %   C6 %  Temp      VCore
        Core 1 [0]:       870.39 (8.73x)           1     100       0       0    32      0.8345
        Core 2 [1]:       855.38 (8.58x)           1    99.9       0       0    32      0.8312

 

[sko]

Especially for a professional server OS low power consumption is an important competitive advantage, at least here in Europe where energy is expensive.
Energy consumption is for a server farm a cost factor as gas consumption is for a fleet of trucks.

Given that the typical linux-stack uses multiple layers of abstraction and bloatware, FreeBSD should be fine in that regard...

 

[cracauer@]

Thanks for the i7z hint. On the idling i3 M710Q tiny under FreeBSD it is

Socket [0] - [physical cores=2, logical cores=4, max online cores ever=2]
  TURBO ENABLED on 2 Cores, Hyper Threading ON
  Max Frequency without considering Turbo 3290.72 MHz (99.72 x [33])
  Max TURBO Multiplier (if Enabled) with 1/2/3/4 Cores is  32x/32x/32x/32x
  Real Current Frequency 1338.51 MHz [99.72 x 13.42] (Max of below)
        Core [core-id]  :Actual Freq (Mult.)      C0%   Halt(C1)%  C3 %   C6 %  Temp      VCore
        Core 1 [0]:       1226.56 (12.30x)         1     100       0       0    37      0.8335
        Core 2 [1]:       1338.51 (13.42x)         1     100       0       0    37      0.8333

Yeah, C1 will not cut it. Linux can often reach C6.

As I said, i7z seems to sometimes not display C-states correctly. Cross-check with sysutils/intel-pcm

 

[kpedersen]

How can it happen, that an OS needs more energy than another?

Linux kernel modules expose more "tunables" than FreeBSD which in turn exposes more than OpenBSD.

If you look at some of the tweaks that powertop --auto-tune does, it is substantial (a good 90% of them are safe to do).

But if there is indeed a similar 30% difference in a server farm, it makes all the difference for corporate bean counters.

Indeed. Though the correct solution is to consolidate on running less servers with higher utilization. Weirdly the bean counters don't always get that though. IBM a while ago calculated that most of their servers were usually only around 40% utilized.

And for a laptop it is not about a little more warmth, but about time away from the cord.

Whether a laptop can do 8 hours with FreeBSD or 12 hours with Linux, I find you get diminishing return. You will never be on it that long and you will generally charge it overnight anyway. If you are in the wilderness, so unable to regularly charge it, then a laptop is possibly the wrong tool for the job.

 

[smithi]

From several posts, we're still comparing apples with oranges ( or raspberries :-)

On the Pi 4 it's different:

FreeBSD: 2,7W
Raspbian Bullseye: 2,8W

Nice, but hardly comparable with systems with their own:
. powersupply regulator/charger
. onboard keyboard, wifi ...
. onboard screen (especially)
. fan/s

and yet another one: Lenovo ThinkCentre M710Q, i3-6100T

Debian 12: 6 Watt
FreeBSD 13.2: 8.2 Watt

This one has hwpstate_intel0: <Intel Speed Shift>
so powerd is useless, I guess.

So far. I expect someone will make it and/or powerdxx work with Speed Shift eventually.

What tuning, especially [edit] cx_lowest, on this one? freebsd_version?

8.2W sounds good, but without a screen, which is usually a good fraction of total power use.

Thanks for the i7z hint. On the idling i3 M710Q tiny under FreeBSD it is

Socket [0] - [physical cores=2, logical cores=4, max online cores ever=2]  [...]
  Real Current Frequency 1338.51 MHz [99.72 x 13.42] (Max of below)
        Core [core-id]  :Actual Freq (Mult.)      C0%   Halt(C1)%  C3 %   C6 %  Temp      VCore
        Core 1 [0]:       1226.56 (12.30x)         1     100       0       0    37      0.8335
        Core 2 [1]:       1338.51 (13.42x)         1     100       0       0    37      0.8333

Do temps from coretemp match those? Pretty cool, literally.

Yeah, C1 will not cut it. Linux can often reach C6.

As I said, i7z seems to sometimes not display C-states correctly. Cross-check with sysutils/intel-pcm

I see C6 (C7 on battery) in i7z (ie C2 and C3 in sysctls) on my older i5 with cx_lowest=Cmax, but of course only C1 without, and that's without Speed Shift.

 

[failederror]

Recently, i had the chance to get my hands on a workstation with 14 cores 64GB RAM, 4 nvme drives, 1x256GB SSD, 1x2TB HDD and a 4 port NIC. Measured the power consumption using a lab calibrated wattmeter as follows:

• FreeBSD 13.2 Baremetal fresh install, chose powerd during installation step: 70 Watt
• OPNsense 24.1(based on FREEBSD 13.2). Baremetal fresh install: 70 Watt
• Proxmox 8.1(based on Debian 12.2). Baremetal fresh install: 46 Watt
• OPNsense 24.1 as a VM on Proxmox 8.1(VM config: 10 vCPUs, 10GB RAM, 32GB disk). Fresh install: 49 Watt

The difference of 30 watt between FreeBSD and Debian is significant and not sure what can be done to bring its consumption low.

 

[Crivens]

Power saving things are not active by default. That is your job as an admin. What can be done about it is reading up on things and activating what you need and want.

 

[kpedersen]

The difference of 30 watt between FreeBSD and Debian is significant and not sure what can be done to bring its consumption low.

Following on from Crivens' advice. I would start here:

https://wiki.freebsd.org/TuningPowerConsumption

and here:

https://docs.freebsd.org/en/books/handbook/config/index.html#acpi-overview

FreeBSD possibly isn't *quite* as flexible as Linux when it comes to reducing power (neither is Windows or macOS) but you can almost half the power draw on many machines.

 

[cracauer@]

Recently, i had the chance to get my hands on a workstation with 14 cores 64GB RAM, 4 nvme drives, 1x256GB SSD, 1x2TB HDD and a 4 port NIC. Measured the power consumption using a lab calibrated wattmeter as follows:

• FreeBSD 13.2 Baremetal fresh install, chose powerd during installation step: 70 Watt
• OPNsense 24.1(based on FREEBSD 13.2). Baremetal fresh install: 70 Watt
• Proxmox 8.1(based on Debian 12.2). Baremetal fresh install: 46 Watt
• OPNsense 24.1 as a VM on Proxmox 8.1(VM config: 10 vCPUs, 10GB RAM, 32GB disk). Fresh install: 49 Watt

The difference of 30 watt between FreeBSD and Debian is significant and not sure what can be done to bring its consumption low.

That approximately mirrors what I got with a SMP Xeon v4. I went down all the power saving advise but nothing helped.

 

[smithi]

Recently, i had the chance to get my hands on a workstation with 14 cores 64GB RAM, 4 nvme drives, 1x256GB SSD, 1x2TB HDD and a 4 port NIC. Measured the power consumption using a lab calibrated wattmeter as follows:

• FreeBSD 13.2 Baremetal fresh install, chose powerd during installation step: 70 Watt

Does sysctl dev.hwpstate_intel say anything on this system?

 

[weberjn]

Seems people with deep knowledge in power savings configuration do not have a big system at hand to compare and tune FreeBSD and Debian, do they?

 

[kpedersen]

Seems people with deep knowledge in power savings configuration do not have a big system at hand to compare and tune FreeBSD and Debian, do they?

Likely when you are as big as Netflix, saving a little bit of power per (utilized) machine is too trivial to think about when using FreeBSD.

They tend to only care about performance because they don't have computers laying around idling. FreeBSD at 100% utilisation will draw the same power as Debian at 100% utilisation. If you are running loads of servers at less than 100% utilisation, the premise is a little wrong and you should scale down the number of servers.

The kind of idle power management you are discussing is more suited to the home desktop PC. You think the IBM Z-series has better power management than FreeBSD? Those are "big systems".

 

[gpatrick]

Likely when you are as big as Netflix, saving a little bit of power per (utilized) machine is too trivial to think about when using FreeBSD.

They tend to only care about performance because they don't have computers laying around idling. FreeBSD at 100% utilisation will draw the same power as Debian at 100% utilisation. If you are running loads of servers at less than 100% utilisation, the premise is a little wrong and you should scale down the number of servers.

The kind of idle power management you are discussing is more suited to the home desktop PC. You think the IBM Z-series has better power management than FreeBSD? Those are "big systems".

Brendan Gregg, a former Sun Microsystems kernel engineer, former performance engineer at Joyent, former senior performance engineer at Netlix, and now performance engineer at Intel, has talked about this.

I'd have to dig up his past comments, but performance tuning at the OS leads to little performance gain; I want to say 5%. However, he has also said that he and the other performance engineers at Netflix have saved the company A LOT of money. This is what kpedersen is mentioning. When you are the size of Netflix, Facebook, or Google, every little bit of performance matters because it saves them server scaling, memory hardware, and network bandwidth, all of which become enormously expensive given their scale of operations.

If you really desire to learn about performance, read Brendan Gregg's books or his blog. A lot is based on Linux, but there is some on FreeBSD since it is used at Netflix.

Best performance engineers I can think of are Brendan Gregg, Richard McDougall, Jim Mauro, Adrian Cockcroft, Richard Pettit. All of them worked for Sun Microsystems, too.

 

[Champion]

I had the same problem on a Sunblade 150. It actually was due to the filesystem.
FreeBSD was using the ZFS filesystem, which requires more caching [power consumption] than Debain's ext4 filesystem.

I'm willing to bet that FreeBSD with UFS operates more similarly to Debain with ext4 and FreeBSD with ZFS operates more similarly to Debian with a ZFS equivalent [openZFS or maybe Btrfs].

 

[cracauer@]

I had the same problem on a Sunblade 150. It actually was due to the filesystem.
FreeBSD was using the ZFS filesystem, which requires more caching [power consumption] than Debain's ext4 filesystem.

I'm willing to bet that FreeBSD with UFS operates more similarly to Debain with ext4 and FreeBSD with ZFS operates more similarly to Debian with a ZFS equivalent [openZFS or maybe Btrfs].

My measurements were with both Linux and FreeBSD booted diskless (PXE + NFS).

 

[sko]

I had the same problem on a Sunblade 150. It actually was due to the filesystem.
FreeBSD was using the ZFS filesystem, which requires more caching [power consumption] than Debain's ext4 filesystem.

I'm willing to bet that FreeBSD with UFS operates more similarly to Debain with ext4 and FreeBSD with ZFS operates more similarly to Debian with a ZFS equivalent [openZFS or maybe Btrfs].

I heard a lot of urban myths about ZFS, but this one is new....
Actually, RAM mostly doesn't care if it is under load or not, the differences in read/write/refresh power usage are miniscule compared to the "baseline" power draw of the module and single DRAM package. Even the total size of the module isn't an indication, but the actual number of DRAM packages - a 32GB module with 8x4GB chips will draw roughly the same power as a 16GB module with 8x2GB chips (for DDR4 usually roughly ~2-3W per module). So if you have a lot of 16GB or especially 8GB modules, you can save *a lot* of power if you replace them with 32GB or even 64GB modules while keeping the same total memory size (but don't underpopulate the available memory channels; the performance penalty will lead to an overall higher power draw for the same workload).
micron released very detailed technical notes on this topic which can be easily found. They are usually named in the form of 'TN-12-34' (TN for 'Technical Note').

I can only join gpatrick with his statement about Brendan Gregg - his books and blog articles are some of the most detailed and first and foremost tichnically founded you will find. He never makes assumptions - everything is found by actual real-world tests and comparisons and followed through to the end, even if it involves some major changes to the kernel or software. If you can get your hand on the first edition of "Systems Performance", it contains a lot of DTrace examples which also work on FreeBSD, but all of that is gone in the more linux-centric second edition. But as the second edition contains a lot of new and additional material, it is also worth a read - so basically you'll want both...

Seems people with deep knowledge in power savings configuration do not have a big system at hand to compare and tune FreeBSD and Debian, do they?

its more the other way around: linux has always been used on a lot of junk small, 'exotic' and old systems and without much actual workload, hence it was desirable to have *very* aggressive power saving in place. For larger systems the idle or low-load power draw is of zero concern; what matters is the actual performance per watt for a system that is fully utilized - any resource that isn't used is wasted money and/or power, so tweaking the system to perform well at or near its maximum capacity is far more critical than some idle power savings by mocking with each and every special sleep-mode of any hardware component one might plug into a dekstop computer...

 

[weberjn]

a lot of junk small, 'exotic' and old systems and without much actual workload

How come that you know my home lab? ;-)

Don't know how many thousand systems are out there in homes idling Home Assistant or openHAB, but in total the two watts difference add up.

 

[Crivens]

How come that you know my home lab? ;-)

Don't know how many thousand systems are out there in homes idling Home Assistant or openHAB, but in total the two watts difference add up.

Exactly. Netflix will do power saving by shutting down servers untill the rest has close to 100% load, and power in servers/racks/DCs when the load warrants it. I't watch the interrupt frequencies (systat :vmstat) and tune there to maximize the length of sleep time for the cores, power down drives, maybe remove unused memory...
Basically, power the thing on and point a thermal camera onto it. Where it is warm, check what you can do about that component.

 

[Champion]

I heard a lot of urban myths about ZFS, but this one is new....
Actually, RAM mostly doesn't care if it is under load or not, the differences in read/write/refresh power usage are miniscule compared to the "baseline" power draw of the module and single DRAM package. Even the total size of the module isn't an indication, but the actual number of DRAM packages - a 32GB module with 8x4GB chips will draw roughly the same power as a 16GB module with 8x2GB chips (for DDR4 usually roughly ~2-3W per module). So if you have a lot of 16GB or especially 8GB modules, you can save *a lot* of power if you replace them with 32GB or even 64GB modules while keeping the same total memory size (but don't underpopulate the available memory channels; the performance penalty will lead to an overall higher power draw for the same workload).
micron released very detailed technical notes on this topic which can be easily found. They are usually named in the form of 'TN-12-34' (TN for 'Technical Note').

I can only join gpatrick with his statement about Brendan Gregg - his books and blog articles are some of the most detailed and first and foremost tichnically founded you will find. He never makes assumptions - everything is found by actual real-world tests and comparisons and followed through to the end, even if it involves some major changes to the kernel or software. If you can get your hand on the first edition of "Systems Performance", it contains a lot of DTrace examples which also work on FreeBSD, but all of that is gone in the more linux-centric second edition. But as the second edition contains a lot of new and additional material, it is also worth a read - so basically you'll want both...



its more the other way around: linux has always been used on a lot of junk small, 'exotic' and old systems and without much actual workload, hence it was desirable to have *very* aggressive power saving in place. For larger systems the idle or low-load power draw is of zero concern; what matters is the actual performance per watt for a system that is fully utilized - any resource that isn't used is wasted money and/or power, so tweaking the system to perform well at or near its maximum capacity is far more critical than some idle power savings by mocking with each and every special sleep-mode of any hardware component one might plug into a dekstop computer...

I apologize if maybe I was misunderstood, by caching I was referring to reading from disk into memory and data integrity checks consuming processor cycles. You are correct that whether or not the RAM is being utilized makes no difference in power consumption.
There is the possibility that after the caching process plateaus, power consumption would reduce, however, the system I was testing on was a very slow sparc machine, which never made it that far.