How to wake FreeBSD 13.2 on usb console cable? (UPS and NUT are used)

Gridsah

Gridsah

I have a HPE microserver Gen10 server running FreeBSD13.2 and a APC bk650M2 ch UPS which offers a rj45 to usb cable to connect to my Gen10.

NUT 2.8.0 installed by pkg running on Gen10 works well:

- system poweroff works well when power is outage and battery.charge is lower than battery.charge.low
- upsc <nutdev1> shows the ups status well

After shutting down the server in power outage, Gen10 should be booted while I restore power supply just like my Synology NAS did. But It is not. Gen10 is not booted when I restore power supply. I can only boot it by press power button and the /var/log/messages shows nothing related to usb information.

I turn on the 'pre-wake on lan' on Gen10 BIOS. The WOL works and I can boot my Gen10. I cannot find any keywords like 'usb wake' 'usb boot' on BIOS and just find the power loss status: (when power back machine should be) power on, power off, last status. Besides, I changed the power loss status to power on, but It is not working.

I want the Gen10 being booted when the power back. But I don't know what I should do next.

(What other information should I provide?
 
Gridsah said:
system poweroff works well when power is outage
Click to expand...

Gridsah said:
I changed the power loss status to power on, but It is not working.
Click to expand...
"Always on" is usually what it says in the bios.
That is what controls this function.

The only thing I can think of is that it is not actually shutting down but suspending.
When power is re-applied suspend is not re-waking the machine.

What happens if you literally rip the cord and make sure the power is off?
Does the machine automatically start then?

This could be BMC related stuff too. Are you updated on the latest BIOS and BMC firmware?

BMC=Broadband Managment Controller
This acts at a layer underneath the BIOS and can control power.
I do not know if HP MicroServer has BMC but if it does it is worth investigating.
 
Phishfry said:
The only thing I can think of is that it is not actually shutting down but suspending.
When power is re-applied suspend is not re-waking the machine.
Click to expand...
I am sure my Gen10 is shutdown, not suspending.
Because I configured SHUTDOWNCMD "/sbin/poweroff" in /usr/local/etc/nut/upsmon.conf and my Gen10 started to do self test while I press the power button (before what I did it was shutdown by upsmon daemon).

Phishfry said:
What happens if you literally rip the cord and make sure the power is off?
Does the machine automatically start then?
Click to expand...
I changed 'power loss status' from 'last status' to 'power on'. It works fine:

1. Gen10 is shutdown by upsmon daemon
2. I pull the power plug (Gen10 to ups)
3. put the Gen10 plug back in ups and Gen10 start to do self test and then boot

But:

1. Gen10 is shutdown by upsmon daemon
2. I pull the power plug (ups to electric supply)
3. put the ups plug back in electric supply and nothing happens on Gen10

Phishfry said:
This could be BMC related stuff too. Are you updated on the latest BIOS and BMC firmware?
Click to expand...
HPE Microserver Gen10 has no BMC (it is called iLO by HP), but Gen10 plus has one.

For now I wrote a script on my router to check if Gen10 is online. If not, wake it by WOL. Although this is also effective, it is a compromise approach. I am still want to find a way to let Gen10 boot from usb signal sent by ups.
 
Apparently your UPS is either not properly configured or lacks some capabilities. A UPS is supposed to power-cycle a load or power-cycle itself (after a configured period of time) even if utility power returns "too quickly" (before the UPS shuts down).
 
Andriy said:
Apparently your UPS is either not properly configured or lacks some capabilities. A UPS is supposed to power-cycle a load or power-cycle itself (after a configured period of time) even if utility power returns "too quickly" (before the UPS shuts down).
Click to expand...
I have two identical UPS, one for my HPE Microserver Gen 10 and another one for my Synology NAS.

My Synology NAS works well. But Gen10 doesn't. Gen10 just does not booting automatically while the utility power is back, but Synology NAS does. I believe there is no problem with my UPS. There must be some places on FreeBSD system or Gen10 BIOS that I didn't configure correctly.
 
Andriy said:
Apparently your UPS is either not properly configured or lacks some capabilities.
Click to expand...

More likely the former.

Andriy said:
A UPS is supposed to power-cycle a load or power-cycle itself (after a configured period of time) even if utility power returns "too quickly" (before the UPS shuts down).
Click to expand...

Yes, and configured periods of time may be relevant here. Perhaps Gridsah will tell us UPS and upsmon settings?

One thing UPS has to guard against, after turning off the load's power due to battery getting below minimum remaining charge and after signaling the load, is to ignore too-short returns of power, as often occur in stormy conditions here.

So there must be a minimum period of time between UPS power return detection and switching on power to loads, that needs to be enough to ensure the load is convinced that it's truly 'off'.

Compared to WoL, very few systems support power on (from ACPI S4 or S5 state) by USB signal. Google e.g.
"wake on usb" bios ...

The USB cable is just for communication between (here) upsmon and the UPS, both for UPS config and it reporting status, including imminent power loss.

Usually the only restart signal is restoration of power, and yes, "always on" or "power on" is the right setting.

And rather than poweroff,
/sbin/shutdown -p now Message to log
may be useful in syslog.
 
smithi said:
Yes, and configured periods of time may be relevant here. Perhaps Gridsah will tell us UPS and upsmon settings?
Click to expand...

Thanks for your analysis. Below are my configs. These configs works on Linux host with NUT older than v2.8.0 (ASUS B360M gaming plus + i3-8100). And I think these configs should also work on my Gen10 with FreeBSD system.

Besides, I am not sure if these configs are right now.

Code: 
root@gen10:/usr/local/etc/nut # cat nut.conf | grep -v "^#"

MODE=netserver

Code: 
root@gen10:/usr/local/etc/nut # cat ups.conf | grep -v "^#"

maxretry = 3

[nutdev1]
    driver = "usbhid-ups"
    port = "auto"
    vendorid = "051D"
    productid = "0002"
    product = "Back-UPS BK650M2-CH FW:294803G -292804G"
    serial = "xxxx"
    vendor = "American Power Conversion"
    ignorelb
    override.battery.runtime.low = -1
    override.battery.charge.low = 60

Code: 
root@gen10:/usr/local/etc/nut # cat upsd.users | grep -v "^#"

[admin]
    password = adminpwd
    upsmon master

[slave1]
    password  = slave1pwd
    upsmon slave

Code: 
root@gen10:/usr/local/etc/nut # cat upsmon.conf | grep -v "^#"

MONITOR nutdev1@localhost 1 admin adminpwd master

MINSUPPLIES 1

SHUTDOWNCMD "/sbin/poweroff"

POLLFREQ 5

POLLFREQALERT 5

HOSTSYNC 15

DEADTIME 15

POWERDOWNFLAG /etc/killpower

RBWARNTIME 43200

NOCOMMWARNTIME 300

FINALDELAY 5
 
smithi said:
One thing UPS has to guard against, after turning off the load's power due to battery getting below minimum remaining charge and after signaling the load, is to ignore too-short returns of power, as often occur in stormy conditions here.

So there must be a minimum period of time between UPS power return detection and switching on power to loads, that needs to be enough to ensure the load is convinced that it's truly 'off'.
Click to expand...
I waited long enough while testing. And there are no harsh conditions such as storms. I pretty much provided an ideal testing environment with cautious.

smithi said:
Compared to WoL, very few systems support power on (from ACPI S4 or S5 state) by USB signal. Google e.g.
"wake on usb" bios ...

The USB cable is just for communication between (here) upsmon and the UPS, both for UPS config and it reporting status, including imminent power loss.
Click to expand...
Yes, I have never seen such a configuration in the BIOS about something like 'wake on usb'.

smithi said:
And rather than poweroff,
/sbin/shutdown -p now Message to log
may be useful in syslog.
Click to expand...
Where should I put these code? Replace SHUTDOWNCMD "/sbin/poweroff" with /sbin/shutdown -p now Message to log in upsmon.conf ?
 
smithi said:
Usually the only restart signal is restoration of power, and yes, "always on" or "power on" is the right setting.
Click to expand...
Yes. I unplugged the UPS from the utility power. After FreeBSD shut down due to low UPS battery,

- I plug UPS to utility power back. Nothing happens.
- I unplugged my Gen10 from UPS, and then plug Gen10 to UPS back. Gen10 booted.

"always on" in BIOS works well.:p
 
Gridsah said:
Yes. I unplugged the UPS from the utility power. After FreeBSD shut down due to low UPS battery,

- I plug UPS to utility power back. Nothing happens.
- I unplugged my Gen10 from UPS, and then plug Gen10 to UPS back. Gen10 booted.

"always on" in BIOS works well.:p
Click to expand...
I think you found the race condition that is a problem in communication between UPS and computer that I haven't solved yet either (I'm using apcupsd, not nut):
  • Utility power fails. UPS starts supplying power.
  • UPS battery becomes empty enough to trigger shutdown. UPS signals computer that power will be lost soon. Computer begins shutdown, and completes shutdown.
  • While the computer shuts down, the UPS is sitting on a delay timer, for example "shut yourself down in 1 minute after signaling the computer".
  • During that delay time, and AFTER the computer has completed shutdown, utility power is restored. The UPS switches back to utility power.
  • Because power to the computer was never interrupted, the computer does not reboot. Ever.
This is a classic case of a race condition in a distributed system. The reason is that there is a bit of "state" that is being stored in two places, namely "the UPS will run out of power". It is stored in the UPS, which correctly signals the computer when entering this state. It is also stored in the computer, by virtue of the computer shutting down and waiting for the power interruption to reboot. Unfortunately, that bit of state is updated without coordinating between the two entities involved, because the computer (being shut down) has no means to listen to state updates.

As Andriy said already, the correct fix would be for the UPS to deliver on its promise that power will be interrupted (a promise that is implicit when it signaled that power will be lost. I get the feeling that consumer-grade APC units are not even capable of doing that. My tests of exactly the same situation with a similar model (I think APC calls it ES-550) have found the same problem.

One of the issues that makes testing this difficult / tedious / impossible is that every test cycle takes 10 to 20 minutes. So what I do is this: I don't actually connect the computer to the UPS. Instead I use a small electric heater as a load on the UPS, and also connect a desk lamp to it, as an indicator. I configure the UPS to shut down when the battery reaches about 90% capacity, so I don't have to wait long. Then I simulate the computer power being turned off and on by pulling its plug. Like this I can avoid reboot cycles, and the tests are a little less painful.
 
Andriy said:
Try to do testing mentioned here: Specifically, see "Testing shutdowns".
Click to expand...
I tested shutdowns and got information below. But I didn't know what these commands actually did:

Code: 
root@gen10:~ # upsdrvctl -t shutdown
Network UPS Tools - UPS driver controller 2.8.0
*** Testing mode: not calling exec/kill
   0.000000     [D2]
If you're not a NUT core developer, chances are that you're told to enable debugging
to see why a driver isn't working for you. We're sorry for the confusion, but this is
the 'upsdrvctl' wrapper, not the driver you're interested in.

Below you'll find one or more lines starting with 'exec:' followed by an absolute
path to the driver binary and some command line option. This is what the driver
starts and you need to copy and paste that line and append the debug flags to that
line (less the 'exec:' prefix).

Alternately, provide an additional '-d' (lower-case) parameter to 'upsdrvctl' to
pass its current debug level to the launched driver.

   0.000518     [D1] Shutdown UPS: nutdev1
   0.000525     [D2] exec:  /usr/local/libexec/nut/usbhid-ups -a nutdev1 -k

ralphbsz said:
As Andriy said already, the correct fix would be for the UPS to deliver on its promise that power will be interrupted (a promise that is implicit when it signaled that power will be lost. I get the feeling that consumer-grade APC units are not even capable of doing that. My tests of exactly the same situation with a similar model (I think APC calls it ES-550) have found the same problem.
Click to expand...
That reminds me something. I found an external circuit board on the main board of my Synology NAS. Even when the NAS is powered off, this small circuit board remains power on. When I bought my UPS, I noticed that Synology specially made adaptations for this UPS. Is this why Synology can work fine (automatically boot when utility power is back) on this UPS?
 
ralphbsz said:
I think you found the race condition that is a problem in communication between UPS and computer that I haven't solved yet either (I'm using apcupsd, not nut):
Click to expand...
So, maybe I need to unplug utility power to wait my Gen10 poweroff and then wait for enough time to see if my UPS shutdown? If my UPS does, plug the utility power back to see if my Gen10 boot automatically?
 
VladiBG said:
You need UPS with switched output outlets. Your backup ups doesn't have this functionality and cannot turn off and on it's outlet when the power is restored so even if you put your computer on always on / power on in the BIOS settings it cannot detect when the power came back to boot up again. Your option is to use ILO to manually power on the computer when the power is restored.

Click to expand...
Thanks. From what the URL said, this is indeed the case. Synology works on it because they did extra adaptations.

Besides, HPE microserver Gen10 have no iLO, but Gen10 plus has a optional one.

Now I write WOL script on my router not plugged to UPS and execute it regularly. This works well.
 
Andriy said:
Try to do testing mentioned here: Specifically, see "Testing shutdowns".
Click to expand...

These are good docs, far better than when I setup upsd etc ~'09 for our FM community radio station's main server in the Rainbow Region, so named for its thunderstorms.

Near impossible to debug remotely, but I'm finding it hard to believe that special equipment is needed to get this working, though we used Eaton not APC UPS.

Reposting part of config:

Gridsah said:
Code: 
root@gen10:/usr/local/etc/nut # cat ups.conf | grep -v "^#"

maxretry = 3

[nutdev1]
driver = "usbhid-ups"
port = "auto"
vendorid = "051D"
productid = "0002"
product = "Back-UPS BK650M2-CH FW:294803G -292804G"
serial = "xxxx"
vendor = "American Power Conversion"
ignorelb
override.battery.runtime.low = -1
override.battery.charge.low = 60
Click to expand...

Wondering what 'ignorelb' and the following overrides accomplish?

No shame in using the WoL solution, but it shouldn't be necessary; I still feel there's likely a timing issue here that longer delays between test actions or better settings might resolve.

That said, I'm in no position to test it, so letting it go ...
 
smithi said:
Near impossible to debug remotely, ...
Click to expand...
And even in person, it's very tedious to debug. I've been having problems with my apsupsd installation at home (similar UPS model), and I need to find a free afternoon to debug it. But now that it's summer and we don't have frequent power outages, I'm just not motivated to work on it.
 
smithi said:
Wondering what 'ignorelb' and the following overrides accomplish?
Click to expand...
Some UPS will immediately send a low-charge-state signal to NUT when it detects that the utility power disappears. NUT detects a low-charge-state signal and immediately triggers the shutdown operation.

After setting ignorelb, you also need to set 'battery.charge.low' and 'battery.runtime.low'. These variables define when the low-charge-state signal is triggered. '-1' indicates that this variable is not used.

The shutdown operation is triggered when the value of a variable meets one of the following conditions:

- Battery.charge < battery.charge.low
- Battery.runtime < battery.runtime.low

"override.battery.runtime.low = -1" means that NUT doesn't use 'battery.runtime.low' to determine if the battery is low.

"override.battery.charge.low = 60" means that when the UPS power is lower than 60%, the low-charge-state signal will be triggered, and NUT will shut down after receiving the signal.
 
ralphbsz said:
And even in person, it's very tedious to debug. I've been having problems with my apsupsd installation at home (similar UPS model), and I need to find a free afternoon to debug it. But now that it's summer and we don't have frequent power outages, I'm just not motivated to work on it.
Click to expand...
Absolutely right. I am working on it because I just purchased it. But now, I don't have much motivation to solve this problem. Anyway, WoL can also be used. 🤣🤣🤣
 
You must log in or register to reply here.
Back
Top