bhyve guest date&time is slower and slower...

[IPTRACE]

I sometimes have a problem with time on bhyve guest.
Time flows slower and slower. After restart OpenNTPd service date and time is ok.
The problem occurs when I'm resetting host without shutting down the guest.
Others VMs work ok with the same settings.

On guest:

user@vm1:~ % sudo sysctl kern.timecounter.hardware
kern.timecounter.hardware: TSC-low
user@vm1:~ % sudo service openntpd restart
Performing sanity check on openntpd configuration:
configuration OK
Stopping openntpd.
Waiting for PIDS: 11156.
Performing sanity check on openntpd configuration:
configuration OK
Starting openntpd.
user@vm1:~ % sudo ntpctl -sa
0/2 peers valid, clock unsynced

peer
   wt tl st  next  poll          offset       delay      jitter
10.1.0.11
    1  3  2    3s    5s             ---- peer not valid ----
10.1.1.11
    1  3  2    5s    7s             ---- peer not valid ----
user@vm1:~ % sudo ntpctl -sa
0/2 peers valid, clock unsynced

peer
   wt tl st  next  poll          offset       delay      jitter
10.1.0.11
    1  4  2    6s    7s             ---- peer not valid ----
10.1.1.11
    1  3  2    1s    7s             ---- peer not valid ----
user@vm1:~ % sudo ntpctl -sa
0/2 peers valid, clock unsynced

peer
   wt tl st  next  poll          offset       delay      jitter
10.1.0.11
    1  4  2    2s    7s             ---- peer not valid ----
10.1.1.11
    1  4  2    2s    5s             ---- peer not valid ----
user@vm1:~ % sudo ntpctl -sa
0/2 peers valid, clock unsynced

peer
   wt tl st  next  poll          offset       delay      jitter
10.1.0.11
    1  4  2    0s    7s             ---- peer not valid ----
10.1.1.11
    1  4  2    0s    5s             ---- peer not valid ----
user@vm1:~ % sudo ntpctl -sa
2/2 peers valid, clock unsynced

peer
   wt tl st  next  poll          offset       delay      jitter
10.1.0.11
    1  8  2    2s    8s      9180.185ms     0.486ms     0.182ms
10.1.1.11
    1  9  2   29s   32s      4494.787ms  7158.636ms  20455.774ms
user@vm1:~ % sudo ntpctl -sa
2/2 peers valid, clock unsynced, clock offset is 1226.543ms

peer
   wt tl st  next  poll          offset       delay      jitter
10.1.0.11
    1 10  2   30s   32s      5129.703ms     0.518ms     0.140ms
10.1.1.11
    1 10  2   25s   31s      4784.770ms     0.799ms     0.434ms
user@vm1:~ % sudo ntpctl -sa
2/2 peers valid, clock unsynced, clock offset is 1091.543ms

peer
   wt tl st  next  poll          offset       delay      jitter
10.1.0.11
    1 10  2    3s   32s      5129.703ms     0.518ms     0.140ms
10.1.1.11
    1 10  2   29s   31s      6395.755ms     0.803ms     0.436ms
user@vm1:~ % sudo ntpctl -sa
2/2 peers valid, clock unsynced, clock offset is 1452.814ms

peer
   wt tl st  next  poll          offset       delay      jitter
10.1.0.11
    1 10  2   18s   31s      6308.620ms     0.531ms     0.140ms
10.1.1.11
    1 10  2   13s   31s      5954.484ms     0.803ms     0.436ms
user@vm1:~ % sudo ntpctl -sa
2/2 peers valid, clock unsynced, clock offset is 1367.814ms

peer
   wt tl st  next  poll          offset       delay      jitter
10.1.0.11
    1 10  2    1s   31s      6308.620ms     0.531ms     0.140ms
10.1.1.11
    1 10  2   27s   31s      7726.575ms     0.768ms     0.446ms
user@vm1:~ % sudo ntpctl -sa
2/2 peers valid, clock unsynced, clock offset is 6509.517ms

peer
   wt tl st  next  poll          offset       delay      jitter
10.1.0.11
    1 10  2   27s   32s      6106.891ms     0.635ms     0.334ms
10.1.1.11
    1 10  2   26s   34s      5788.430ms     0.803ms     0.463ms

I have the followng entries in my dmesg command.
Could it be the reason of the problem?

Trying to mount root from ufs:/dev/vtbd0p2 [rw]...
WARNING: / was not properly dismounted
WARNING: /: mount pending error: blocks 40 files 4
random: unblocking device.

 

[IPTRACE]

Anyway, to avoid this situation I need to shutdown properly my almost 100 bhyve guests.
Is there any way to do it automatically without log in to every system?

 

[IPTRACE]

How to properly shutdown the guest I found here.
https://lists.freebsd.org/pipermail/freebsd-virtualization/2016-April/004348.html

kill -15 $PID

My own script added to /etc/rc.shutdown as

sh -c 'ksh /scripts/vm/stopall.ksh >> /var/log/vm/stopped.log 2>&1'

#!/bin/ksh

function myDate
{
        date +"%Y-%m-%d %H:%M:%S"
}

echo "============================"
echo "Starting at $(myDate)."

arrayVMname[0]=vm1
arrayVMname[1]=vm2
arrayVMname[2]=vm3

for i in ${arrayVMname[@]}; do
        echo "Shutting down virtual machine $i ...";
        VMPID=`pgrep -f $i`
        kill -15 $VMPID
        sleep 2
done

echo "Finished at $(myDate)."
echo "============================"