FreeBSD - clock drift

Hi,

we use FreeBSD 12.0 on an "old" DELL 2950 Server (8-Core, 24GB Memory) as our main ZABBIX Monitoring System.
From time to time, last happened yesterday, the system clock goes of by an significant amount of time and ZABBIX starts to alert because it thinks
that it didn't receive values from the monitored hosts for more than 5 minutes. I mangaged to take a look at the system earlier this month and the time was off by more than 2 hours.
The server itself has no other problems than the clock drift - currently i'm running out of ideas. The last option would be switching to another hardware.
I have more than 40 FreeBSD but all of them are running on VMWARE - not even one has a problem with such significant timedrifts.



The specs:



Code:
root@zabbix:/root # uname -a
FreeBSD zabbix 12.0-RELEASE-p6 FreeBSD 12.0-RELEASE-p6 GENERIC  amd64
(was -p4 yesterday)


Code:
/etc/rc.conf
ntpd_enable="YES"
ntpd_sync_on_start="YES"

Not complete dmesg.boot from today

Code:
root@zabbix:/root # cat /var/log/dmesg.today
---<<BOOT>>---
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-RELEASE-p4 GENERIC amd64
FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)
VT(vga): resolution 640x480
CPU: Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz (2493.83-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x1067a  Family=0x6  Model=0x17  Stepping=10
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0xc0ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,OSXSAVE>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  VT-x: HLT,PAUSE
  TSC: P-state invariant, performance statistics
real memory  = 25769803776 (24576 MB)
avail memory = 24962740224 (23806 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <DELL   PE_SC3  >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s)
random: unblocking device.
ioapic0: Changing APIC ID to 8
ioapic0 <Version 2.0> irqs 0-23 on motherboard
Launching APs: 4 2 1 7 3 5 6
Timecounter "TSC-low" frequency 1246916246 Hz quality 1000
random: entropy device external interface
kbd1 at kbdmux0
netmap: loaded module
[ath_hal] loaded
module_register_init: MOD_LOAD (vesa, 0xffffffff810f9a00, 0) error 19
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <DELL PE_SC3> on motherboard
acpi0: Power Button (fixed)
ipmi0: <IPMI System Interface> port 0xca8,0xcac on acpi0
ipmi0: KCS mode found at io 0xca8 on acpi
cpu0: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x7f irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x5f irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <PCI-PCI bridge> at device 0.0 on pci3
pci4: <PCI bus> on pcib4
bce0: <QLogic NetXtreme II BCM5708 1000Base-T (B2)> mem 0xd6000000-0xd7ffffff irq 16 at device 0.0 on pci4
miibus0: <MII bus> on bce0
brgphy0: <BCM5708C 1000BASE-T media interface> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce0: Using defaults for TSO: 65518/35/2048
bce0: Ethernet address: 00:22:19:b5:8b:56
bce0:
bce0: link state changed to DOWN
ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); B/C (7.4.0); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (UMP 1.1.9)
Coal (RX:6,6,18,18; TX:20,20,80,80)
pcib5: <ACPI PCI-PCI bridge> at device 1.0 on pci2
pci5: <ACPI PCI bus> on pcib5
puc0: <Oxford Semiconductor OXPCIe954 UARTs> mem 0xd5efc000-0xd5efffff,0xd5c00000-0xd5dfffff,0xd5a00000-0xd5bfffff irq 17 at device 0.0 on pci5
puc0: 4 UARTs detected
uart2: <16950 or compatible> at port 1 on puc0
uart3: <16950 or compatible> at port 2 on puc0
uart4: <16950 or compatible> at port 3 on puc0
uart5: <16950 or compatible> at port 4 on puc0
pcib6: <PCI-PCI bridge> at device 0.3 on pci1
pci6: <PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci7: <ACPI PCI bus> on pcib7
pcib11: <PCI-PCI bridge> at device 7.0 on pci0
pci11: <PCI bus> on pcib11
pcib12: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci12: <ACPI PCI bus> on pcib12
pcib13: <PCI-PCI bridge> at device 0.0 on pci12
pci13: <PCI bus> on pcib13
bce1: <QLogic NetXtreme II BCM5708 1000Base-T (B2)> mem 0xda000000-0xdbffffff irq 16 at device 0.0 on pci13
miibus1: <MII bus> on bce1
brgphy1: <BCM5708C 1000BASE-T media interface> PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce1: Using defaults for TSO: 65518/35/2048
bce1: Ethernet address: 00:22:19:b5:8b:54
bce1:
bce1: link state changed to DOWN
ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); B/C (7.4.0); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (UMP 1.1.9)
Coal (RX:6,6,18,18; TX:20,20,80,80)
uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port 0xcce0-0xccff irq 21 at device 29.0 on pci0
usbus0 on uhci0
usbus0: 12Mbps Full Speed USB v1.0
uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port 0xccc0-0xccdf irq 20 at device 29.1 on pci0
usbus1 on uhci1
usbus1: 12Mbps Full Speed USB v1.0
uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port 0xcca0-0xccbf irq 21 at device 29.2 on pci0
usbus2 on uhci2
usbus2: 12Mbps Full Speed USB v1.0
uhci3: <Intel 631XESB/632XESB/3100 USB controller USB-4> port 0xcc80-0xcc9f irq 20 at device 29.3 on pci0
usbus3 on uhci3
usbus3: 12Mbps Full Speed USB v1.0
ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xde500400-0xde5007ff irq 21 at device 29.7 on pci0
usbus4: EHCI version 1.0
usbus4 on ehci0
usbus4: 480Mbps High Speed USB v2.0
pcib14: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci14: <ACPI PCI bus> on pcib14
vgapci0: <VGA-compatible display> port 0xdc00-0xdcff mem 0xf0000000-0xf7ffffff,0xde2d0000-0xde2dffff irq 19 at device 13.0 on pci14
vgapci0: Boot video device
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 SATA300 controller> port 0xcc78-0xcc7f,0xcc70-0xcc73,0xcc60-0xcc67,0xcc58-0xcc5b,0xcc40-0xcc4f mem 0xde500000-0xde5003ff irq 23 at device 31.2 on pci0
ata2: <ATA channel> at channel 0 on atapci0
ata3: <ATA channel> at channel 1 on atapci0
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
uart0: <Non-standard ns8250 class UART with FIFOs> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc8fff,0xc9000-0xc9fff,0xd2800-0xd37ff,0xec000-0xeffff pnpid ORM0000 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbdc0: non-PNP ISA device will be removed from GENERIC in FreeBSD 12.
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 471f471f0600471f
device_attach: est0 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 471f471f0600471f
device_attach: est1 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 471f471f0600471f
device_attach: est2 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 471f471f0600471f
device_attach: est3 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 471f471f0600471f
device_attach: est4 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 471f471f0600471f
device_attach: est5 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 471f471f0600471f
device_attach: est6 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 471f471f0600471f
device_attach: est7 attach returned 6
Timecounters tick every 1.000 msec
ipmi0: IPMI device rev. 0, firmware rev. 2.50, version 2.0, device support mask 0xdf
ipmi0: Number of channels 4
ipmi0: Attached watchdog
ipmi0: Establishing power cycle handler

Code:
root@zabbix:/root # sysctl kern.timecounter
kern.timecounter.tsc_shift: 1
kern.timecounter.smp_tsc_adjust: 0
kern.timecounter.smp_tsc: 1
kern.timecounter.invariant_tsc: 1
kern.timecounter.fast_gettime: 1
kern.timecounter.tick: 1
kern.timecounter.choice: ACPI-fast(900) HPET(950) i8254(0) TSC-low(1000) dummy(-1000000)
kern.timecounter.hardware: HPET
kern.timecounter.alloweddeviation: 5
kern.timecounter.stepwarnings: 0
kern.timecounter.tc.ACPI-fast.quality: 900
kern.timecounter.tc.ACPI-fast.frequency: 3579545
kern.timecounter.tc.ACPI-fast.counter: 8148665
kern.timecounter.tc.ACPI-fast.mask: 16777215
kern.timecounter.tc.HPET.quality: 950
kern.timecounter.tc.HPET.frequency: 14318180
kern.timecounter.tc.HPET.counter: 118052310
kern.timecounter.tc.HPET.mask: 4294967295
kern.timecounter.tc.i8254.quality: 0
kern.timecounter.tc.i8254.frequency: 1193182
kern.timecounter.tc.i8254.counter: 12981
kern.timecounter.tc.i8254.mask: 65535
kern.timecounter.tc.TSC-low.quality: 1000
kern.timecounter.tc.TSC-low.frequency: 1246913092
kern.timecounter.tc.TSC-low.counter: 3950691641
kern.timecounter.tc.TSC-low.mask: 4294967295

Currently using HPET, i tried every time shown in kern.timecounter.choice -> no difference.

Code:
root@zabbix:/root # sysctl kern.eventtimer
kern.eventtimer.periodic: 0
kern.eventtimer.timer: HPET
kern.eventtimer.idletick: 0
kern.eventtimer.singlemul: 2
kern.eventtimer.choice: HPET(350) HPET1(340) HPET2(340) LAPIC(100) i8254(100) RTC(0)
kern.eventtimer.et.HPET2.quality: 340
kern.eventtimer.et.HPET2.frequency: 14318180
kern.eventtimer.et.HPET2.flags: 3
kern.eventtimer.et.HPET1.quality: 340
kern.eventtimer.et.HPET1.frequency: 14318180
kern.eventtimer.et.HPET1.flags: 3
kern.eventtimer.et.HPET.quality: 350
kern.eventtimer.et.HPET.frequency: 14318180
kern.eventtimer.et.HPET.flags: 3
kern.eventtimer.et.i8254.quality: 100
kern.eventtimer.et.i8254.frequency: 1193182
kern.eventtimer.et.i8254.flags: 1
kern.eventtimer.et.RTC.quality: 0
kern.eventtimer.et.RTC.frequency: 32768
kern.eventtimer.et.RTC.flags: 17
kern.eventtimer.et.LAPIC.quality: 100
kern.eventtimer.et.LAPIC.frequency: 0
kern.eventtimer.et.LAPIC.flags: 15

The messages and ntp.log entries from yesterday's SMS storm:


Code:
/var/log/messages
<11>1 2019-06-28T15:52:54.122797+02:00 zabbix monit 781 - - 'ntpd' process is not running
<101>1 2019-06-28T15:52:54.334250+02:00 zabbix ntpd 74557 - - ntpd 4.2.8p12-a (1): Starting
<101>1 2019-06-28T15:52:54.476211+02:00 zabbix ntpd 74558 - - leapsecond file ('/var/db/ntpd.leap-seconds.list'): good hash signature
<101>1 2019-06-28T15:52:54.476451+02:00 zabbix ntpd 74558 - - leapsecond file ('/var/db/ntpd.leap-seconds.list'): loaded, expire=2019-12-28T00:00:00Z last=2017-01-01T00:00:00Z ofs=37
<101>1 2019-06-28T15:52:54.476510+02:00 zabbix ntpd 74558 - - switching logging to file /var/log/ntp.log
<11>1 2019-06-28T18:23:44.607055+02:00 zabbix monit 781 - - 'ntpd' failed protocol test [NTP3] at [127.0.0.1]:123 [UDP/IP] -- NTP: Server not synchronized
<11>1 2019-06-28T18:24:14.731092+02:00 zabbix syslogd - - - last message repeated 1 times
<11>1 2019-06-28T18:26:14.884308+02:00 zabbix syslogd - - - last message repeated 4 times
<11>1 2019-06-28T18:27:44.922089+02:00 zabbix syslogd - - - last message repeated 3 times

from 2 hours later:

Code:
<11>1 2019-06-28T21:09:58.399197+02:00 zabbix monit 781 - - 'ntpd' failed protocol test [NTP3] at [127.0.0.1]:123 [UDP/IP] -- NTP: Server not synchronized
<11>1 2019-06-28T21:10:28.425858+02:00 zabbix syslogd - - - last message repeated 1 times
<101>1 2019-06-28T21:10:32.782783+02:00 zabbix ntpd 98136 - - ntpd 4.2.8p12-a (1): Starting
<101>1 2019-06-28T21:10:32.919627+02:00 zabbix ntpd 98137 - - leapsecond file ('/var/db/ntpd.leap-seconds.list'): good hash signature
<101>1 2019-06-28T21:10:32.919862+02:00 zabbix ntpd 98137 - - leapsecond file ('/var/db/ntpd.leap-seconds.list'): loaded, expire=2019-12-28T00:00:00Z last=2017-01-01T00:00:00Z ofs=37
<101>1 2019-06-28T21:10:32.919915+02:00 zabbix ntpd 98137 - - switching logging to file /var/log/ntp.log
<11>1 2019-06-28T21:11:28.447013+02:00 zabbix monit 781 - - 'ntpd' failed protocol test [NTP3] at [127.0.0.1]:123 [UDP/IP] -- NTP: Server not synchronized
<11>1 2019-06-28T21:11:58.453586+02:00 zabbix syslogd - - - last message repeated 1 times
<11>1 2019-06-28T21:13:28.501482+02:00 zabbix syslogd - - - last message repeated 3 times
<2>1 2019-06-29T03:00:20.876307+02:00 zabbix kernel - - - mfi0: 14601 (615092400s/0x0020/info) - Patrol Read started
<2>1 2019-06-29T03:54:55.823605+02:00 zabbix kernel - - - mfi0: 14602 (615095674s/0x0020/info) - Patrol Read complete

/var/log/ntp.log
Code:
28 Jun 15:48:58 ntpd[3297]: Soliciting pool server 194.112.182.172
28 Jun 15:48:58 ntpd[3297]: Soliciting pool server 91.206.8.34
28 Jun 15:48:59 ntpd[3297]: Soliciting pool server 80.92.126.65
28 Jun 15:52:23 ntpd[3297]: Soliciting pool server 144.208.210.122
28 Jun 15:52:23 ntpd[3297]: Soliciting pool server 217.196.145.42
28 Jun 15:52:24 ntpd[3297]: Soliciting pool server 91.206.8.70
28 Jun 15:52:54 ntpd[74558]: Listen and drop on 0 v6wildcard [::]:123
28 Jun 15:52:54 ntpd[74558]: Listen and drop on 1 v4wildcard 0.0.0.0:123
28 Jun 15:52:54 ntpd[74558]: Listen normally on 2 lo0 [::1]:123
28 Jun 15:52:54 ntpd[74558]: Listen normally on 3 lo0 [fe80::1%3]:123
28 Jun 15:52:54 ntpd[74558]: Listen normally on 4 lo0 127.0.0.1:123
28 Jun 15:52:54 ntpd[74558]: Listen normally on 5 lagg0 192.168.20.49:123
28 Jun 15:52:54 ntpd[74558]: Listening on routing socket on fd #26 for interface updates
28 Jun 15:52:54 ntpd[74558]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized
28 Jun 15:52:54 ntpd[74558]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized
28 Jun 15:52:55 ntpd[74558]: Soliciting pool server 144.208.210.122
28 Jun 15:52:56 ntpd[74558]: Soliciting pool server 144.76.197.108
28 Jun 15:52:57 ntpd[74558]: Soliciting pool server 91.206.8.70
28 Jun 15:52:58 ntpd[74558]: Soliciting pool server 217.196.145.42
28 Jun 18:25:02 ntpd[74558]: Soliciting pool server 193.170.62.252
28 Jun 18:27:57 ntpd[74558]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized
28 Jun 19:46:58 ntpd[74558]: Soliciting pool server 193.171.23.163
28 Jun 19:46:59 ntpd[74558]: Soliciting pool server 212.69.166.153
28 Jun 19:47:00 ntpd[74558]: Soliciting pool server 185.144.161.170
28 Jun 19:47:01 ntpd[74558]: Soliciting pool server 91.206.8.34
28 Jun 19:47:02 ntpd[74558]: Soliciting pool server 194.112.182.172
28 Jun 19:47:19 ntpd[90947]: Listen and drop on 0 v6wildcard [::]:123
28 Jun 19:47:19 ntpd[90947]: Listen and drop on 1 v4wildcard 0.0.0.0:123
28 Jun 19:47:19 ntpd[90947]: Listen normally on 2 lo0 [::1]:123
28 Jun 19:47:19 ntpd[90947]: Listen normally on 3 lo0 [fe80::1%3]:123
28 Jun 19:47:19 ntpd[90947]: Listen normally on 4 lo0 127.0.0.1:123
28 Jun 19:47:19 ntpd[90947]: Listen normally on 5 lagg0 192.168.20.49:123
28 Jun 19:47:19 ntpd[90947]: Listening on routing socket on fd #26 for interface updates
28 Jun 19:47:19 ntpd[90947]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized
28 Jun 19:47:19 ntpd[90947]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized
28 Jun 19:47:20 ntpd[90947]: Soliciting pool server 185.144.161.170
28 Jun 19:47:21 ntpd[90947]: Soliciting pool server 194.112.182.172
28 Jun 19:47:22 ntpd[90947]: Soliciting pool server 91.206.8.34
28 Jun 19:47:23 ntpd[90947]: Soliciting pool server 144.76.197.108
28 Jun 21:06:37 ntpd[90947]: receive: Unexpected origin timestamp 0xe0c0d131.e425c864 does not match aorg 0000000000.00000000 from server@194.112.182.172 xmt 0xe0c0e3bd.f91843f5
28 Jun 21:08:46 ntpd[90947]: Soliciting pool server 217.196.145.42
28 Jun 21:10:31 ntpd[90947]: ntpd exiting on signal 15 (Terminated)
28 Jun 21:10:31 ntpd[90947]: 185.144.161.170 local addr 192.168.20.49 -> <null>
28 Jun 21:10:31 ntpd[90947]: 194.112.182.172 local addr 192.168.20.49 -> <null>
28 Jun 21:10:31 ntpd[90947]: 91.206.8.34 local addr 192.168.20.49 -> <null>
28 Jun 21:10:31 ntpd[90947]: 144.76.197.108 local addr 192.168.20.49 -> <null>
28 Jun 21:10:31 ntpd[90947]: 217.196.145.42 local addr 192.168.20.49 -> <null>
28 Jun 21:10:32 ntpd[98137]: Listen and drop on 0 v6wildcard [::]:123
28 Jun 21:10:32 ntpd[98137]: Listen and drop on 1 v4wildcard 0.0.0.0:123
28 Jun 21:10:32 ntpd[98137]: Listen normally on 2 lo0 [::1]:123
28 Jun 21:10:32 ntpd[98137]: Listen normally on 3 lo0 [fe80::1%3]:123
28 Jun 21:10:32 ntpd[98137]: Listen normally on 4 lo0 127.0.0.1:123
28 Jun 21:10:32 ntpd[98137]: Listen normally on 5 lagg0 192.168.20.49:123
28 Jun 21:10:32 ntpd[98137]: Listening on routing socket on fd #26 for interface updates
28 Jun 21:10:32 ntpd[98137]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized
28 Jun 21:10:32 ntpd[98137]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized
28 Jun 21:19:18 ntpd[98137]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized


Code:
root@zabbix:/root # cat /etc/ntp.conf
tos minclock 3 maxclock 6
server 192.168.20.11
server 192.168.20.12
server 192.168.20.13
restrict default limited kod nomodify notrap noquery nopeer
restrict source  limited kod nomodify notrap noquery
restrict 127.0.0.1
restrict ::1
leapfile "/var/db/ntpd.leap-seconds.list"
logfile /var/log/ntp.log
driftfile /var/db/ntpd.drift

Code:
root@zabbix:/root # cat /var/db/ntpd.drift
36.521

Code:
root@zabbix:/root # tail -n1 /etc/localtime
CET-1CEST,M3.5.0,M10.5.0/3

Code:
root@zabbix:/tmp # ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+192.168.20.11   86.59.113.114    3 u   22   64  377    0.195   12.266   3.090
*192.168.20.12   178.255.158.14   2 u   12   64  337    0.401   11.513   2.852
+192.168.20.13   217.196.145.42   2 u   36   64  377    0.243   10.874   2.028


At the moment (changed back today), i use our internal NTP servers again - they use pool 0.freebsd.pool.ntp.org iburst as clock source.
I tried to use several other servers in the servers ntp.conf - no difference.

NTP on the server:
Code:
root@zabbix:/tmp # ps ax | grep ntp
 821  -  Ss     0:00.47 /usr/sbin/ntpd -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -g

thanks for listening.
 
Just had the same problem again (at ~07:22h) :

Code:
root@zabbix:/root #  ipmitool sel time get && date
07/01/2019 07:22:55
Mon Jul  1 04:21:26 CEST 2019

No errors in ntpd.log or messages.

Will try to replace the CMOS Battery, although ipmi reports that the battery status is ok.
 
If changing the battery doesn't fix your problem you might want to have a look at this:

After some googling I've set this value:
kern.timecounter.hardware=HPET (from the default ACPI-low) without rebooting and it seems that it instantly fixed the time lagging.
And it also needs to be set in /etc/sysctl.conf to persist across reboots.
Hope this helps someone.
 
Changed the CMOS battery a couple of days ago - did not help.
Just had a massive kernel time drift (12.Jul.2019 1926h):

Code:
root@zabbix:/root # ipmitool sel time get && date
07/12/2019 19:26:38
Fri Jul 12 16:35:43 CEST 2019

No errors in /var/log/messages
Last lines from /var/log/ntp.log:

Code:
29 Jun 14:11:59 ntpd[13134]: Listen and drop on 0 v6wildcard [::]:123
29 Jun 14:11:59 ntpd[13134]: Listen and drop on 1 v4wildcard 0.0.0.0:123
29 Jun 14:11:59 ntpd[13134]: Listen normally on 2 lo0 [::1]:123
29 Jun 14:11:59 ntpd[13134]: Listen normally on 3 lo0 [fe80::1%3]:123
29 Jun 14:11:59 ntpd[13134]: Listen normally on 4 lo0 127.0.0.1:123
29 Jun 14:11:59 ntpd[13134]: Listen normally on 5 lagg0 192.168.20.49:123
29 Jun 14:11:59 ntpd[13134]: Listening on routing socket on fd #26 for interface updates
29 Jun 14:11:59 ntpd[13134]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized
29 Jun 14:11:59 ntpd[13134]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized
29 Jun 14:20:40 ntpd[13134]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized
12 Jul 16:36:08 ntpd[13134]: ntpd exiting on signal 15 (Terminated)
12 Jul 16:36:08 ntpd[13134]: 192.168.20.11 local addr 192.168.20.49 -> <null>
12 Jul 16:36:08 ntpd[13134]: 192.168.20.12 local addr 192.168.20.49 -> <null>
12 Jul 16:36:08 ntpd[13134]: 192.168.20.13 local addr 192.168.20.49 -> <null>
12 Jul 16:36:09 ntpd[97266]: Listen and drop on 0 v6wildcard [::]:123
12 Jul 16:36:09 ntpd[97266]: Listen and drop on 1 v4wildcard 0.0.0.0:123
12 Jul 16:36:09 ntpd[97266]: Listen normally on 2 lo0 [::1]:123
12 Jul 16:36:09 ntpd[97266]: Listen normally on 3 lo0 [fe80::1%3]:123
12 Jul 16:36:09 ntpd[97266]: Listen normally on 4 lo0 127.0.0.1:123
12 Jul 16:36:09 ntpd[97266]: Listen normally on 5 lagg0 192.168.20.49:123
12 Jul 16:36:09 ntpd[97266]: Listening on routing socket on fd #26 for interface updates
12 Jul 16:36:09 ntpd[97266]: kernel reports TIME_ERROR: 0x6041: Clock Unsynchronized
12 Jul 16:36:09 ntpd[97266]: kernel reports TIME_ERROR: 0x6041: Clock Unsynchronized

I guess the problems have something to do with ntp. After restarting nptd, the time got corrected after 2-3mins.

I'm running out of ideas - any help would be appreciated.

Thanks.
 
Not that i know, i installed ntp only. Any chance for some dtrace or any other magic to find out ?
 
Not that i know, i installed ntp only. Any chance for some dtrace or any other magic to find out ?

I don't know about qtrace, but I have another thought...

Disable ntp, i.e., stop it completely (service ntpd stop ?), and see if the clock continues to jump around. If it does, then maybe try booting a different OS's live CD, set the date/time with the date command, and see if it continues to jump around. If it were me, I'd tr OpenBSD because that OS has a different ntp daemon (openntpd).

If it does, then I'd susect some manner of hardware issue. Wild guess: Is there some hardware supervisor that sets the clock?
 
Disabled ntpd on Jul 15 - just had an massive clock drift.
Kernel Time and IPMI Time differed more than 3h.
Guess its time to change the hardware ?

Is there some hardware supervisor that sets the clock?

Not that i know of, its a plain old standard DELL 2950 - how can i check that ?
 
Hi,

Maybe try a "tinker panic 0" in ntp.conf (as you would with a VMware client). That should catch and correct the drifts fairly early...
 
Back
Top