Where will I find these clocks? Are all of them built in to CPU? Are any of them likely to be on the motherboard itself?
They are somewhere on the mainboard. They are hardware controller chips, just like disk or network controllers. When the kernel writes certain commands into a network controller chip, it may send out a packet to the network; and when it writes certain commands to a timer chip, it may receive a reply after a specified time or have some time measured.
So, if you get the engineering documents from the manufacturer of the chips that are on your mainboard, you should find there the description on how the implemented timers should work (on a level where an hardware engineer should understand these things, i.e. no fun reading).
.
Now, when the kernel starts up, it searches for the existing hardware, and it does find the timers, just like all the other hardware, and then it attaches the appropriate device drivers to them.
But, as there do exist multiple timers on a modern system, we need to make a decision on which of them to use, and we need to make that decision during boot, when we learn to know our currently available hardware. Therefore when a specific device driver initalizes a timer, it learns the specific make+model, and it assignes it a quality rating (from known best practice). Then the kernel can easily choose the timer with the highest rating.
Can replacing a CMOS battery improve these numbers?
I don't think so. That battery is in action only when main power is off, and then no kernel is running. And the data kept by that battery is checksummed; so the board should complain on boot if the battery is dead.
When we see low numbers of a clock, what components should get attention?
I would think these numbers are hard-coded and reflect only the make+model of a timer, not an actual condition. So there is nothing to do (except maybe choosing a different hardware model with better ratings). The concerns here are construction-specific in the style of <by the way this timer works, is it prone to distortion from interrupt load?> and similar, i.e. best practices.
The more important question is: what do we want to achieve?
These timers work with accuracies in the realm of nanoseconds, and the engineering documents even talk about femtoseconds. This is quite ahead of what NTP can do at its best (getting NTP below 1 millisecond is quite good). Where do we need such accuracy?
The SPECTRE exploits have shown that such precision can be put to a use, i.e. the machines are able to measure the timing differences between accessing cached and noncached memory.
What I dont know (but am eager to learn): what are real-word applications where you would use these timers?
