Clock drift issues

[copumpkin]

See machyve/xhyve#46 and docker/for-mac#17 (and dozens of other confused threads all over the net, mostly stemming from Docker for Mac) for more details. I think this is the right place to address the issue, though.

See also LnL7/nix-docker#7 which links other related threads together.

[copumpkin]

It's not currently clear to me if anything needs to be done on the hyperkit side of things or whether this is entirely up to the VM guests to correct.

[rn]

The macOS hypervisor framework does not provide sufficient information to emulate one of the standard (ie included in Linux) PV clocks so the guest needs to run it's own NTP client and sync the time, ideally to the host time in case the host is not fully in sync. Things get more complex on resume from suspend, where sometimes it may take time for the guest time to resynch with the host/other NTP servers. On Docker for Mac we recently also discovered that the NTP client we use in the Linux VM sometimes sometime looses connection with the NTP server on the host, cause time drift.

Long story short, this is not really a HyperKit issue, so I'm closing it. docker/for-mac is the best place to raise issues regarding time sync issues on Docker for Mac.

[copumpkin]

@rneugeba ah, but minikube also uses HyperKit in a very similar manner to Docker for Mac and suffers from very similar time issues, which made me think that the issue could be solved more centrally. I guess we'll just need them to update their image to be more eager about ntp then?

[copumpkin]

@rneugeba also curious, are the Hypervisor.framework shortcomings documented somewhere? Talking about the thing you said about not being able to create a PV-like interface.

[rn]

You can create a PV-clock interface but it this would require additions to the Linux kernel. The limitation is that we can't emulate any of the existing PV clock interfaces for which the Linux kernel already has support

[copumpkin]

Oh, I see, thanks. If I want to read more, is there a good resource on this sort of stuff with respect to Hypervisor.framework?

[ijc]

@copumpkin there is not a lot of documentation for Hypervisor.framework. There is some in the relevant MacOS header files and in general the interface exposes much of the underlying VMX hardware interface fairly directly so you can infer a bunch of stuff from the Intel SDM.

The specific issue with implementing one of the existing Linux pvclock interfaces is that upon resume the MacOS kernel appears (this bit is completely undocumented AFAWCT) to apply an offset to the TSC (the hardware timer, which is virtualised via VMX) but does not expose that offset back to the userspace part of the hypervisor (which sees the unvirtualised/unoffset host level TSC only), therefore we cannot correctly calculate the correction factors which form part of the pvclock protocols.

[copumpkin]

Thank you, very helpful!

[avsm]

@ijc25 I think I'll file an Apple Radar bug about this as well. Do we have any issues showing an example of this behaviour so I can include it in the report?

[ijc]

@avsm It's in the internal tracker issue 1700. 229 is also interesting.