I’ve observed that when my Fly.io virtual machines resume after being suspended, they initially boot with the system clock set to the time when they were suspended, not the current time. The system clock eventually self-corrects after handling a few requests, but initial requests still receive the incorrect time.
Questions:
Is this expected behavior for suspended VMs on Fly.io?
What’s the recommended approach to ensure accurate time immediately after VM resume post-suspend?
Are there any built-in mechanisms I should be using to handle this situation?
Has anyone else encountered and solved this issue?
Any recommendations would be greatly appreciated!
We observed the same and had to switch back to "stop" because the clock skew caused crypto signature failures for all requests serviced before the system clock resynced.
Anyone have any recommendations?
We are running a django app on focal.
We were thinking of telling our machine to listen to a the suspend signal in fly and trigger a clock update before allowing handling requests. Though we’re not sure what that signal would be.
It looks like docker sends a SIGCONT signal when restarting after pause, see:
Subscribed.