Hasura on Fly + Firebase Auth JWTIssuedAtFuture error

We’re using Hasura on Fly and having intermittent issues with a JWTIssuedAtFuture error using Firebase auth, which we think is based on an out of sync server time. I’ve exhausted options trying to solve within our codebase, including adding a JWT allowed_skew of up to an hour on Hasura without luck - I wondered if you guys had any insight into this issue, or if there was any known sync issues with time on fly VMs?

Edit: We’re running into this issue when passing the token via subscriptions (websockets) primarily.

Hrmmm that’s a super interesting problem. Just so I understand, Hasura validates the JWT and throws a JWTIssuedAtFuture error. The JWT itself is issued by Firebase.

It does seem like clocks are behind by a few seconds within some app instances, which might account for that. We’ll investigate and see what we can do about that.

Thanks for the quick reply @kurt, your understanding is correct afaik. Ok great! I’ll keep digging too, one would assume a few seconds doesn’t seem like enough to cause an issue!

Well I did find one that’s 25 seconds behind right now, so it’s definitely more than a few seconds. Do you know how long your instances have been running when this happen?

If you feel like doing some Docker image hacking, you can setup NTP inside your instances. You should absolutely not have to to this, though, but it might take us a few days to fix.

We should have a fix for this today. Great catch.

Amazing! Thanks for the super fast fixes and updates.

Your Hasura should be good to go now. We’re rolling the fix out to other regions over the next few days. If you need any particular region earlier let me know.