I am getting the following error intermittently with one of the instances of my Grafana Mimir deployment.
It will get this error a few times in a row causing that replica to drop out of the ring and restart.
"Get \"https://consul-syd-4.fly-shared.net/v1/kv/mimir-aerl-cloud-***/store-gateway?index=172384235&stale=&wait=10000ms\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
I’ve made sure the timeout is 20s. Although I can’t find any information anywhere on rate limits so that could be it, but it’s confusing that it’s only happening to one instance.
FYI I’ve censored some stuff with *** in the error that I’m not sure is private or not.