One of my instances is not responding to HTTPS requests.
With curl -vvv, it gets stuck at:
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 32)!
This behaviour is the same for both custom domain, and my-instance.fly.dev.
The app responds as normal if I access with flyctl ssh console and then wget localhost/....
There is nothing obviously wrong in the fly.io dashboard.
Any tips on debugging this?
It has now started working. According to my monitoring, it was down for about 35 minutes, roughly starting at the time I deployed a new version. I can’t see anything relevant in logs.
Any suggestions of what went on, or how to investigate further would be greatly appreciated - obviously this is quite concerning!
I can see the same behaviour on my instances.
When I deploy a new version it get stuck. According to logs the app is up and running, and resolving health checks.
The only way to get it responding again is changing the size and, after applied, rolling it back.
In my case I have an application with two Ruby processes.
Hi @odabajo, did you find any way to debug the issue?
The only way to get it responding again is changing the size
What are you changing the size of?
In my case, I changed the memory size and it forced the instance to be recreated.
However, that ewas a temporal issue and now it works fine once a deploy is done.
Are you still experiencing the issues?
Sadly not - from my perspective, it was completely random.
This was likely caused by our new state propagation infrastructure. It’s still early days with these changes, but it looks better since a few days ago.
Hopefully this doesn’t happen again.
Thanks, @jerome - this is good to know.
In future do you have suggestions for debugging an issue like this, or checking status of fly services? It’s quite alarming having a production outage with no way to diagnose, or a way to confirm it’s an external problem.