We deployed new version of app in Singapore region 10 minutes ago. Our app fires request to external API for every request it receives. As soon as new version got deployed, singapore region started having timeout errors for external API calls. We run dual-region workload and Los Angeles was unaffected.
We have removed Singapore region and added Amsterdam for now. What can be cause of this and how can we be sure that our app does not face problems like this in future?
Hi thanks for sharing this, I can confirm this issue. I will try to change region for now.
This depends entirely on the external API(s). What hostnames were you hitting?
The best way to troubleshoot this is to fly console ssh into a VM, then see if you can use curl -v to reproduce the error. Then post results here.
If you have a Launch or Scale plan, you can also send details to us via your premium support email address: Fly.io Support: community vs email (Read this first)
I’m also facing a similar issue in Singapore region Help with intermittent network issue
@kurt it’s not possible to re-produce now because it was production workload and we had to switch region immediately. There was definitely some issue as confirmed by others. Hostname did not matter, all requests were timing out.
Singapore region in general has many network issues. In past, we faced image pull failures while every other region works fine. For now, we are using ams and lax for our production but I hope Fly team has some way to immediately debug such issues on their own when they occur.
I am not convinced this is an error in our networking. We’re happy to look into it, though, so if you do have more details about what kinds of errors you were hitting please let us know. We’ll need to know the target IP address and port at the very least.
Is there a way I can send details privately?