Hey,
Got a downtime error. Maybe with ams?
Thanks.
Did that give you any indication of location? Cloudflare’s errors are not super helpful, but I think they do include city.
Ah, yes it was ams.
I checked the logs and the app was running in ams. It seems 50/50 for me whether they start in lhr or ams as both are in the region pool. Spontaneously died without touching it:
2021-04-10T20:53:03.356Z 3914b5bb ams [info] GET /healthcheck 200 16 - 0.306 ms
2021-04-10T20:53:23.800Z 3914b5bb ams [info] Shutting down virtual machine
2021-04-10T20:53:53.808Z 3914b5bb ams [info] Sending signal SIGINT to main child process w/ PID 501
2021-04-10T20:53:54.812Z 3914b5bb ams [info] Main child exited with signal (with signal ‘SIGINT’, core dumped? false)
2021-04-10T21:01:52.937Z 3914b5bb ams [info] Starting instance
2021-04-10T21:01:52.977Z 3914b5bb ams [info] Configuring virtual machine
2021-04-10T21:01:52.986Z 3914b5bb ams [info] Pulling container image
2021-04-10T21:02:07.951Z 3914b5bb ams [info] Pull failed, retrying (attempt #0)
2021-04-10T21:02:22.799Z 3914b5bb ams [info] Pull failed, retrying (attempt #1)
2021-04-10T21:02:37.651Z 3914b5bb ams [info] Pulling image failed
2021-04-10T21:02:37.651Z 3914b5bb ams [info] Pull failed, retrying (attempt #2)
2021-04-10T21:03:08.857Z d8399649 lhr [info] Starting instance
2021-04-10T21:03:08.893Z d8399649 lhr [info] Configuring virtual machine
2021-04-10T21:03:08.896Z d8399649 lhr [info] Pulling container image
2021-04-10T21:03:12.582Z d8399649 lhr [info] Unpacking image
2021-04-10T21:03:12.591Z d8399649 lhr [info] Preparing kernel init
2021-04-10T21:03:12.888Z d8399649 lhr [info] Configuring firecracker
2021-04-10T21:03:12.978Z d8399649 lhr [info] Starting virtual machine
… and it’s back up now.
If I look in flyctl status the region is now lhr and it says it was created 22 minutes ago, which would explain it. As I haven’t touched it and it was under no load. But I’m guessing there was some issue with ams as the logs show it then being made in lhr. And now it works again.
Oh yes, we’re having network issues in AMS. So that was almost definitely it. We’ve been moving traffic and instances around to mitigate it.
1 Like