Any issues with AMS at the moment? Getting failed deploys there

1

Hello,

I tried deploying an app multiple times and it kept failing due to “unhealthy allocations”. No reason that should happen code-wise, so I looked at the logs and saw each time it was deploying to … ams.

I looked at the app’s regions and saw that while the primary region is lhr, where I would prefer it to deploy, the backup region of ams was set. So I guess that is why it was deploying there.

I went into backup regions in the CLI and removed ams.

Did another deploy without changing anything, and this time deployed to lhr - yay - and worked perfectly. Which suggests it is ams that is the problem. Not the app. Else it wouldn’t work there.

No hurry fixing it as I am getting deploys working in lhr. But a heads-up may be an issue there.

Bonus: it would be nice if it was possible to specify the region to deploy to in the deploy command. Is that possible? Backup regions are nice, but I’d rather have the logic be that it use the primary region unless it is down and so can’t. So, lhr in my case. At the moment I had to manually remove a backup region to tell it not to use that one. And at some point I’ll have to remember to add it back. Which inevitably I won’t.

2

There was an issue with an ams host you happened to land on. Should be fixed now and we’ll keep an eye on it.

3

Ah.

Yes, I have been getting various issues and not sure how related they are. Lots of timeouts, like when connecting to external services. Not sure if it is a fly issue or networking issue. Sigh, variables.

There does seem to be an issue with deploying though. Like I tried a deploy to lhr which was working earlier today, after ams didn’t. But I’ve just checked a deploy which took half an hour :slight_smile:

Comparing the logs to one that worked earlier vs a later one that has got stuck, it seems my app is fine, but the deployment isn’t completing. It seems to be stuck waiting for that last line, and neveer getting it …

So, like, this worked from earlier:

Image size: 196 MB                                                                                                                                               
==> Creating release                                                                                                                                             
Release v342 created                                                                                                                                             
                                                                                                                                                                 
{"TS":"2021-07-31T19:40:06Z","Source":"deploy","Status":"detail","Message":"You can detach the terminal anytime without stopping the deployment"}                
{"TS":"2021-07-31T19:40:06Z","Source":"deploy","Status":"title","Message":"Monitoring Deployment"}                                                               
{"TS":"2021-07-31T19:40:06Z","Source":"deploy","Status":"info","Message":"v342 is being deployed"}                                                               
{"TS":"2021-07-31T19:40:10Z","Source":"deploy","Status":"info","Message":"5ca7e6f1: lhr pending"}                                                                
{"TS":"2021-07-31T19:40:12Z","Source":"deploy","Status":"info","Message":"5ca7e6f1: lhr pending"}                                                                
{"TS":"2021-07-31T19:40:13Z","Source":"deploy","Status":"info","Message":"5ca7e6f1: lhr pending"}                                                                
{"TS":"2021-07-31T19:40:16Z","Source":"deploy","Status":"info","Message":"5ca7e6f1: lhr running unhealthy [health checks: 2 total, 1 passing, 1 critical]"}      
{"TS":"2021-07-31T19:40:18Z","Source":"deploy","Status":"info","Message":"5ca7e6f1: lhr running healthy [health checks: 2 total, 2 passing]"}                    
{"TS":"2021-07-31T19:40:28Z","Source":"deploy","Status":"done","Message":"v342 deployed successfully\n"}

All good. That’s what should happen.

But this recent one got stuck. The last lines from it:

Image size: 196 MB                                                                                                                                               
==> Creating release                                                                                                                                             
Release v344 created                                                                                                                                             
                                                                                                                                                                 
{"TS":"2021-07-31T20:54:52Z","Source":"deploy","Status":"detail","Message":"You can detach the terminal anytime without stopping the deployment"}                
{"TS":"2021-07-31T20:54:52Z","Source":"deploy","Status":"title","Message":"Monitoring Deployment"}                                                               
{"TS":"2021-07-31T20:54:52Z","Source":"deploy","Status":"info","Message":"v343 is being deployed"}                                                               
{"TS":"2021-07-31T20:55:22Z","Source":"deploy","Status":"info","Message":"323dc703: lhr complete healthy [health checks: 2 total, 2 passing]"}

… and never gets to those last lines to say it completed. Hmm. So it seems stuck. I’ll terminate and try again, but does seem a bit random.

4

Also: I don’t know if you have changed your deploy builder/process because I’ve noticed the additional line:

Creating WireGuard peer "interactive-name-208" in region "iad"

… and it then takes a minute connecting to it. Seems slower than it used to.

Is there a reason why it picks iad? It seems to always be iad from the ones I’ve looked back at. Can I pick my own region, like lhr? I’m wondering if that is part of why builds have got slower too.