it’s not only ams
edit: I’m back up, love you guys
please try pointing flyctl to registry3.fly.io
Oh buddy, you sent me in a panic loop for a moment 
what was your issue? what region are you connecting from?
same thing. stuck, from registry3.fly.io
Someone here posted that it also occurred in WAW region btw
what is your app builder name? try destroying it and redeploy. long shot but that may clear things
Just tried that, didn’t change anything
lax. maintenance message is still present but apps are back.
during incident, machines failed to load in cli + web interface and db was unreachable by app. emergency maintenance message appeared after ~maybe fifteen minutes. situation resolved for us in ~maybe thirty minutes total. going to make that specific error message a little more informative/pretty – didn’t anticipate a situation where app is reachable but db isn’t
shareholders have been soothed, warm milk dispensed, etc
@Yaeger I am going to recreate the builder in another region. Letting you know in case you noticed it going MIA in the middle of a deploy.
1 Like
Go ahead
done, try to deploy now please
it works!
Are there still issues with postgres volumes? My app is unable to connect to anything. I have hosts in two separate regions and it’s unreachable still.
yay!
For me it still doesn’t work. Can you do the same trick? Builder machine id 6e825169f3e558
1 Like
Are the slow requests in AMS related to this or is it a different issue?
Soon you will be able to do it by yourself, give me 10 minutes to finish Builder region by dangra · Pull Request #3534 · superfly/flyctl · GitHub
Thanks for the help @dangra our app is back up!
2 Likes
Have y’all gotten to the bottom of this one yet? My machines haven’t been reliably starting for about 24 hours now, this is one of the errors I’m seeing as well. App name is patentmapr-server … I didn’t experience the deployment hangs. Just different errors including:
[PM07] failed to change machine state: machine still active, refusing to start
proxy[9080565ce51948] ord [error] [PM05] failed to connect to machine: gave up after 15 attempts (in 9.327168032s)
proxy[9080565ce51948] ord [error] [PC01] instance refused connection. is your app listening on 0.0.0.0:8080? make sure it is not only listening on 127.0.0.1 (hint: look at your startup logs, servers often print the address they are listening on)
It’s all very weird since none of my deploy configs have changed in months 
Okay, can you just give some quick explanation on how to rebuild the builder in some other region now, to make it work?