Cannot deploy, getting 504s from Fly API

[pheuter]

Run flyctl deploy --remote-only --config ./apps/prod/fly.toml
==> Verifying app config
Error server returned a non-200 status code: 504

[martinvanaken]

Same.

[bkspace]

Yep…production broken again. How frustrating.

[wjordan]

Hello, we experienced an issue with our deployment cluster that caused a ~10 minute interruption, it should be stable again so you can continue your deployments. Sorry for the inconvenience!

Deployment issues

[pheuter]

Appreciate the rapid response, releases appear to be going through now but stuck/taking much longer than usual on health checks, confirming if that’s expected.

[userbradley]

I’m having issues in LHR still

[ckarpyszyn]

This says it’s resolved on the status page, but I’m still having deployments stuck at pending. @wjordan Do you have an update? This is blocking a critical path to hotfixing or application today.

[bkspace]

This has just reoccured again. It has managed to start one instance thankfully. Multi region + multi instance taken down without a re-deployment seems extremely sketchy.

[taylrd]

Im also having deployment failures in the MIA region. Also, the fly.io dashboard will not load.

[tofran]

We are also having issues: the dashboard does not load and newly deployed apps are returning 503

[ckarpyszyn]

I just got a successful deploy to LAX

[tofran]

Here it also went back to normal.

[bkspace]

For us, it’s continuously redeploying:

We haven’t re-deployed once

[wjordan]

Hi folks, sorry for the inconvenience. The service became unstable again and caused a second interruption to deployments. It should be once again stable, and we’re still actively monitoring and investigating the root cause to try and prevent any further issues.

Deployment issues

[wjordan]

@bkspace you’re experiencing an unrelated application issue- your app is exiting with an out of memory error which is why it keeps restarting (apps will automatically restart after they crash, and re-deploy if they crash quickly). You can find the exact error in your app logs.

[abuznego]

@wjordan I have not been able to deploy all day.

This is what I see after it takes super long to build:

[+] Building 815.4s (1/1) FINISHED
 => ERROR [internal] load remote build context                                                                                                                                                                   815.4s
------
 > [internal] load remote build context:
------
Error failed to fetch an image or build from source: error building: error during connect

I’ve also tried with --local-only

[bkspace]

Thank you, we’re looking into that. Strange as the app has been running smoothly for weeks afaik - perhaps all the traffic going to one instance, due to the restarts, is causing the issue.