Fly app servers down!

Just to confirm, running fly m destroy --force to destroy all my machines, followed by a fly deploy to re-redeploy my app did the trick for all my apps too.

Is there any other solution? I cannot destroy the machine as it would mean losing a lot of data that has not been backed-up. Is a fix going to be deployed to the alredy working machines? I’m really in a crunch time situation.

destroyed instances and the builder. Redeployed and now the apps are in a reboot loop continously restarting :dizzy_face:


I re-deployed the app but now I need to bind the volume I had with the previous machine (it contains most of my data):

fly machine clone NEW_WORKING_MACHINE -r ams --attach-volume MY_PREVIOUS_VOLUME_ID

Error: failed to launch VM: unable to use requested volume, ‘MY_PREVIOUS_VOLUME_ID’ due to capacity constraints (Request ID: …-fra)

Even though on the dashboard it shows that i’m using 335mb/2gb

I get the same error (after destroying all the previous machines) when using:

fly scale count 1

Any other ideas on how to bind the previous volume to a new machine/existing one? Can someone please give me some feedback? This problem is really messing up my work, might get fired lol

Hey @gdrose ! Would you mind to write to support@fly.io and share you machine and volumes details? I can take a look

1 Like

Sent. I got back a “This is an unmonitored support mailbox […]”, did you receive my email?. Thank you so much for your time.

my machines is stuck in a restart loop it keeps failing , even the issue seem to be resolved as per the dashboard. How to fix my app here ?

Hey @dar ! I’ll take a look and update you here. Thanks for your patience.

1 Like

any update ?

Hey @dar,

Looks like it’s something wrong with how we pass the image ref to variable FLY_IMAGE_REF. that variable is used to build the node name and is invalid because it includes two @ symbols.

We are investigating how this is happening to get it fixed.

In the meantime, a new deploy may fix the problem for you. Could you please try that and let us know while we try to figure out the FLY_IMAGE_REF bug?

1 Like

My app is still stuck in suspended and can’t get my machines running anymore, I tried redeploying on new machines and still can’t get it to work.

Well this was a very unlucky day. All 8 apps were down and didn’t work anymore.
Had to fly m destroy all instances separate and then fly deploy.

For you info we have min machines to 0 and auto_stop and auto_start machines

To be honest my trust is now a bit less in Fly.io

I hope Fly will come with a good debrief and fix for the future. Else Fly.io is not good enough for production yet.

1 Like

Thanks @aschiavo it has worked for now

Great news! We are looking into this edge case and should deploy a fix for the imageRef soon.

Thanks for your patience.

Hey @adamgiebl !

Error: failed to spawn command: /app/docker-entrypoint.js npm run start: Permission denied (os error 13)
does `/app/docker-entrypoint.js` exist and is it executable?

Could you please confirm if that file is present and executable?

Thanks!

Hey, it is, I haven’t changed anything since the machines went down.

any updates? My entrypoint file is there and has been working for the last 3 months.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.