Dead App, stuck: Fly Scale not unsticking dead app

Context

  • I have a -prod app that i’ve been able to run quite a while.
  • I have decided to deploy a -dev clone of the prod app.
  • I created the app in my organization, which was shared cpu 256 mb scale.
  • It failed to deploy, same exact code
  • It was stuck pending mode and failed allocation
  • I tried to “unstick” it, in which I was able to unstick other apps as a indicated in a prior discourse thread . But scaling to dedicated cpu did not unstick it.

Issue

  • I have my btwb-sts-dev app stuck in dead state despite attempts to scale and redeploy.

An Aside

  • Is there a direct support email? I would generally a non-public forum method of communication about the specific details of support/ops issues for priced-metered apps I have deployed? thanks

Did you set and then delete a secret in this app by chance? It looks like it’s failing to start because it’s expecting a secret that doesn’t exist.

  1. the prior secret i set and then deleted was a dangling orphaned secret that did not need to be loaded; this was intentional.
  2. Doing a fly logs with the -dev doesn’t return any information; same with looking at logs via the console
  3. Generally, with missing secrets, I’ve seen such errors in the logs when running the app locally; same with having done it the -prod app.
  4. I’ve double check the diff between my fly.dev.toml and fly.toml (prod), and that each toml is specifying the correct list of secrets to lookup.
  5. diff fly.dev.toml fly.toml is returning the very minimal changes to just change hostnames to -dev and pointing to the respective STS_RSA_KEY_1 vs STS_DEV_RSA_KEY_1 .
  6. I’ve double checked the list of -dev vs -prod fly secrets set. The MD5 sums of each secret matches except for the intentionally different SECRET_KEY_BASE and respective RSA Keys.
  7. I can’t see anywhere that it would be a missing secret key given fly tomls and secrets list are the same (except for the intentional differences).

earlier today when I initially tried to deploy -dev, I did a parallel deploy using what was basically the same fly.toml, and the same code base, I successfully deployed the -prod

I just tried to redeploy -dev just now with the double checked tomls and secrets, and again got failure w/o any log output.

I’d love the issue to having been missing/incorrect secrets, but I don’t have any fly logs -a appname to confirm or deny that, or even show startup/restart attempts.

I think this is just a corrupted internal app config. You can delete it and create a new one, if you’d like, and it should work.

What’s failing is not actually in your app, it’s in our infrastructure. When we create your VM it’s attempting to use the secret you deleted. This is definitely a bug, it’ll just take us a bit to get it fixed.

Also I’m a big fan of 0 indexed lists. :slight_smile:

ok, I destroyed and recreated the app.

:sunny: Thanks Kurt, it worked! :sunflower:

/insert joke about us users always finding a way to break the system

1 Like

I don’t think you can take credit for that one, I’m pretty sure I wrote the code that made that happen. :slight_smile:

1 Like