Failed releases remain pre-promotion with no release-stage metadata

Questions / Help
1

Hi Fly team,

We are diagnosing two failed releases for app b3s-vault in CDG.

Confirmed facts:

  • Release v45 completed and remains live, serving build SHA 232e3bce7bd3c088f8aad4c024986829fc669dd7.
  • Releases v46 and v47 terminated as failed.
  • v47 release identifier: RGb7R06QQkobJSoNP62lqgPl (failed at 2026-08-28T13:46:43Z).
  • Candidate build SHA 1f40cf9d0927ddc6dc83e2162392c0b9be463a1d was never activated.
  • The top-level remote deploy command returned exit 1.
  • The retained Machine remains started/passing and /health reports the old v45 SHA.

The candidate release contract has three chained phases:

  1. build identity verification;
  2. PostgreSQL migration using the app’s immutable target profile;
  3. read-only verification of the migration head.

What we cannot prove:

  • whether the image build completed;
  • whether the release command itself exited non-zero;
  • which internal lifecycle stage failed.

The preserved v47 metadata exposes neither an image reference nor the ephemeral release-command Machine. Generic log output could not be safely correlated to v47, and a previous stage classifier could match a script name merely because it appeared in the full command.

Could Fly staff provide a lifecycle record bound specifically to release RGb7R06QQkobJSoNP62lqgPl, ideally identifying only:

  • lifecycle stage;
  • terminal error class;
  • timestamp?

We need to distinguish among build/image orchestration, release-command execution, and replacement-Machine startup/health before attempting another deploy.

No secrets, credentials, DSNs, raw logs, or customer data are included here.

2

Hi, you can likely get all the information you need from the deploy command logs. Do you have those?

3

Hi. Unfortunately, we did not retain the original stdout/stderr from the v47 deploy command. We only have the top-level exit code, release metadata, and a later log collection that cannot be safely correlated to that release.

Does Fly retain a server-side copy of the deploy output for release RGb7R06QQkobJSoNP62lqgPl? If not, we will need to perform one controlled retry while preserving the complete deploy output.

4

Hi - we do not retain the deploy output if you did a local fly deploy. We may have it if you used the “deploy with github” thing. Let me know.

If not, I agree your best bet is to do a retry on your side and keep the full log (and if you get it, throw your LLM at it, sometimes they can spot what’s wrong from the log).

5

Thanks — we captured the complete output from a controlled v48 retry and found the exact cause.

The image built and pushed successfully. The release-command Machine started, but the first command failed with exit code 2:

python: can't open file '/app/scripts/verify_release_build.py': [Errno 13] Permission denied

Our isolated build context had the scripts directory as 0700 and verify_release_build.py as 0600. Docker preserved those modes, while the entrypoint drops privileges to the b3s user before executing the release command. The successful v45 context used 0755/0644.

Therefore the migration and database verification were never reached. v48 failed safely and the previous v45 Machine remains healthy. We will rebuild from a fresh exact-SHA context with normal readable modes.