422 Error Code on Fly Deploy

Phoenix
, ,
1

Hi,

I’ve made changes to my deployed database configuration, and now fly deploy is failing, not providing logs, and providing a 422 error code. See below:

Running app-staging release_command: /app/bin/migrate

-------
 ✖ release_command failed
-------
Error release_command failed running on machine <machine code> with exit code 1.
Check its logs: here's the last 100 lines below, or run 'fly logs -i <machine code>':
-------
Error: release command failed - aborting deployment. error getting release_command logs: 422 Unprocessable Entity

I go into the console and try to lok at the logs from the machine that was supposed to setup this deploy, and I just get this:

$ fly logs -a app-staging -i <machine code>

Waiting for logs...

I get the same when running the fly logs -i ... in my own terminal.

Any ideas for how to unstick this?

2

I am getting the same error with our NextJS application for FlyIO Apps V2.

Our FLyIO Application has been deploying just fine for some time now, and only broke once we introduced the release_command into our fly.toml:

app = 'arch1ve-web'
primary_region = 'iad'
kill_signal = "SIGINT"
kill_timeout = "5s"
[build]
dockerfile = "Dockerfile"
[deploy]
release_command = "echo 'hello world'"

With this simple release_command, our application will fail to deploy, offering the following error similar to @RoboZoom:

✖ Failed: error getting release_command logs: 422 Unprocessable Entity
Error: release command failed - aborting deployment. error getting release_command logs: 422 Unprocessable Entity

Checking the Grafana Logs dashboard adds a little more color, showing this might be related to some networking or the Firecracker runtime, but still not very helpful:

 INFO [fly api proxy] listening at /.fly/api
 INFO Preparing to run: `docker-entrypoint.sh cd ../../packages/db && echo 'hello world'` as root
 INFO Starting init (commit: 04656915)...

Similar to @RoboZoom, we are not able to get any more context on our logs via the fly logs command for this app.

To help the FlyIO team or other community members gain more context to our application, we can share that the Dockerfile was always working up until this recent addition, and has a final lines to start the app as:

RUN pnpm install --frozen-lockfile
ENV NODE_ENV production
EXPOSE 3000
WORKDIR /app/apps/web
CMD ["pnpm", "start"]

Furthermore, we have tried and verified the exemplary failing release command on a locally built docker container, as well as the intended release command that we will soon replace echo 'hello world' with (a migration script), both of which worked just fine on our local Docker container.

  • Is this a FlyIO platform error? If so then why is this error not reported on the FlyIO Status page?
  • Does this have to do with the instance being in the iad region specifically? One or two other forum posts seemed to have this as a potential root cause w/DNS in iad
  • How can one monitor-for and prevent this critical failure in the future if something like production database migrations depends on it?

Thank you for any help here!

3

yep fly messed up again. lets just wait for them to fix.

4

There’s no platform-wide issue here, but we’re looking at the report that came in via support.

Can you link to the post in question?

5

Same issue here, no recent changes to our fly script though. It was working fine last week.

6

We’ve resolved the migration issue by setting up other fake environments to trigger it and find the problem.

The fly issue seems to be that the failed logs weren’t being fetched, and we see that error getting release_command logs: 422 Unprocessable Entity error.

7

Yep, the issue is that log fetching is failing. fly logs still works for streaming logs, but fetching backlog is currently not working. We’re looking into it.

8

Added logs

9

I believe we are facing the same problem here… The team was unable to deploy since Friday.

Edit: On our scenario I’ve traced it back to using --verbose option with fly deploy. Without any real error on the release_command. We use that to show its fly logs, so we can see the results of successful database migrations that have been run.

10

The 422 errors coming from log fetching should be fixed now, both on release commands and fly logs.

1 Like
11

Can confirm this has stopped blocking our deploys :+1:.