504 on deploy, but retry works

1

I deploy my app from circleci, using fly deploy –local-only –build-arg CURRENT_VERSION=${CI_CALCED_VERSION}. This used to work quite reliably, but recently has started failing with errors like:

==> Verifying app config
Validating /home/circleci/jars/fly/fly.toml
Platform: machines
✓ Configuration is valid
--> Verified app config
Error: server returned a non-200 status code: 504

If I retry (which spins up a new container to do the fly deploy), it tends to work immediately afterwards; is there something I can do to make it succeed the first time?

2

Is that the deploy API coming back with 5xx, or your app? The latter can be quite common on application restarts in any cloud system; it means the listener became available before the application finished booting.

Do you get more useful console logs if you add --verbose to your fly command?

3

can you add LOG_LEVEL=debug to the environment to see exactly which api call is 504ing?

4

pretty sure its the deploy api 504’ing and not my app. will try with debug / verbose logging and report back

5 
DEBUG app config loaded from /home/circleci/jars/fly/fly.toml
DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "query ($appName: String!) { appbasic:app(name: $appName) { id name platformVersion organization { id slug rawSlug paidPlan } } }",
  "variables": {
    "appName": "<APP_NAME>"
  }
}


DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (160.75ms)

DEBUG {
  "data": {
    "appbasic": {
      "id": "<ID>",
      "name": "<NAME>",
      "platformVersion": "machines",
      "organization": {
        "id": "<ID>",
        "slug": "personal",
        "rawSlug": "<SLUG>",
        "paidPlan": false
      }
    }
  }
}


==> Verifying app config
Validating /home/circleci/jars/fly/fly.toml
Platform: machines
✓ Configuration is valid
--> Verified app config
DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "query ($appName: String!) { appcompact:app(name: $appName) { id name hostname deployed status appUrl platformVersion organization { id slug paidPlan } postgresAppRole: role { name } imageDetails { repository version } } }",
  "variables": {
    "appName": "<APP_NAME>"
  }
}


DEBUG {}
DEBUG Starting task manager
DEBUG <-- 504 https://api.fly.io/graphql (1m0.05s)

DEBUG <html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>

DEBUG Task manager done
Error: server returned a non-200 status code: 504

Exited with code exit status 1

Note that on retry, this request succeeds but takes ~11 seconds

6

Put that block in code fences if you can @jjjoooeee - the forum is converting the URLs within the console output to page titles.

It looks like your machine or connection is not able to POST to the GraphQL endpoint. Can you try a different connection e.g. a mobile phone hotspot?

7

updated to code block and added some more debug logs from earlier in the log. Pretty sure this is not a connection/network issue, as other requests to the same graphql endpoint return fast (see the request before the failed one), and this returns but slowly on a retry.

8

fwiw was running a pretty old version of flyctl, updated to latest and appears to be no longer using that graphql request, and appears to be deploying successfully on the first try.