Failing to pull docker image during deploy?

Questions / Help
1

Hi all,

New fly user here :smile:. I am trying to deploy GitHub - luc-tielen/bakery: Serving freshly baked Eclair programs over HTTP to fly.io for a demo, but I am getting an error about it not being able to pull the image?

Here’s an excerpt of the logs:

2022-11-21T22:04:15Z runner[d55547ee] ams [info]Starting instance
2022-11-21T22:04:16Z runner[d55547ee] ams [info]Configuring virtual machine
2022-11-21T22:04:16Z runner[d55547ee] ams [info]Pulling container image
2022-11-21T22:04:18Z runner[d55547ee] ams [info]Unpacking image
2022-11-21T22:05:05Z runner[d55547ee] ams [info]Pull failed, retrying (attempt #0)
2022-11-21T22:05:06Z runner[d55547ee] ams [info]Unpacking image
2022-11-21T22:05:08Z runner[d55547ee] ams [info]Pull failed, retrying (attempt #1)
2022-11-21T22:05:09Z runner[d55547ee] ams [info]Unpacking image
2022-11-21T22:05:11Z runner[d55547ee] ams [info]Pull failed, retrying (attempt #2)
2022-11-21T22:05:11Z runner[d55547ee] ams [info]Pulling image failed

I tried a bunch of tweaking of settings, but can’t seem to figure out what’s exactly wrong.
Also, I double checked that the docker image works locally.

Any help would be welcome!

2

Hi!

Try out LOG_LEVEL=debug fly deploy (I’m assuming you’re running the deploy command there), it will output the API calls, etc, that are being made and may provide some more useful information.

3

This is what the command outputs (I removed all extra duplicate logs)

{
  "query": "query ($appName: String!, $deploymentId: ID!, $evaluationId: String!) { app(name: $appName) { deploymentStatus(id: $deploymentId, evaluationId: $evaluationId) { id inProgress status successful description version desiredCount placedCount healthyCount unhealthyCount allocations { id idShort status region desiredStatus version healthy failed canary restarts checks { status serviceName } } } } }",
  "variables": {
    "appName": "bakery",
    "deploymentId": "3e1c17a4-8e8d-dcb5-d4db-84b3f124b0bc",
    "evaluationId": "2d77eba8-7977-fbf7-8665-b241a7d1689b"
  }
}

DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (377.1ms)

{
  "data": {
    "app": {
      "deploymentStatus": {
        "id": "3e1c17a4-8e8d-dcb5-d4db-84b3f124b0bc",
        "inProgress": true,
        "status": "running",
        "successful": false,
        "description": "Deployment is running",
        "version": 9,
        "desiredCount": 1,
        "placedCount": 1,
        "healthyCount": 0,
        "unhealthyCount": 1,
        "allocations": [
          {
            "id": "0d3bcd2c-adc8-6a0e-4dab-23eb2bb377d0",
            "idShort": "0d3bcd2c",
            "status": "pending",
            "region": "ams",
            "desiredStatus": "run",
            "version": 9,
            "healthy": true,
            "failed": false,
            "canary": false,
            "restarts": 0,
            "checks": []
          }
        ]
      }
    }
  }
 1 desired, 1 placed, 0 healthy, 1 unhealthy
DEBUG --> POST https://api.fly.io/graphql

{
  "query": "query ($appName: String!, $deploymentId: ID!, $evaluationId: String!) { app(name: $appName) { deploymentStatus(id: $deploymentId, evaluationId: $evaluationId) { id inProgress status successful description version desiredCount placedCount healthyCount unhealthyCount allocations { id idShort status region desiredStatus version healthy failed canary restarts checks { status serviceName } } } } }",
  "variables": {
    "appName": "bakery",
    "deploymentId": "3e1c17a4-8e8d-dcb5-d4db-84b3f124b0bc",
    "evaluationId": "2d77eba8-7977-fbf7-8665-b241a7d1689b"
  }
}

DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (367.9ms)

{
  "data": {
    "app": {
      "deploymentStatus": {
        "id": "3e1c17a4-8e8d-dcb5-d4db-84b3f124b0bc",
        "inProgress": false,
        "status": "failed",
        "successful": false,
        "description": "Failed due to unhealthy allocations - no stable job version to auto revert to",
        "version": 9,
        "desiredCount": 1,
        "placedCount": 1,
        "healthyCount": 0,
        "unhealthyCount": 1,
        "allocations": [
          {
            "id": "0d3bcd2c-adc8-6a0e-4dab-23eb2bb377d0",
            "idShort": "0d3bcd2c",
            "status": "pending",
            "region": "ams",
            "desiredStatus": "run",
            "version": 9,
            "healthy": true,
            "failed": false,
            "canary": false,
            "restarts": 0,
            "checks": []
          }
        ]
      }
    }
  }
 1 desired, 1 placed, 0 healthy, 1 unhealthy
--> v9 failed - Failed due to unhealthy allocations - no stable job version to auto revert to and deploying as v10 

--> Troubleshooting guide at https://fly.io/docs/getting-started/troubleshooting/
Error abort

Seems like my image is unhealthy? How does fly determine this?
My image exposes port 8080, and this is also configured as the “internal_port” in the fly.toml (see below). It’s an express server that calls app.listen(8080, '0.0.0.0').

# fly.toml file generated for bakery on 2022-11-21T22:56:54+01:00

app = "bakery"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 8080
  processes = ["app"]
  protocol = "tcp"
  script_checks = []
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "30s"
    interval = "15s"
    restart_limit = 6
    timeout = "2s"
    port = "8080"
4

Hey,

your fly.toml and everything else looks good. You should make sure the app listens to both ipv4 and ipv6 with app.listen(8080, '::') instead of app.listen(8080, '0.0.0.0').

5

Hi @rugwiro, I applied your suggestion, but unfortunately still the same result :sweat_smile:

Any other things it could be? Is there a max image size or something? (It is kind of a big image, and I can’t shrink it down…)

6

Yeah, the image is definitely big. You should trim the fat down if you can. Has any of your attempted deploys gone beyond the pulling image stage?

7

It’s not possible unfortunately… It’s an exotic image that needs both the LLVM and Haskell toolchain, as well as node for the webserver :sweat_smile:

And no, always failed at the pulling step.

8

Our root devices are limited to 8GB I believe. If your image, uncompressed, ends up bigger than this, we just can’t extract it to the root drive for your VM.

I see that might be what’s happening here.

3 Likes
9

@jerome you were right! I managed to trim down the image significantly by directly copying over dynamic libs in a multistage build. It is running now!

Thanks so much everyone!

2 Likes