Failing to pull docker image during deploy?

Hi all,

New fly user here :smile:. I am trying to deploy GitHub - luc-tielen/bakery: Serving freshly baked Eclair programs over HTTP to fly.io for a demo, but I am getting an error about it not being able to pull the image?

Here’s an excerpt of the logs:

2022-11-21T22:04:15Z runner[d55547ee] ams [info]Starting instance
2022-11-21T22:04:16Z runner[d55547ee] ams [info]Configuring virtual machine
2022-11-21T22:04:16Z runner[d55547ee] ams [info]Pulling container image
2022-11-21T22:04:18Z runner[d55547ee] ams [info]Unpacking image
2022-11-21T22:05:05Z runner[d55547ee] ams [info]Pull failed, retrying (attempt #0)
2022-11-21T22:05:06Z runner[d55547ee] ams [info]Unpacking image
2022-11-21T22:05:08Z runner[d55547ee] ams [info]Pull failed, retrying (attempt #1)
2022-11-21T22:05:09Z runner[d55547ee] ams [info]Unpacking image
2022-11-21T22:05:11Z runner[d55547ee] ams [info]Pull failed, retrying (attempt #2)
2022-11-21T22:05:11Z runner[d55547ee] ams [info]Pulling image failed

I tried a bunch of tweaking of settings, but can’t seem to figure out what’s exactly wrong.
Also, I double checked that the docker image works locally.

Any help would be welcome!

Hi!

Try out LOG_LEVEL=debug fly deploy (I’m assuming you’re running the deploy command there), it will output the API calls, etc, that are being made and may provide some more useful information.

This is what the command outputs (I removed all extra duplicate logs)

{
  "query": "query ($appName: String!, $deploymentId: ID!, $evaluationId: String!) { app(name: $appName) { deploymentStatus(id: $deploymentId, evaluationId: $evaluationId) { id inProgress status successful description version desiredCount placedCount healthyCount unhealthyCount allocations { id idShort status region desiredStatus version healthy failed canary restarts checks { status serviceName } } } } }",
  "variables": {
    "appName": "bakery",
    "deploymentId": "3e1c17a4-8e8d-dcb5-d4db-84b3f124b0bc",
    "evaluationId": "2d77eba8-7977-fbf7-8665-b241a7d1689b"
  }
}

DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (377.1ms)

{
  "data": {
    "app": {
      "deploymentStatus": {
        "id": "3e1c17a4-8e8d-dcb5-d4db-84b3f124b0bc",
        "inProgress": true,
        "status": "running",
        "successful": false,
        "description": "Deployment is running",
        "version": 9,
        "desiredCount": 1,
        "placedCount": 1,
        "healthyCount": 0,
        "unhealthyCount": 1,
        "allocations": [
          {
            "id": "0d3bcd2c-adc8-6a0e-4dab-23eb2bb377d0",
            "idShort": "0d3bcd2c",
            "status": "pending",
            "region": "ams",
            "desiredStatus": "run",
            "version": 9,
            "healthy": true,
            "failed": false,
            "canary": false,
            "restarts": 0,
            "checks": []
          }
        ]
      }
    }
  }
 1 desired, 1 placed, 0 healthy, 1 unhealthy
DEBUG --> POST https://api.fly.io/graphql

{
  "query": "query ($appName: String!, $deploymentId: ID!, $evaluationId: String!) { app(name: $appName) { deploymentStatus(id: $deploymentId, evaluationId: $evaluationId) { id inProgress status successful description version desiredCount placedCount healthyCount unhealthyCount allocations { id idShort status region desiredStatus version healthy failed canary restarts checks { status serviceName } } } } }",
  "variables": {
    "appName": "bakery",
    "deploymentId": "3e1c17a4-8e8d-dcb5-d4db-84b3f124b0bc",
    "evaluationId": "2d77eba8-7977-fbf7-8665-b241a7d1689b"
  }
}

DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (367.9ms)

{
  "data": {
    "app": {
      "deploymentStatus": {
        "id": "3e1c17a4-8e8d-dcb5-d4db-84b3f124b0bc",
        "inProgress": false,
        "status": "failed",
        "successful": false,
        "description": "Failed due to unhealthy allocations - no stable job version to auto revert to",
        "version": 9,
        "desiredCount": 1,
        "placedCount": 1,
        "healthyCount": 0,
        "unhealthyCount": 1,
        "allocations": [
          {
            "id": "0d3bcd2c-adc8-6a0e-4dab-23eb2bb377d0",
            "idShort": "0d3bcd2c",
            "status": "pending",
            "region": "ams",
            "desiredStatus": "run",
            "version": 9,
            "healthy": true,
            "failed": false,
            "canary": false,
            "restarts": 0,
            "checks": []
          }
        ]
      }
    }
  }
 1 desired, 1 placed, 0 healthy, 1 unhealthy
--> v9 failed - Failed due to unhealthy allocations - no stable job version to auto revert to and deploying as v10 

--> Troubleshooting guide at https://fly.io/docs/getting-started/troubleshooting/
Error abort

Seems like my image is unhealthy? How does fly determine this?
My image exposes port 8080, and this is also configured as the “internal_port” in the fly.toml (see below). It’s an express server that calls app.listen(8080, '0.0.0.0').

# fly.toml file generated for bakery on 2022-11-21T22:56:54+01:00

app = "bakery"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 8080
  processes = ["app"]
  protocol = "tcp"
  script_checks = []
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "30s"
    interval = "15s"
    restart_limit = 6
    timeout = "2s"
    port = "8080"

Hey,

your fly.toml and everything else looks good. You should make sure the app listens to both ipv4 and ipv6 with app.listen(8080, '::') instead of app.listen(8080, '0.0.0.0').

Hi @rugwiro, I applied your suggestion, but unfortunately still the same result :sweat_smile:

Any other things it could be? Is there a max image size or something? (It is kind of a big image, and I can’t shrink it down…)

Yeah, the image is definitely big. You should trim the fat down if you can. Has any of your attempted deploys gone beyond the pulling image stage?

It’s not possible unfortunately… It’s an exotic image that needs both the LLVM and Haskell toolchain, as well as node for the webserver :sweat_smile:

And no, always failed at the pulling step.

Our root devices are limited to 8GB I believe. If your image, uncompressed, ends up bigger than this, we just can’t extract it to the root drive for your VM.

I see that might be what’s happening here.

2 Likes

@jerome you were right! I managed to trim down the image significantly by directly copying over dynamic libs in a multistage build. It is running now!

Thanks so much everyone!

2 Likes