Unable to pull image, not found, canceling deploy?

1

My deploys claim to succeed, but then the machine is still running the old code.
Logs at the end of the deploy:

Pushed 
deployment-01KXKVEASHJEENM2S4NX23NM2P: digest: sha256:ff8b400eb5243a37c66ba9cc6df80d2de5ccc995d6eaa181945b2bc159168c8a size: 1376
--> Pushing image done
image: registry.fly.io/app:deployment-01KXKVEASHJEENM2S4NX23NM2P
image size: 396 MB

Watch your deployment at https://fly.io/apps/app/monitoring

-------
Updating existing machines in 'app' with rolling strategy
-------
Checking DNS configuration for app.fly.dev
✓ DNS configuration verified

Visit your newly deployed app at https://app.fly.dev/

Watching logs as I attempt a deploy, i see:

 21:38:30 Pulling container image registry.fly.io/app@sha256:ff8b400eb5243a37c66ba9cc6df80d2de5ccc995d6eaa181945b2bc159168c8a
21:38:31 Pulling container image registry.fly.io/app@sha256:ff8b400eb5243a37c66ba9cc6df80d2de5ccc995d6eaa181945b2bc159168c8a
21:38:34 Pulling container image registry.fly.io/app:deployment-01KXKVEASHJEENM2S4NX23NM2P
21:38:36 Pulling container image registry.fly.io/app:deployment-01KXKVEASHJEENM2S4NX23NM2P
21:38:37 Pulling container image registry.fly.io/app:deployment-01KXKVEASHJEENM2S4NX23NM2P
21:38:42 Unable to pull image, not found, canceling deploy

However locally if I fly docker login I can then do a docker pull registry.fly.io/app:deployment-01KXKVEASHJEENM2S4NX23NM2P and it successfully pulls an image. What can I look into to see why the fly machine can’t pull?

Also the web UI shows my app as ‘suspended’ but it’s still serving successfully when I call it, just doing so from the old image.

3

Hi… The “suspended” label is just the dashboard’s way of saying that you have no Machines currently running. (That phrasing confuses a lot of people.)

If you have auto-stop or auto-suspend enabled, then it’s (weird but) normal.

It’s the underlying physical host that does the pull, so there’s unfortunately nothing you can check directly (as far as I know).

As far as indirect measures go… Since you can pull it locally, I’d try creating a new Machine in a different region. Fly.io implemented a kind of region-specific cache recently, and two other users were reporting problems with it earlier today (although not exactly the same behavior).

It might also help to post the full fly.toml and the full output of fly m list and fly logs. From the edited logs that you gave above, it’s not clear whether that’s 6 Machines with only the last one failing, or a single Machine with all of its attempts timing out…

Aside: You can double-check Machine details, including image, with fly m status -d.

4

fly.toml:

app = "app"
primary_region = "sjc"
kill_signal = "SIGINT"
kill_timeout = "5s"
swap_size_mb = 512

[[services]]
  protocol = "tcp"
  internal_port = 8080

  [[services.ports]]
    port = 80
    handlers = ["http"]
    force_https = true

  [[services.ports]]
    port = 443
    handlers = ["tls", "http"]
  [services.concurrency]
    type = "connections"
    hard_limit = 25
    soft_limit = 20

  [[services.tcp_checks]]
    interval = "10s"
    timeout = "2s"
    grace_period = "1s"
    restart_limit = 0

[metrics]
  port = 8080
  path = "/admin/prometheus"

fly m list:

1 machines have been retrieved from app app.
View them in the UI here (https://fly.io/apps/app/machines/)

app
 ID             │ NAME                    │ STATE   │ CHECKS │ REGION │ ROLE │ IMAGE                                         │ IP ADDRESS                      │ VOLUME │ CREATED              │ LAST UPDATED         │ PROCESS GROUP │ SIZE
 e784e3d1b47518 │ fragrant-resonance-6414 │ started │ 1/1    │ sjc    │      │ app:deployment-01KWCZHHD9H17WRCJEYT12YSKA │ fdaa:0:86d5:a7b:122:9b77:43a4:2 │        │ 2026-03-04T21:14:48Z │ 2026-07-15T21:38:42Z │ app           │ shared-cpu-1x:256MB

the logs from above were from the single machine.

One thing I see in fly m status -d (beyond the image still being the old one):

  "env": {
    "FLY_PROCESS_GROUP": "app",
    "PRIMARY_REGION": "sea"
  }

vs/ the sjc region – could something have gone weird when sea got folded into sjc? if so is cloning the machine directy into sjc and shutting down the current one the way to go? or something else?

5

It’s the underlying physical host that does the pull, so there’s unfortunately nothing you can check directly (as far as I know).

That’s right, and that’s also where the problem was: The physical host’s containerd broke. We’re in the process of migrating all of this host’s machines to a new host, so this should be fixed in a bit. Sorry about the trouble!

6

ok - i ended up spinning up a new machine (likely on a separate underlying host) successfully, but good to know it would have gotten fixed