`fly deploy` stuck at pulling container image

1

I’ve seen bunch of posts about this from the past but nothing has resolved the issue for me. This is my first time using fly.io. fly deploy is repeatedly failing with:

-------
 ✖ Machine 3d8d7419b41128 [app] update failed: failed to update VM 3d8d7419b41128: request returned non-2xx status, 504
-------
Checking DNS configuration for kpbj-fm.fly.dev
Error: failed to update VM 3d8d7419b41128: request returned non-2xx status, 504 (Request ID: 01HVFXGF8SDS1DTCP070NG489A-lax)

And fly logs is full of:

2024-04-15T03:44:25Z runner[3d8d7419b41128] sjc [info]Pulling container image docker-hub-mirror.fly.io/sbothwell/kpbj.fm:0.1.0.1
2024-04-15T03:44:27Z runner[3d8d7419b41128] sjc [info]Pulling container image docker-hub-mirror.fly.io/sbothwell/kpbj.fm:0.1.0.1
2024-04-15T03:44:33Z runner[3d8d7419b41128] sjc [info]Pulling container image docker-hub-mirror.fly.io/sbothwell/kpbj.fm:0.1.0.1
2024-04-15T03:44:37Z runner[3d8d7419b41128] sjc [info]Pulling container image docker-hub-mirror.fly.io/sbothwell/kpbj.fm:0.1.0.1
2024-04-15T03:44:40Z runner[3d8d7419b41128] sjc [info]Pulling container image docker-hub-mirror.fly.io/sbothwell/kpbj.fm:0.1.0.1
2024-04-15T03:44:45Z runner[3d8d7419b41128] sjc [info]Pulling container image docker-hub-mirror.fly.io/sbothwell/kpbj.fm:0.1.0.1
2024-04-15T03:44:51Z runner[3d8d7419b41128] sjc [info]Pulling container image docker-hub-mirror.fly.io/sbothwell/kpbj.fm:0.1.0.1

I’ve tried deleting the machine and switching regions to no effect. In fact, the error message above references lax but this message was from AFTER switching to sjc, which seems very suspicious.

2

Have you tried deleting your fly builder? ran in to this non stop and this always fixed things. I just use BuildJet to build my images and deploy them to fly in my GitHub Actions workflows now without any issues

3

The fly io docs say a builder is a dockerfile, buildpack or image: Builders and Fly · Fly Docs

What does it mean to delete my fly builder?

I’m using a container I built with nix and pushed manually to docker hub.

4

Hi @solomon—by chance do you know what the ENTRYPOINT of your image is? Based on our internal logs, it seems like the image unpack is failing and retrying because the ENTRYPOINT is one string instead of an array of strings. I’m not very familiar with this, but a quick web search makes it sound like Docker accepts a single string ENTRYPOINT but other tools don’t. (FWIW, we use containerd to pull and unpack images.)

In any case, it’s definitely a frustrating UX that this fails silently, so thank you for bringing this to our attention!

5

Thanks, that fixed it. How can I get observability into errors like this?

6

I’m not sure if this should be a different thread but I’m getting bad requests sent to my app:

2024-04-15T17:42:44Z app[91852ed0c61178] lax [info]{"component":"kpbj-backend","domain":[],"time":"2024-04-15T17:42:44.953033297Z","level":"info","message":"Request","data":{"headers":[],"httpVersion":"HTTP/1.0","isSecure":false,"method":"GET","path":"","queryString":[],"remoteHost":"172.16.13.74:38196","requestBody":"","requestSize":134,"responseStatusCode":400}}

If I run the same curl against the container locally then I get a success. The path field on the request being an empty string looks suspicious.

7

Sorry for the delayed response! We just deployed a change so that image configuration errors like this will cause deployment to fail, and you’ll see an error message in fly logs rather than an image pull crash-loop. Hopefully this will make it clear what’s happening in the future.

8

Thank you! Its really cool to see you deploy a fix like this so quickly and in response to a user issue.

9

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.