Rails deployment fails: "Error failed to fetch an image or build from source: error building: unexpected EOF"

Hi!

I just created a new app on fly.io to deploy a new rails app.
The setup process with the flyctl cli seemed to be smooth.

However I am not able to deploy the app.

Output of fly deploy --remote-only

==> Verifying app config
--> Verified app config
==> Building image
Remote builder fly-builder-icy-sunset-6025 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
Sending build context to Docker daemon  128.6kB
[+] Building 598.7s (0/1)
 => [internal] load remote build context                                          598.7s
Error failed to fetch an image or build from source: error building: unexpected EOF

It is either a very short error message:

Error failed to fetch an image or build from source: error building: unexpected EOF

Or a more detailed one:

Error failed to fetch an image or build from source: error building: error during connect: Post "http://[fdaa:0:49cc:xxx:xxxx:xxxx:xxx:2]:2375/v1.41/build?buildargs=null&buildid=upload-request%somehash&cachefrom=null&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=&labels=null&memory=0&memswap=0&networkmode=&rm=0&shmsize=0&target=&ulimits=null&version=2": EOF

I am not able to make any sense of this error message. It always seems to timeout around 10 minutes.

Any hints are highly appreciated! :slight_smile:


Output of flyctl doctor


Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... PASSED
Pinging WireGuard gateway (give us a sec)... PASSED

Output of flyctl status

App
  Name     = my-app-name
  Owner    = personal
  Version  = 0
  Status   = pending
  Hostname = my-app-name.fly.dev

App has not been deployed yet.

Thanks,

Christian

Do you have Docker installed locally? I’m guessing not by your use of --remote-only but if you do you can bypass any possible issue with remote builders by building the image locally. That shouldn’t be necessary but it can sometimes get you going again.

I do wonder if something may be up given another user today reported an issue that also ends with an unexpected EOF. It’s not entirely clear why that is happening for them either. Hmm. Normally that would be because a response is not being completed or parsed. The status page does not indicate any issues though.

I’m sure someone from Fly will be investigating.

Hi Greg!
thank you for the reply!

You are correct: I do not use Docker locally. The Dockerfile was created by flyctl which also told me to deploy it with the --remote-only flag. :slight_smile:

I might try building it locally as a workaround - thank you for the hint!
However, I would like to learn about another solution as I am not planning to use Docker for local development.

Christian

1 Like

This is still not working. Any updates on this from fly.io side?

I would prefer not to setup Docker locally, aka adjust my local development workflow just for the deployment.

Would a buildpack be the better alternative?

Understandable. I noticed other people having issues with remote builders. It seems like there was a problem on Fly’s end and they reported they’d fixed it:

However you are still getting an error. Of course you could be getting a different, unrelated error with remote building … but the timing suggests it is related.

Do you have any request ID shown? Seems like Fly want that, at least based on that other thread. In the meantime they do track these forums so hopefully they should get back to you.

Thank you for your patience!

I don’t get these kind of IDs in the output of flyctl, I posted the whole output above.

It is normally this one

fly deploy --remote-only --verbose
==> Verifying app config
--> Verified app config
==> Building image
Remote builder fly-builder-icy-sunset-6025 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
[+] Building 0.0s (0/1)
[+] Building 597.9s (0/1)
 => [internal] load remote build context                                        597.9s
Error failed to fetch an image or build from source: error building: unexpected EOF

Trying it again right now, it is either the one above (that takes 10 mins to fail) or the one below (fails almost immediately:

fly deploy --remote-only --verbose
==> Verifying app config
--> Verified app config
==> Building image
Error failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing "personal": context deadline exceeded

It seems like I always connect to the same builder fly-builder-icy-sunset-6025

hi! would you try destroying the remote builder and redeploying? The issues from yesterday with remote builders should be all taken care of, so this can help us rule out issues specific to your particular builder

Hi there!

I destroyed the remote builder, but nothing changed.
However, after I also updated the flyctl cli, it finally “just worked”!! :tada:

I am still not sure what the problem was, but I am happy now :slight_smile:

Thanks!

Christian

1 Like

Hello again!

Sad news: I tried to deploy a second time today and I am facing the same issues again.

Most of the time it will just keep counting seconds during the Building image with Docker stage and will time out after quite a while:

$> flyctl deploy --remote-only --verbose --no-cache
==> Verifying app config
--> Verified app config
==> Building image
Remote builder fly-builder-old-shadow-2352 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
Sending build context to Docker daemon  97.06kB
[+] Building 274.5s (0/1)

One time I got another error message:

$> flyctl deploy --remote-only --verbose --no-cache
==> Verifying app config
--> Verified app config
==> Building image
Error failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing "personal": context deadline exceeded

Destroyed builder app, updated flyctl, still it does not work.

Just for context:
This is a super simple, new Rails app. It has like 2 models and afaik nothing special to it.
I am also happy to provide access to the source code, because I would really be interested in what the problem is.

Are you running any VPN software?

Hi! Thank you for the reply!
No, I don’t.

Hmm weird - now it worked. I basically closed the lid of my laptop in agony, reopened it this morning and it just deployed on the first try.

I would really love to find out what is going on there ;D

Yeah! So EOF means the connection to our API was cut short, one way or another. That could be a result of poor network connectivity. Where are you located?

I am located in Frankfurt/Main, Germany and I think I can say that I have a very good internet connection with no issues at the time of deployment. :slight_smile:

Has it been okay for you since it worked last week? I’m getting the same unexpected EOF error when trying to deploy a barebones Remix app. I’m also using Frankfurt as I’m in Germany.

Update: the only way I’ve been able to deploy was by tethering my MacBook to my iPhone, and using cellular data. When on Wi-Fi, I get the EOF error every time, including when trying different VPN regions.

What do you mean by ‘different VPN regions’? It would be useful to know if you have more success after running:

flyctl wg websockets enable

1 Like

Thanks. I’ll give that a go!

By different VPN regions, I mean I connected to VPN as if I was in various regions — Germany, Austria, etc. — just to see if that made any difference. I was testing different countries as I’d tried everything else I could think of.

Ah OK. So VPNs can conflict with flyctl which uses its own UDP-based VPN (Wireguard). It’s best to try without them where possible. Or, use the websockets change mentioned above, which will run traffic over websockets.

Hi there,

because @Darryl asked: I just tried it again, and the usual deployment command did again not work for me. I am not using a VPN of any kind, so I am still not sure what causes the problem.

However, I applied the websockets setting and now it is working for me! :slight_smile:

2 Likes

That’s great. It’s also working for me since I applied the web sockets setting. Thanks, @jsierles!

I’m encountering the exact same issue: trying to deploy my first Rails app on Fly, not using Docker locally, good internet & no VPN, websockets enabled, builder deleted and rebuilt, and I made sure my firewall was open on UDP port 51820, inbound and outbound. I even tried over my partner’s iPhone hotspot.

In all of the above I’m still getting

fly deploy --remote-only --verbose                                                                         ✹ ✭ 
==> Verifying app config
--> Verified app config
==> Building image
Error failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing "personal": context deadline exceeded

and

fly doctor                                                                                                 ✹ ✭ 
Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... Nope
Pinging WireGuard gateway (give us a sec)... FAILED
(Error: ping gateway: no response from gateway received)

We can't establish connectivity with WireGuard for your personal organization.

WireGuard runs on 51820/udp, which your local network may block.

Any idea what might be going on? I’m at the end of my rope and given that this is my very first time trying to deploy with Fly, I’m inclined to start searching for alternatives.