fly deploy fails as cannot reach rubygems.org

Questions / Help Build debugging
,
1

When running fly deploy to update my existing app it fails on 5/5 with this message:

 => ERROR [base 5/5] RUN gem update --system --no-document &&   gem install -N bundler -v 2.2.33                                                                                                          60.8s
------
 > [base 5/5] RUN gem update --system --no-document &&   gem install -N bundler -v 2.2.33:
#10 60.76 ERROR:  While executing gem ... (Gem::RemoteFetcher::FetchError)
#10 60.76     Net::OpenTimeout: execution expired (https://rubygems.org/specs.4.8.gz)
------
Error failed to fetch an image or build from source: error building: executor failed running [/bin/bash -o pipefail -c gem update --system --no-document &&   gem install -N bundler -v ${BUNDLER_VERSION}]: exit code: 1

Having read a few posts on SO it seems this is to do with the source ISP not resolving IPv6 addresses which ruby is using to try to reach rubygemsorg.

Can anyone help? Ive been trying to deploy an update for almost a week with no success.

2

Can you deploy the first example from this page: Minimal Rails application · Fly Docs ?

I just tried, and I can. If you can deploy this app, then we need to figure out what is different about your app. If you can’t, then there is something different about your network; perhaps there is a regional outage.

If I run fly regions list on my app, I get:

Region Pool: 
iad
Backup Region:

Let us know what region you get, and perhaps try deploying to iad. Not that that’s a fix; but just to help narrow down what the problem is.

3

Hi

Thanks for replying,

This issue just started recently, I was able to deploy the app originally and deploy updates with my last update being about a month ago. I have not changed any of the deployment config since, just updated some of the application code.

No new config to docker or toml and no new gems. just some changes to controllers and views within rails.

This is the output from fly regions list

Region Pool: 
sin
Backup Region:
4

Trying to deploy the minimal rails example it fails trying to install rails with error:

 => [2/5] RUN gem install rails                                                                                                                                                          207.1s
 => => # ERROR:  Could not find a valid gem 'rails' (>= 0), here is why:                                                                                                                       
 => => #           Unable to download data from https://rubygems.org/ - Net::OpenTimeout: Failed to open TCP connection to rubygems.org:443 (execution expired) (https://rubygems.org/specs.4.8
 => => # .gz)
5

I also just tried to deploy it to London rather than Singapore and got the same error.

6

Apparently gems are hosted on Amazon S3 and S3 buckets are hosted in different AWS regions, the default being “US-East”.

7

Apparently the issue is regarding fetching gems via IPv6 rather than IPv4

Is anyone else having this issue, its rendering this service useless at the moment.

8

Hi Rubys

Do you have any suggestions on how to fix this? I’m at a complete stand still here. Any help is appreciated.

9

Unfortunately, I’m not a networking export; I doubt that this has anything to do with IPv6 as that would affect others, but rather some sort of connectivity issue between fly.io’s Singapore site and rubygem’s AWS. But as I said, I’m not a networking expert.

The key doesn’t seem to be where your app is deployed, but rather where the builder is run.

10

@BradChelly I started having this issue on Friday March 4th. My app was setup to run in the lax region (not sure if that matters) and still won’t deploy.

11

And I just tried deploying a brand new rails app to Denver and it failed also. I suppose it’s still using my same builder which is lax so that may be the issue.

12

Deleting my free builder and then running fly deploy again seems to have solved the problem. :slight_smile: FYI @BradChelly

13

Thanks for the info, does deleting the builder and then deploy recreate the builder?

14

Correct. I deleted it via the UI and when I did my next deploy it made a new one.

Interestingly, on a new app that I just “fly launched” I cannot get my builder to pop up in anything but lax (I’m trying dfw) so maybe the deletion trick isn’t as reliable as I thought.

15

The following may be of help:

Not ideal, but might get you up and running for now…