Remote builder machines in CDG region are stuck in 'created' state [update: this is a muti-region failure]

1

As of now, remote builder machines cannot be launched in CDG region.

The client side:

> flyctl deploy

==> Verifying app config                                                                                                                                             
Validating fly.toml                                                                                 
Platform: machines                                                                                                                                                   
✓ Configuration is valid                                                                                                                                            
--> Verified app config                                                                                                                                              
==> Building image                                                                                                                                                   
WARN Failed to start remote builder heartbeat: server returned a non-200 status code: 504                                                                                                                                                                                                                                                 
WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-patient-flower-5659`                                                                                                                                                                                                                

Error: failed to fetch an image or build from source: error connecting to docker: remote builder app unavailable

The Fly side:

$ fly logs -a fly-builder-patient-flower-5659

2023-10-30T11:03:18.085 runner[d891241fe11268] cdg [info] Pulling container image docker-hub-mirror.fly.io/flyio/rchab:sha-a4467b8
2023-10-30T11:03:18.699 runner[d891241fe11268] cdg [info] Successfully prepared image docker-hub-mirror.fly.io/flyio/rchab:sha-a4467b8 (613.504136ms)
<EOF>

I have added <EOF> to mark the end of file. The builder machine log contains only two entries.

The Fly status page does not indicate any ongoing incidents.

1 Like
2

+1 (WAW region)

3

I’ve tried to be fancy and checked whether this issue can be worked around by creating a new remote builder machine in another region.

What I did is I used a temporary VPN connection to a server in Netherlands just to check the hypothesis. While fly deploy then indeed created a remote builder machine in the AMS region as expected, the experiment outcome is the same - a freshly created builder machine is just keeps hanging in the created state.

From the looks of it, the deployment outage is a multi-region incident this time.

1 Like
4

I wonder why it isn’t reflected in the fly.io status page

1 Like
5

Thanks for the report. We’re looking into this and have opened an incident on our status page.

2 Likes
6

I wonder why there are always problems with deploy on fly.io… now only local build is working and additionally I need increase timeout time flag and deploy takes 15 minutes or more instead of 5 minutes

7

i love fly so much but the past few days deploys are just failing constantly, app is unreachable, and latency to same region db is terrible

hope team is on it though!

1 Like
8

Thank you, seems to work now. I was able to deploy the apps with both CDG and AMS builders.

9

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.