Unable to launch new instances

Hey there,

I’ve been unable to launch new instances today. At the moment, fly launch command stops at the beginning with 504 error.

Here is the output:

Detected a Phoenix app
? Choose an app name (leave blank to generate one): sway-p5
automatically selected personal organization: [my full name]
? Choose a region for deployment: Amsterdam, Netherlands (ams)

Error server returned a non-200 status code: 504

Any ideas what might be causing this ?

Also, in the instances I was able to create earlier today, I was getting these errors:

2023-02-03T20:56:09.302 proxy[9e08548f] ams [error] instance refused connection

2023-02-03T20:56:13.488 proxy[9e08548f] ams [warn] Could not proxy HTTP request. Retrying in 1000 ms (attempt 10)

2023-02-03T20:56:19.477 proxy[9e08548f] ams [error] instance refused connection

2023-02-03T20:56:23.483 proxy[9e08548f] ams [warn] Could not proxy HTTP request. Retrying in 1000 ms (attempt 20)

Any ideas? This has prevented me to share my work today.

Hi @elephantshadow, the deploy issues were due to a brief network outage and the earlier connection issues were due to a separate routing issue. Both incidents have been resolved, things should all be working fine now.

Thanks for the reply but I’m still not able to launch new instances. A fresh launch command produced following output;

==> Monitoring deployment
Logs: https://fly.io/apps/sway-p7/monitoring

 1 desired, 1 placed, 0 healthy, 0 unhealthy [health checks: 1 total, 1 critical]


 1 desired, 1 placed, 0 healthy, 1 unhealthy [health checks: 1 total, 1 critical]
Failed Instances

Failure #1

Instance
ID      	PROCESS	VERSION	REGION	DESIRED	STATUS 	HEALTH CHECKS      	RESTARTS	CREATED
671c48b1	app    	0      	ams   	run    	running	1 total, 1 critical	0       	4m57s ago

Recent Events
TIMESTAMP           	TYPE      	MESSAGE
2023-02-03T22:48:39Z	Received  	Task received by client
2023-02-03T22:48:39Z	Task Setup	Building Task Directory
2023-02-03T22:48:42Z	Started   	Task started by client

--> v0 failed - Failed due to unhealthy allocations - no stable job version to auto revert to and deploying as v1

--> Troubleshooting guide at https://fly.io/docs/getting-started/troubleshooting/
Error abort

In the monitoring page, it shows this message:

“Failed due to unhealthy allocations - no stable job version to auto revert to.”

That’s a separate error somewhere in your application, the configured health check doesn’t seem to be returning the expected result. In any case it’s unrelated to the previous platform-related issues.

Thanks for the response, I didn’t run into this issue until today though. I’m not sure what’s causing the issue though. Here is the output from monitoring:

2023-02-03T23:01:39.142 runner[bf23889b] ams [info] Starting virtual machine

2023-02-03T23:01:39.425 app[bf23889b] ams [info] Starting init (commit: e3cff9e)...

2023-02-03T23:01:39.450 app[bf23889b] ams [info] Preparing to run: `/app/bin/server` as nobody

2023-02-03T23:01:39.467 runner[bf23889b] ams [info] Virtual machine started successfully

2023-02-03T23:01:39.483 app[bf23889b] ams [info] 2023/02/03 23:01:39 listening on [fdaa:0:c354:a7b:c6ef:bf23:889b:2]:22 (DNS: [fdaa::3]:53)

2023-02-03T23:01:51.446 proxy[bf23889b] ams [error] instance refused connection

After some investigation, I made sure that the issue is with Fly.io, not with my app. The health check fails because the app can not become available.

Here are current logs of my app;

2023-02-04T20:10:49.198 proxy[a5c480b8] ams [error] instance refused connection

2023-02-04T20:10:53.391 proxy[a5c480b8] ams [warn] Could not proxy HTTP request. Retrying in 1000 ms (attempt 10)

2023-02-04T20:10:59.349 proxy[a5c480b8] ams [error] instance refused connection

2023-02-04T20:11:03.355 proxy[a5c480b8] ams [warn] Could not proxy HTTP request. Retrying in 1000 ms (attempt 20)

2023-02-04T20:11:10.250 proxy[a5c480b8] ams [error] instance refused connection

2023-02-04T20:11:13.256 proxy[a5c480b8] ams [warn] Could not proxy HTTP request. Retrying in 1000 ms (attempt 30)

2023-02-04T20:11:23.221 proxy[a5c480b8] ams [warn] Could not proxy HTTP request. Retrying in 1000 ms (attempt 40)

Still giving errors, unable to deploy new projects.
And on some older projects, when trying to deploy also giving errors now, example:

==> Verifying app config
--> Verified app config
==> Building image
WARN Failed to start remote builder heartbeat: You hit a Fly API error with request ID: 01GRF4KC0QTTRSJF49EHF73R42-mad
Error failed to fetch an image or build from source: error connecting to docker: You hit a Fly API error with request ID: 01GRF4KCNE6SEYJ53SWN7ZCNWK-mad
2 Likes
instance refused connection

means your app is not listening on 0.0.0.0 or :: (all ip addresses) and therefore is unreachable through its private IP (172.19.x.x)

@joao, I´m on the same boat. Any ideas how to solve this problem?

Same issue here as well

$ fly deploy 
==> Verifying app config
--> Verified app config
==> Building image
WARN Failed to start remote builder heartbeat: You hit a Fly API error with request ID: 01GRF83QBEC1G5DA4HVZGQNY3K-sjc
Error failed to fetch an image or build from source: error connecting to docker: You hit a Fly API error with request ID: 01GRF83QX31ZY9YESAGB7A3H3B-sjc
1 Like

Just posted this status page incident: Fly.io Status - Unable the schedule remote builder causing errors

Users getting errors while starting remote builders should retry.

1 Like

Thanks @jerome! I tried a new deploy and everything is allright now.