Hi, fly deploy seems to be hanging for me out of no where, I successfuly deployed on the 11th and site was live. I went to go to it today and I was stuck permanently loading which prompted me to try re-deploying locally upon which I get the following issues:
Context:
region: sydney,
WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-floral-glade-5750
Reuslt of checking: flyctl logs -a fly-builder-floral-glade-5750
Error: failed to fetch an image or build from source: error connecting to docker: remote builder app unavailable
And if i pull the logs I just see: runner[4d89d9d7f02278] syd [info]Configuring firecracker
When I look at my fly machine logs from the website: I can see the following:
2024-04-10T11:02:34.852 runner[3d8d9652b07618] syd [info] Successfully prepared image registry.fly.io/whats-changed:deployment-01HV3TNW0BSDR5SWPA8VHDWCES (8.242477673s)
2024-04-10T11:02:36.347 runner[3d8d9652b07618] syd [info] Configuring firecracker
2024-04-12T05:28:36.091 proxy[3d8d9652b07618] syd [info] Starting machine
2024-04-12T05:28:36.111 proxy[3d8d9652b07618] syd [error] machines API returned an error: "failed to compare \"/opt/firecracker-v0.24.x/bin/firecracker\" and \"/opt/flyd/firefly/01HV3TRR0Y5D509QF4Q2X4VB2M/root/firecracker\": stat /opt/firecracker-v0.24.x/bin/firecracker: no such file or directory"
2024-04-12T05:28:38.431 proxy[3d8d9652b07618] syd [info] Starting machine
2024-04-12T05:28:40.433 proxy[3d8d9652b07618] syd [error] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
Hey! We had an incident caused by a bad configuration to be pushed to our workers. We deployed a fix, and it looks like machine creation and updates are working again. Can you please check if its working again?
For me, I’ve triggered another fly deploy locally and its progressed nicely but its now stuck on the following:
deployment-01HV8G1PYFN4VJJKM87GKWRD72: digest: sha256:aa9a7719fe1045bed26f6f4744ff3330964e4d999282ed59a9ac6a8d73b5c28a size: 1781
--> Pushing image done
image: registry.fly.io/whats-changed:deployment-01HV8G1PYFN4VJJKM87GKWRD72
image size: 384 MB
Watch your deployment at https://fly.io/apps/whats-changed/monitoring
-------
Updating existing machines in 'whats-changed' with rolling strategy
-------
✖ [1/2] Machine 148e5375ad6108 [app] update failed: timeout reached waiting for machine's state to change
[2/2] Waiting for job
-------
Checking DNS configuration for whats-changed.fly.dev
Error: timeout reached waiting for machine's state to change
Your machine was created, but never started. This could mean that your app is taking a long time to start,
but it could be indicative of a region issue.
You can try deploying to a different region,
or you can try increasing the timeout with the --wait-timeout flag
When I view the logs in fly.io dashboard for that particular machine I see the following:
2024-04-12T06:30:19.222 proxy[148e5375ad6108] syd [error] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2024-04-12T06:30:23.395 proxy[148e5375ad6108] syd [error] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2024-04-12T06:30:28.478 proxy[148e5375ad6108] syd [error] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2024-04-12T06:34:56.413 runner[148e5375ad6108] syd [info] Pulling container image registry.fly.io/whats-changed:deployment-01HV8G1PYFN4VJJKM87GKWRD72
2024-04-12T06:35:05.972 runner[148e5375ad6108] syd [info] Successfully prepared image registry.fly.io/whats-changed:deployment-01HV8G1PYFN4VJJKM87GKWRD72 (9.559451517s)
2024-04-12T06:35:07.430 runner[148e5375ad6108] syd [info] Configuring firecracker
