==> Verifying app config
Validating apps/bullet/fly.develop.toml
✓ Configuration is valid
--> Verified app config
==> Building image
WARN Failed to start remote builder heartbeat: failed to restart VM e7843274cd2028: failed_precondition: unable to restart machine, not currently started or stopped
==> Building image
WARN Failed to start remote builder heartbeat: failed to restart VM e7843274cd2028: resource_exhausted: rate limit exceeded
Error: failed to fetch an image or build from source: failed to restart VM e7843274cd2028: resource_exhausted: rate limit exceeded (Request ID: 01J3M7VWV23W35JD81KAEVK2GQ-iad) (Trace ID: 2e7828453fdfde1aa213f71ad2e4733a)
Error: ==> Verifying app config
--> Verified app config
==> Building image
==> Building image
Error: failed to fetch an image or build from source: failed to restart VM e7843274cd2028: resource_exhausted: rate limit exceeded (Request ID: 01J3M7VWV23W35JD81KAEVK2GQ-iad) (Trace ID: 2e7828453fdfde1aa213f71ad2e4733a)
at /home/circleci/project/repo/libs/plugins/fly/src/executors/deploy/executor.ts:44:20
exit code: 1
at ChildProcess.<anonymous> (file:///home/circleci/project/repo/node_modules/zx/build/core.js:146:26)
at ChildProcess.emit (node:events:520:28)
at maybeClose (node:internal/child_process:1105:16)
at Process.ChildProcess._handle.onexit (node:internal/child_process:305:5)
at Process.callbackTrampoline (node:internal/async_hooks:130:17)
_code,_signal,_stdout,_stderr,_combined
Apps just started failing to deploy, seems to be inconsistent what ones. Weirdly enough deploys keep creating new app builders.
On another one of our apps I just got Forbidden. We usually deploy dozens of times a day and never run into this issue unless there is a problem on the Fly side.
Our organizations slug is: trellis-social-enterprise-inc
WARN Failed to start remote builder heartbeat: failed to restart VM 328715e7a47478: failed_precondition: unable to restart machine, not currently started or stopped
==> Building image
WARN Failed to start remote builder heartbeat: failed to restart VM 328715e7a47478: failed_precondition: unable to restart machine, not currently started or stopped
Error: failed to fetch an image or build from source: failed to restart VM 328715e7a47478: failed_precondition: unable to restart machine, not currently started or stopped (Request ID: 01J3N9XN751Q1V43QPVNNMMAZ9-iad) (Trace ID: 30c8403c0ffb18aa391122db605b380f)
Error: ==> Verifying app config
--> Verified app config
==> Building image
==> Building image
Error: failed to fetch an image or build from source: failed to restart VM 328715e7a47478: failed_precondition: unable to restart machine, not currently started or stopped (Request ID: 01J3N9XN751Q1V43QPVNNMMAZ9-iad) (Trace ID: 30c8403c0ffb18aa391122db605b380f)
at /home/circleci/project/repo/libs/plugins/fly/src/executors/deploy/executor.ts:44:20
exit code: 1
at ChildProcess.<anonymous> (file:///home/circleci/project/repo/node_modules/zx/build/core.js:146:26)
at ChildProcess.emit (node:events:520:28)
at maybeClose (node:internal/child_process:1105:16)
at Process.ChildProcess._handle.onexit (node:internal/child_process:305:5)
at Process.callbackTrampoline (node:internal/async_hooks:130:17)
_code,_signal,_stdout,_stderr,_combined
Looks like the remote builder may have run into an RPC issue, rerunning.
Just retried the deploy this morning @kwaw and @billy
Seems like it is still happening for me
core: $ e[92mflye[39m deploy --confige[96m=e[39me[93m"apps/core/fly.develop.tomle[39me[93m"e[39m e[92m-e[39m-remote-only --build-arge[96m=e[39me[93m"TRELLIS_VERSION=equivalent-purplee[39me[93m"e[39m
core: ==> Verifying app config
core: Validating apps/core/fly.develop.toml
core: e[32m✓e[0m Configuration is valid
core: --> Verified app config
core: ==> Building image
core: e[33mWARNe[0m Failed to start remote builder heartbeat: failed to restart VM 6e825470a37e18: resource_exhausted: rate limit exceeded
core: ==> Building image
core: e[33mWARNe[0m Failed to start remote builder heartbeat: failed to restart VM 6e825470a37e18: failed_precondition: unable to restart machine, not currently started or stopped
core: Error: failed to fetch an image or build from source: failed to restart VM 6e825470a37e18: failed_precondition: unable to restart machine, not currently started or stopped (Request ID: 01J3QVXA5GNHCGRAMXV2HCCJHW-iad) (Trace ID: 8bc200d230fe7615c7ef7017683f97b0)
core: Error: ==> Verifying app config
core: --> Verified app config
core: ==> Building image
core: ==> Building image
core: Error: failed to fetch an image or build from source: failed to restart VM 6e825470a37e18: failed_precondition: unable to restart machine, not currently started or stopped (Request ID: 01J3QVXA5GNHCGRAMXV2HCCJHW-iad) (Trace ID: 8bc200d230fe7615c7ef7017683f97b0)
core: at /home/circleci/project/repo/libs/plugins/fly/src/executors/deploy/executor.ts:44:20
core: exit code: 1
core: at ChildProcess.<anonymous> (file:///home/circleci/project/repo/node_modules/zx/build/core.js:146:26)
core: at ChildProcess.emit (node:events:520:28)
core: at maybeClose (node:internal/child_process:1105:16)
core: at Process.ChildProcess._handle.onexit (node:internal/child_process:305:5)
core: at Process.callbackTrampoline (node:internal/async_hooks:130:17)
core: _code,_signal,_stdout,_stderr,_combined
core: Looks like the remote builder may have run into an RPC issue, rerunning.
in our CI the CLI is installed at the time of the run so it should be the newest version always.
OK this is weird, none of the machines referenced in this log (me deploying it locally), are the machine actually running in the fly dashboard
Watch your deployment at https://fly.io/apps/aurora-develop/monitoring
Updating existing machines in 'aurora-develop' with bluegreen strategy
Verifying if app can be safely deployed
Creating green machines
Created machine 48ed676b729728 [app]
Waiting for all green machines to start
Machine 48ed676b729728 [app] - created
Machine 48ed676b729728 [app] - started
Waiting for all green machines to be healthy
Machine 48ed676b729728 [app] - unchecked
Machine 48ed676b729728 [app] - 1/1 passing
Marking green machines as ready
Machine 48ed676b729728 [app] now ready
Checkpointing deployment, this may take a few seconds...
Waiting before cordoning all blue machines
Machine 080e474c6d9758 [app] cordoned
Waiting before stopping all blue machines
Stopping all blue machines
Waiting for all blue machines to stop
Machine 080e474c6d9758 [app] - started
Machine 080e474c6d9758 [app] - stopped
Destroying all blue machines
Machine 080e474c6d9758 [app] destroyed
Deployment Complete
Checking DNS configuration for aurora-develop.fly.dev
Visit your newly deployed app at https://aurora-develop.fly.dev/
Successfully ran target fly-deploy for aurora in develop configuration.
scotsman-develop
==> Building image
WARN Failed to start remote builder heartbeat: failed to restart VM e784937ce172d8: failed_precondition: unable to restart machine, not currently started or stopped
==> Building image
WARN Failed to start remote builder heartbeat: failed to restart VM e784937ce172d8: failed_precondition: unable to restart machine, not currently started or stopped
Error: failed to fetch an image or build from source: failed to restart VM e784937ce172d8: failed_precondition: unable to restart machine, not currently started or stopped (Request ID: 01J3R8HJX7MJPZE1DEF7M7Z8DJ-iad) (Trace ID: bc2e44003657571bc7cff3ec99363d8e)
Error: ==> Verifying app config
WARN Failed to start remote builder heartbeat: failed to launch VM: App not found
==> Building image
WARN Failed to start remote builder heartbeat: failed to restart VM e784937ce172d8: failed_precondition: unable to restart machine, not currently started or stopped
Looks like I can deploy them one by one locally using an authenticated fly CLI, but our CI keeps throwing the above errors. Did something change with deploy tokens? I checked this morning, all of our tokens are still valid.
Confirmed our CI is on the latest version of the Fly CLI.
One run: core-develop, aurora-develop, orient-develop fail
Another run: aurora-develop, scotsman-develop, core-develop, star-develop, and polar-develop fail
Every run seems to be slightly different. I am at a loss with no idea what to do from here on out.
