error connecting to docker: could not fulfill resource requirements: could not retrieve volume with id

Hello, Fly newbie here, for testing and learning the service I’m trying to launch a basic Phoenix app (no Postgres) following the Elixir instructions, but I got the could not retrieve volume with id docker error.

My steps:

Setting up the Phoenix app

$ mix phx.new hello_elixir --no-ecto --install
$ cd hello_elixir

Trying to deploy the app

$ fly auth login
Opening https://fly.io/app/auth/cli/eafb9626b4c9497c1ae418076fdd7ff0 ...
Waiting for session... Done
successfully logged in as ...
$
$ fly launch
[...]
Services
TCP 80/443 ⇢ 8080
Error error connecting to docker: could not fulfill resource requirements: could not retrieve volume with id '44346':

Lets view the status

$ fly status
App
  Name     = my-phx-test
  Owner    = personal
  Version  = 0
  Status   = pending
  Hostname = my-phx-test.fly.dev

App has not been deployed yet.

Lets get some debug info

$ LOG_LEVEL=debug fly deploy
DEBUG Loaded flyctl config from/Users/Development/.fly/config.yml
DEBUG determined hostname: "MacBookAir.local"
DEBUG determined working directory: "/Users/Development/hello_elixir"
DEBUG determined user home directory: "/Users/Development"
DEBUG determined config directory: "/Users/Development/.fly"
DEBUG ensured config directory exists.
DEBUG ensured config directory perms.
DEBUG cache loaded.
DEBUG config initialized.
DEBUG initialized task manager.
DEBUG skipped querying for new release
DEBUG client initialized.
DEBUG app config loaded from /Users/Development/hello_elixir/fly.toml
==> Verifying app config
--> Verified app config
==> Building image
DEBUG trying local docker daemon
DEBUG Local docker daemon unavailable
DEBUG trying remote docker daemon
DEBUG Reporting buildDEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: StartSourceBuildInput!) { startSourceBuild(input: $input) { sourceBuild { id } } }","variables":{"input":{"appId":"my-phx-test"}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (507.36ms) {"errors":[{"message":"StartSourceBuildInput isn't a defined input type (on $input)","locations":[{"line":1,"column":10}],"path":["mutation"],"extensions":{"code":"variableRequiresValidType","typeName":"StartSourceBuildInput","variableName":"input"}},{"message":"Field 'startSourceBuild' doesn't exist on type 'Mutations'","locations":[{"line":1,"column":44}],"path":["mutation","startSourceBuild"],"extensions":{"code":"undefinedField","typeName":"Mutations","fieldName":"startSourceBuild"}},{"message":"Variable $input is declared by anonymous mutation but not used","locations":[{"line":1,"column":1}],"path":["mutation"],"extensions":{"code":"variableNotUsed","variableName":"input"}}]}
DEBUG Failed storing buildDEBUG Trying 'Buildpacks' strategy
DEBUG no buildpack builder configured, skipping
DEBUG result image:<nil> error:<nil>
DEBUG Trying 'Dockerfile' strategy
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: EnsureMachineRemoteBuilderInput!) { ensureMachineRemoteBuilder(input: $input) { machine { id state ips { nodes { family kind ip } } }, app { name organization { id slug } } } }","variables":{"input":{"appName":"my-phx-test","organizationId":null}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (3.59s) {"data":{"ensureMachineRemoteBuilder":null},"errors":[{"message":"could not fulfill resource requirements: could not retrieve volume with id '44346': ","locations":[{"line":1,"column":54}],"path":["ensureMachineRemoteBuilder"],"extensions":{"code":"NOT_FOUND"}}]}
DEBUG result image:<nil> error:error connecting to docker: could not fulfill resource requirements: could not retrieve volume with id '44346':
Error failed to fetch an image or build from source: error connecting to docker: could not fulfill resource requirements: could not retrieve volume with id '44346':

Some more tests

$ fly doctor
Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... Nope
Pinging WireGuard gateway (give us a sec)... PASSED
$

My environment

$ uname -a
Darwin MacBookAir.local 21.4.0 Darwin Kernel Version 21.4.0: Mon Feb 21 20:34:37 PST 2022; root:xnu-8020.101.4~2/RELEASE_X86_64 x86_64
$
$ fly version
fly v0.0.306 darwin/amd64 Commit: b5101f2 BuildDate: 2022-03-15T18:17:35Z
$ 
$ mix phx.new --version
Phoenix installer v1.6.6
$
$ elixir --version
Erlang/OTP 24 [erts-12.3] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [jit] [dtrace]
Elixir 1.13.3 (compiled with Erlang/OTP 24)
$
$ docker --version
Docker version 20.10.13, build a224086349
$
$ git --version
git version 2.35.1

How long did that log command take before it gave you the error? That’s a weird internal Docker error that I haven’t seen before.

If this happened more than once, you can try deleting the remote builder to start fresh. fly apps destroy fly-builder-???. You should see the builder name in your logs.

Hello Kurt,

How long did that log command take before it gave you the error?

it takes about a second

you can try deleting the remote builder to start fresh. fly apps destroy fly-builder-???

I had already tried to delete both the app and the builder, but while the app is deletable, trying to delete the builder raise the error Error NatsProxyClient::NotFoundError:

$ fly apps list
NAME                     	OWNER   	STATUS 	LATEST DEPLOY
fly-builder-dry-dust-5657	personal	pending
my-phx-test
$
$ fly apps destroy fly-builder-dry-dust-5657
Destroying an app is not reversible.
? Destroy app fly-builder-dry-dust-5657? Yes
Error NatsProxyClient::NotFoundError

I’ve retried $ fly deploy this morning a couple of time (without any changes on my side): first time it rise the same error, but second time one hour later, it worked. So probably in the meantime something has been reset on your side, now it seems that the builder is working regularly.

Hi guys, I am also getting this issue and I have no idea how to fix it!

Hello @adam-collier, from your tweets I’m not sure your issue is related to the problem the OP had.
From a quick search, I wonder whether your remote builder app is out of disk space as in this thread.

One thing you can try is to run fly apps destroy your-builder-app and try another deploy.

Hey, so I have destroyed the builder app but unfortunately that didn’t fix anything when it came to deploying. The same thing happened with the newly created builder. Like the OP I get this when running fly doctor:

Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... Nope
Pinging WireGuard gateway (give us a sec)... PASSED

and the error after trying to deploy:

WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-empty-wood-3870`
Error failed to fetch an image or build from source: error connecting to docker: remote builder app unavailable

I’ve tweeted the Remix guys to see whether the Indie Stack can be deployed via the CLI (I’m assuming it should be able to) rather than going through GitHub actions, but I’m pretty stumped by whats happening.

@adam-collier I just looked, this builder seemed to be coming up with a bad disk. Can you give it one more try?

That looks to be working for me now! Is there anything I can do my end to remedy this if it happens again in the future?

No this was a bug that hit your builder in particular. Our builders are amazing, usually, but they’re pretty complicated and have weird corner cases. You can workaround builder issues by running Docker locally (which is actually better in GitHub actions, worse on an M1 macbook).