grant
April 13, 2023, 8:21pm
1
Hey!
I have a fly app that has been working for me but suddenly this morning I started getting “Pull failed, retrying” when I try to deploy.
I am deploying via fly deploy and it is a v1 app running on nomad.
Not sure if there is any other context I can share here that would be helpful besides the logs:
2023-04-13T20:12:42.659 runner[3588d999] sjc [info] Starting instance
2023-04-13T20:12:42.702 runner[3588d999] sjc [info] Configuring virtual machine
2023-04-13T20:12:42.706 runner[3588d999] sjc [info] Pulling container image
2023-04-13T20:12:43.294 runner[3588d999] sjc [info] Unpacking image
2023-04-13T20:12:43.298 runner[3588d999] sjc [info] Pull failed, retrying (attempt #0)
2023-04-13T20:12:43.702 runner[3588d999] sjc [info] Unpacking image
2023-04-13T20:12:43.708 runner[3588d999] sjc [info] Pull failed, retrying (attempt #1)
2023-04-13T20:12:44.124 runner[3588d999] sjc [info] Unpacking image
2023-04-13T20:12:44.129 runner[3588d999] sjc [info] Pull failed, retrying (attempt #2)
2023-04-13T20:12:44.129 runner[3588d999] sjc [info] Pulling image failed
grant
April 13, 2023, 8:28pm
2
Some more context, was able to get a stack trace:
TIMESTAMP TYPE MESSAGE
2023-04-13T20:26:50Z Received Task received by client
2023-04-13T20:26:50Z Task Setup Building Task Directory
2023-04-13T20:27:04Z Driver Failure rpc error: code = Unknown desc = unable to create microvm: error pulling image: invalid argument
github.com/containerd/containerd/errdefs.init
/go/pkg/mod/github.com/containerd/containerd@v1.4.12/errdefs/errors.go:44
runtime.doInit
/usr/local/go/src/runtime/proc.go:6506
runtime.doInit
/usr/local/go/src/runtime/proc.go:6483
runtime.doInit
/usr/local/go/src/runtime/proc.go:6483
runtime.doInit
/usr/local/go/src/runtime/proc.go:6483
runtime.doInit
/usr/local/go/src/runtime/proc.go:6483
runtime.main
/usr/local/go/src/runtime/proc.go:233
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
snapshotter not loaded: devmapper
github.com/containerd/containerd/errdefs.FromGRPC
/go/pkg/mod/github.com/containerd/containerd@v1.4.12/errdefs/grpc.go:107
github.com/containerd/containerd/snapshots/proxy.(*proxySnapshotter).Stat
/go/pkg/mod/github.com/containerd/containerd@v1.4.12/snapshots/proxy/proxy.go:52
github.com/containerd/containerd/rootfs.ApplyLayerWithOpts
/go/pkg/mod/github.com/containerd/containerd@v1.4.12/rootfs/apply.go:97
github.com/containerd/containerd.(*image).Unpack
/go/pkg/mod/github.com/containerd/containerd@v1.4.12/image.go:324
github.com/superfly/nomad-firecracker/driver.pullImage
/app/driver/machine.go:160
github.com/superfly/nomad-firecracker/driver.NewMachine.func3
/app/driver/machine.go:543
github.com/superfly/nomad-firecracker/driver.NewMachine
/app/driver/machine.go:1135
github.com/superfly/nomad-firecracker/driver.(*Driver).startTask
/app/driver/driver.go:1180
github.com/superfly/nomad-firecracker/driver.(*Driver).StartTask
/app/driver/driver.go:1002
github.com/hashicorp/nomad/plugins/drivers.(*driverPluginServer).StartTask
/go/pkg/mod/github.com/hashicorp/nomad@v0.12.0/plugins/drivers/server.go:105
github.com/hashicorp/nomad/plugins/drivers/proto._Driver_StartTask_Handler
/go/pkg/mod/github.com/hashicorp/nomad@v0.12.0/plugins/drivers/proto/driver.pb.go:4260
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:1336
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:965
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
failed to stat snapshot sha256:ed7b0ef3bf5bbec74379c3ae3d5339e666a314223e863c70644f7522a7527461
github.com/containerd/containerd/rootfs.ApplyLayerWithOpts
/go/pkg/mod/github.com/containerd/containerd@v1.4.12/rootfs/apply.go:99
github.com/containerd/containerd.(*image).Unpack
/go/pkg/mod/github.com/containerd/containerd@v1.4.12/image.go:324
github.com/superfly/nomad-firecracker/driver.pullImage
/app/driver/machine.go:160
github.com/superfly/nomad-firecracker/driver.NewMachine.func3
/app/driver/machine.go:543
github.com/superfly/nomad-firecracker/driver.NewMachine
/app/driver/machine.go:1135
github.com/superfly/nomad-firecracker/driver.(*Driver).startTask
/app/driver/driver.go:1180
github.com/superfly/nomad-firecracker/driver.(*Driver).StartTask
/app/driver/driver.go:1002
github.com/hashicorp/nomad/plugins/drivers.(*driverPluginServer).StartTask
/go/pkg/mod/github.com/hashicorp/nomad@v0.12.0/plugins/drivers/server.go:105
github.com/hashicorp/nomad/plugins/drivers/proto._Driver_StartTask_Handler
/go/pkg/mod/github.com/hashicorp/nomad@v0.12.0/plugins/drivers/proto/driver.pb.go:4260
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:1336
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:965
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
2023-04-13T20:27:04Z Not Restarting Error was unrecoverable
2023-04-13T20:27:04Z Alloc Unhealthy Unhealthy because of failed task
kwaw
April 13, 2023, 11:33pm
3
Hey Grant! first off, you should move off nomad if you can. (it’s easy )
We’re investigating an issue with a host in the SJC region. You should see a related issue on your fly dashboard. You can deploy in a closer region for now till the issue is resolved.
grant
April 13, 2023, 11:34pm
4
Hey @kwaw , thanks for the help, got on chat with some other people at Fly and it seems like the issue was resolve. A blocker for moving to v2 for us is that we use gRPC via tls termination which isn’t supported yet.
1 Like
This should be fixed in the latest flyctl v0.0.524 . Can you give it a shot?
system
April 25, 2023, 9:33am
6
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.