Deployment consistently failing

My deployments are constantly failing, it seems pulling the image fails

2021-02-08T11:06:23.919Z 203a8786 fra [info] Starting instance

2021-02-08T11:06:23.946Z 203a8786 fra [info] Configuring virtual machine
2021-02-08T11:06:23.948Z 203a8786 fra [info] Pulling container image
2021-02-08T11:06:29.611Z 203a8786 fra [info] Unpacking image
2021-02-08T11:07:41.163Z 203a8786 fra [info] Pull failed, retrying (attempt #0)
2021-02-08T11:07:41.902Z 203a8786 fra [info] Unpacking image
2021-02-08T11:07:42.029Z 203a8786 fra [info] Pull failed, retrying (attempt #1)
2021-02-08T11:07:42.679Z 203a8786 fra [info] Unpacking image
2021-02-08T11:07:42.802Z 203a8786 fra [info] Pulling image failed
2021-02-08T11:07:42.802Z 203a8786 fra [info] Pull failed, retrying (attempt #2)
2021-02-08T11:12:08.669Z c2db211e fra [info] Starting instance
2021-02-08T11:12:08.705Z c2db211e fra [info] Configuring virtual machine
2021-02-08T11:12:08.709Z c2db211e fra [info] Pulling container image
2021-02-08T11:12:14.593Z c2db211e fra [info] Unpacking image
2021-02-08T11:13:27.366Z c2db211e fra [info] Pull failed, retrying (attempt #0)
2021-02-08T11:13:28.155Z c2db211e fra [info] Unpacking image
2021-02-08T11:13:28.279Z c2db211e fra [info] Pull failed, retrying (attempt #1)
2021-02-08T11:13:28.925Z c2db211e fra [info] Unpacking image
2021-02-08T11:13:29.025Z c2db211e fra [info] Pulling image failed
2021-02-08T11:13:29.025Z c2db211e fra [info] Pull failed, retrying (attempt #2)

More info

Instance

ID = f6aa3f71
Version = 48
Region = fra
Desired = stop
Status = failed
Health Checks =
Restarts = 0
Created = 10s ago

Recent Events
TIMESTAMP TYPE MESSAGE
2021-02-08T11:43:41Z Received Task received by client
2021-02-08T11:43:41Z Task Setup Building Task Directory
2021-02-08T11:43:49Z Driver Failure rpc error: code = Unknown desc = unable to create microvm: error pulling image: unknown
github. com/containerd/containerd/errdefs.init
/go/pkg/mod/github. com/containerd/containerd@v1.4.1/errdefs/errors.go:43
runtime.doInit
/usr/local/go/src/runtime/proc.go:5652
runtime.doInit
/usr/local/go/src/runtime/proc.go:5647
runtime.doInit
/usr/local/go/src/runtime/proc.go:5647
runtime.doInit
/usr/local/go/src/runtime/proc.go:5647
runtime.doInit
/usr/local/go/src/runtime/proc.go:5647
runtime.main
/usr/local/go/src/runtime/proc.go:191
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1374
failed to mount /var/lib/containerd/tmpmounts/containerd-mount779145311: input/output error
github. com/containerd/containerd/errdefs.FromGRPC
/go/pkg/mod/github. com/containerd/containerd@v1.4.1/errdefs/grpc.go:107
github. com/containerd/containerd.(*diffRemote).Apply
/go/pkg/mod/github. com/containerd/containerd@v1.4.1/diff.go:62
github. com/containerd/containerd/rootfs.applyLayers
/go/pkg/mod/github. com/containerd/containerd@v1.4.1/rootfs/apply.go:163
github. com/containerd/containerd/rootfs.ApplyLayerWithOpts
/go/pkg/mod/github. com/containerd/containerd@v1.4.1/rootfs/apply.go:102
github. com/containerd/containerd.(*image).Unpack
/go/pkg/mod/github. com/containerd/containerd@v1.4.1/image.go:324
github. com/superfly/nomad-firecracker/driver.pullImage
/app/driver/machine.go:129
github. com/superfly/nomad-firecracker/driver.NewMachine.func1
/app/driver/machine.go:181
github. com/superfly/nomad-firecracker/driver.NewMachine
/app/driver/machine.go:855
github. com/superfly/nomad-firecracker/driver.(*Driver).StartTask
/app/driver/driver.go:863
github. com/hashicorp/nomad/plugins/drivers.(*driverPluginServer).StartTask
/go/pkg/mod/github. com/hashicorp/nomad@v0.10.2/plugins/drivers/server.go:105
github. com/hashicorp/nomad/plugins/drivers/proto._Driver_StartTask_Handler
/go/pkg/mod/github. com/hashicorp/nomad@v0.10.2/plugins/drivers/proto/driver.pb.go:3797
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.33.2/server.go:1210
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.33.2/server.go:1533
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.33.2/server.go:871
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1374
failed to extract layer sha256:3d1b0ce0c789c89277235961debbd0ee08f69eed09e55d503f0865e490ba354e
github. com/containerd/containerd/rootfs.applyLayers
/go/pkg/mod/github. com/containerd/containerd@v1.4.1/rootfs/apply.go:165
github. com/containerd/containerd/rootfs.ApplyLayerWithOpts
/go/pkg/mod/github. com/containerd/containerd@v1.4.1/rootfs/apply.go:102
github. com/containerd/containerd.(*image).Unpack
/go/pkg/mod/github. com/containerd/containerd@v1.4.1/image.go:324
github. com/superfly/nomad-firecracker/driver.pullImage
/app/driver/machine.go:129
github. com/superfly/nomad-firecracker/driver.NewMachine.func1
/app/driver/machine.go:181
github. com/superfly/nomad-firecracker/driver.NewMachine
/app/driver/machine.go:855
github. com/superfly/nomad-firecracker/driver.(*Driver).StartTask
/app/driver/driver.go:863
github. com/hashicorp/nomad/plugins/drivers.(*driverPluginServer).StartTask
/go/pkg/mod/github. com/hashicorp/nomad@v0.10.2/plugins/drivers/server.go:105
github. com/hashicorp/nomad/plugins/drivers/proto._Driver_StartTask_Handler
/go/pkg/mod/github. com/hashicorp/nomad@v0.10.2/plugins/drivers/proto/driver.pb.go:3797
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.33.2/server.go:1210
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.33.2/server.go:1533
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.33.2/server.go:871
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1374
2021-02-08T11:43:49Z Not Restarting Error was unrecoverable
2021-02-08T11:43:49Z Alloc Unhealthy Unhealthy because of failed task
2021-02-08T11:43:50Z Killing Sent interrupt. Waiting 5s before force killing

I’m investigating this right now. Could you try deploying to AMS instead?

I had the same problem using fra.
Deploying to ams worked fine.

This should now be fixed in FRA

Thanks, I’m redeploying now. There’ve been several issues on FRA recently, should I change region in Europe? ThereIs there something wrong with FRA specifically?

Our alerts for a full disk were not being triggered correctly. We fixed the disk space issue and fixed the alert so we’ll be notified the next time.

Deploying to FRA should be safe. If this happens again it’s likely a different bug and we need to fix it.

I think the issue you had before was the same as this one. We re-enabled the host thinking it had been fixed but it had not.

Perfect, thanks. Meanwhile the new deployment went smooth!