Deployment issue this afternoon

1

Hello Fly friends,

I’m trying to deploy nikola-sharder and it keeps failing for unclear reasons. Here was my output. FWIW I don’t believe I changed anything material on my end that should cause the build to fail. Any ideas? Thank you!

v87 failed - Failed due to unhealthy allocations - rolling back to job version 86
Failed Instances

==> Failure #1

Instance
  ID            = 88bc694a  
  Version       = 87        
  Region        = iad       
  Desired       = stop      
  Status        = failed    
  Health Checks =           
  Restarts      = 0         
  Created       = 2s ago    

Recent Events
TIMESTAMP            TYPE            MESSAGE                                                                                                                                
2020-12-01T22:44:24Z Received        Task received by client                                                                                                                
2020-12-01T22:44:24Z Task Setup      Building Task Directory                                                                                                                
2020-12-01T22:44:25Z Driver Failure  rpc error: code = Unknown desc = unable to create microvm: error pulling image: unknown                                                
                                     github.com/containerd/containerd/errdefs.init                                                                                          
                                     	/go/pkg/mod/github.com/containerd/containerd@v1.4.1/errdefs/errors.go:43                                                               
                                     runtime.doInit                                                                                                                         
                                     	/usr/local/go/src/runtime/proc.go:5625                                                                                                 
                                     runtime.doInit                                                                                                                         
                                     	/usr/local/go/src/runtime/proc.go:5620                                                                                                 
                                     runtime.doInit                                                                                                                         
                                     	/usr/local/go/src/runtime/proc.go:5620                                                                                                 
                                     runtime.doInit                                                                                                                         
                                     	/usr/local/go/src/runtime/proc.go:5620                                                                                                 
                                     runtime.doInit                                                                                                                         
                                     	/usr/local/go/src/runtime/proc.go:5620                                                                                                 
                                     runtime.main                                                                                                                           
                                     	/usr/local/go/src/runtime/proc.go:191                                                                                                  
                                     runtime.goexit                                                                                                                         
                                     	/usr/local/go/src/runtime/asm_amd64.s:1374                                                                                             
                                     failed to create new thin device "data_0-nomadfc_layers-snap-1139" (dev: 4): file exists                                               
                                     github.com/containerd/containerd/errdefs.FromGRPC                                                                                      
                                     	/go/pkg/mod/github.com/containerd/containerd@v1.4.1/errdefs/grpc.go:107                                                                
                                     github.com/containerd/containerd/snapshots/proxy.(*proxySnapshotter).Prepare                                                           
                                     	/go/pkg/mod/github.com/containerd/containerd@v1.4.1/snapshots/proxy/proxy.go:108                                                       
                                     github.com/containerd/containerd/rootfs.applyLayers                                                                                    
                                     	/go/pkg/mod/github.com/containerd/containerd@v1.4.1/rootfs/apply.go:129                                                                
                                     github.com/containerd/containerd/rootfs.ApplyLayerWithOpts                                                                             
                                     	/go/pkg/mod/github.com/containerd/containerd@v1.4.1/rootfs/apply.go:102                                                                
                                     github.com/containerd/containerd.(*image).Unpack                                                                                       
                                     	/go/pkg/mod/github.com/containerd/containerd@v1.4.1/image.go:324                                                                       
                                     github.com/superfly/nomad-firecracker/driver.pullImage                                                                                 
                                     	/app/driver/machine.go:125                                                                                                             
                                     github.com/superfly/nomad-firecracker/driver.NewMachine                                                                                
                                     	/app/driver/machine.go:167                                                                                                             
                                     github.com/superfly/nomad-firecracker/driver.(*Driver).StartTask                                                                       
                                     	/app/driver/driver.go:781                                                                                                              
                                     github.com/hashicorp/nomad/plugins/drivers.(*driverPluginServer).StartTask                                                             
                                     	/go/pkg/mod/github.com/hashicorp/nomad@v0.10.2/plugins/drivers/server.go:105                                                           
                                     github.com/hashicorp/nomad/plugins/drivers/proto._Driver_StartTask_Handler                                                             
                                     	/go/pkg/mod/github.com/hashicorp/nomad@v0.10.2/plugins/drivers/proto/driver.pb.go:3797                                                 
                                     google.golang.org/grpc.(*Server).processUnaryRPC                                                                                       
                                     	/go/pkg/mod/google.golang.org/grpc@v1.33.2/server.go:1210                                                                              
                                     google.golang.org/grpc.(*Server).handleStream                                                                                          
                                     	/go/pkg/mod/google.golang.org/grpc@v1.33.2/server.go:1533                                                                              
                                     google.golang.org/grpc.(*Server).serveStreams.func1.2                                                                                  
                                     	/go/pkg/mod/google.golang.org/grpc@v1.33.2/server.go:871                                                                               
                                     runtime.goexit                                                                                                                         
                                     	/usr/local/go/src/runtime/asm_amd64.s:1374                                                                                             
                                     failed to prepare extraction snapshot "extract-480247730-KKLp sha256:e40d297cf5f89a9822af4c2f63caa2f2085d5aa188137506918e603774b083cb" 
                                     github.com/containerd/containerd/rootfs.applyLayers                                                                                    
                                     	/go/pkg/mod/github.com/containerd/containerd@v1.4.1/rootfs/apply.go:146                                                                
                                     github.com/containerd/containerd/rootfs.ApplyLayerWithOpts                                                                             
                                     	/go/pkg/mod/github.com/containerd/containerd@v1.4.1/rootfs/apply.go:102                                                                
                                     github.com/containerd/containerd.(*image).Unpack                                                                                       
                                     	/go/pkg/mod/github.com/containerd/containerd@v1.4.1/image.go:324                                                                       
                                     github.com/superfly/nomad-firecracker/driver.pullImage                                                                                 
                                     	/app/driver/machine.go:125                                                                                                             
                                     github.com/superfly/nomad-firecracker/driver.NewMachine                                                                                
                                     	/app/driver/machine.go:167                                                                                                             
                                     github.com/superfly/nomad-firecracker/driver.(*Driver).StartTask                                                                       
                                     	/app/driver/driver.go:781                                                                                                              
                                     github.com/hashicorp/nomad/plugins/drivers.(*driverPluginServer).StartTask                                                             
                                     	/go/pkg/mod/github.com/hashicorp/nomad@v0.10.2/plugins/drivers/server.go:105                                                           
                                     github.com/hashicorp/nomad/plugins/drivers/proto._Driver_StartTask_Handler                                                             
                                     	/go/pkg/mod/github.com/hashicorp/nomad@v0.10.2/plugins/drivers/proto/driver.pb.go:3797                                                 
                                     google.golang.org/grpc.(*Server).processUnaryRPC                                                                                       
                                     	/go/pkg/mod/google.golang.org/grpc@v1.33.2/server.go:1210                                                                              
                                     google.golang.org/grpc.(*Server).handleStream                                                                                          
                                     	/go/pkg/mod/google.golang.org/grpc@v1.33.2/server.go:1533                                                                              
                                     google.golang.org/grpc.(*Server).serveStreams.func1.2                                                                                  
                                     	/go/pkg/mod/google.golang.org/grpc@v1.33.2/server.go:871                                                                               
                                     runtime.goexit                                                                                                                         
                                     	/usr/local/go/src/runtime/asm_amd64.s:1374                                                                                             
2020-12-01T22:44:25Z Not Restarting  Error was unrecoverable                                                                                                                
2020-12-01T22:44:25Z Alloc Unhealthy Unhealthy because of failed task                                                                                                       
2020-12-01T22:44:25Z Killing         Sent interrupt. Waiting 5s before force killing                                                                                        
2020-12-01T22:44:25Z Killing         Sent interrupt. Waiting 5s before force killing
2

Very sorry about that.

We’re working on a big change with deployments (to make them faster). Looks like it’s not quite ready yet. I’ve rolled back our change for now. Feel free to retry!

1 Like
3

No problem! Thanks for the quick response. I’ll try again now.

4

Sadly it failed again, but a little less verbosely?

Release v93 created

Deploying to : nikola-sharder.fly.dev

Monitoring Deployment

You can detach the terminal anytime without stopping the deployment

2 desired, 1 placed, 0 healthy, 1 unhealthy

v93 failed - Failed due to unhealthy allocations - rolling back to job version 92

2 desired, 2 placed, 2 healthy, 0 unhealthy

--> v94 deployed successfully

***v93 failed - Failed due to unhealthy allocations - rolling back to job version 92 and deploying as v94```
5

This seems unrelated to our deployment process. It appears your allocations came up unhealthy. If this is still occurring can you retry while running flyctl logs? You should see it come up and fail.

6

Thanks Jerome! I’m not sure what it was, but I tried again a few minutes ago and it’s working. :man_shrugging:

So we’re good!