Unexpected VM downtime

1

My iad Redis app has had about an hour of downtime this afternoon due to the VM getting shut down and staying offline for a long while. I’m trying to understand what’s happening.

The iad VM for bootspoon-redis was offline from 18:16 - 18:29 UTC, came up for ten minutes, then went back down from 18:38 - 19:26 UTC.

The app’s activity panel shows no activity in almost two weeks.

App logs show routine Redis stuff until it’s abruptly shut down:

2022-04-11T18:30:30Z app[2ce3a518] iad [info]599:M 11 Apr 2022 18:30:30.085 * Background saving started by pid 604                                             
2022-04-11T18:30:30Z app[2ce3a518] iad [info]604:C 11 Apr 2022 18:30:30.105 * DB saved on disk                                                                 
2022-04-11T18:30:30Z app[2ce3a518] iad [info]604:C 11 Apr 2022 18:30:30.106 * RDB: 0 MB of memory used by copy-on-write                                        
2022-04-11T18:30:30Z app[2ce3a518] iad [info]599:M 11 Apr 2022 18:30:30.185 * Background saving terminated with success                                        
2022-04-11T18:38:29Z runner[2ce3a518] iad [info]Shutting down virtual machine                                                                                  
2022-04-11T18:38:29Z app[2ce3a518] iad [info]Sending signal SIGINT to main child process w/ PID 524                                                            
2022-04-11T19:26:50Z runner[f41c3e93] iad [info]Starting instance                                                                                              
2022-04-11T19:26:51Z runner[f41c3e93] iad [info]Configuring virtual machine                                                                                    
2022-04-11T19:26:51Z runner[f41c3e93] iad [info]Pulling container image                                                                                        
2022-04-11T19:26:51Z runner[f41c3e93] iad [info]Unpacking image                                                                                                
2022-04-11T19:26:51Z runner[f41c3e93] iad [info]Preparing kernel init                                                                                          
2022-04-11T19:26:51Z runner[f41c3e93] iad [info]Setting up volume 'bootspoon_redis'                                                                            
2022-04-11T19:26:51Z runner[f41c3e93] iad [info]Opening encrypted volume                                                                                       
2022-04-11T19:26:52Z runner[f41c3e93] iad [info]Configuring firecracker                                                                                        
2022-04-11T19:26:52Z runner[f41c3e93] iad [info]Starting virtual machine
2

We investigated this, and it was a problem on our end. The host hardware your VM was running on was at max capacity. We disabled it for new VMs.

Shortly after we did that, your VM was stopped by an automated process designed to move apps around to free up capacity. This should not have happened, VMs with disks should never be stopped like this (only stateless VMs should be).

It came back up when we re-enabled the original hardware it was running on.

We’re working on this problem, I don’t think you’ll see it happen again.

3

Excellent, thanks!

1 Like