No instances can be created in hongkong

Questions / Help
1

As title said, the app I normally use suddenly shuts down during normal operation and tries to deploy again. It always shows that pending is not working normally. The area is Hong Kong
In addition, a similar situation has occurred in Japan and Singapore

2

Me too. Status shows:

Deployment Status
  ID          = 694d29d5-ac3d-e561-4949-b5e8dea03f22               
  Version     = v66                                                
  Status      = running                                            
  Description = Deployment is running pending automatic promotion  
  Instances   = 1 desired, 0 placed, 0 healthy, 0 unhealthy

HKG region.

Update: It’s done.

3

We had a bunch of fraudulent signups get through over night and use up all our capacity in hkg/nrt/sin. Sorry about that, we’ve tweaked some of our abuse prevention to catch similar attempts.

2 Likes
4

Looks like it happen again, I update a secret then deployment is block over 10 minutes and my website is down…

# web
Deployment Status
  ID          = 5ed41b44-0e7c-7035-85e8-900912029f54               
  Version     = v87                                                
  Status      = running                                            
  Description = Deployment is running pending automatic promotion  
  Instances   = 1 desired, 1 placed, 0 healthy, 0 unhealthy    

# worker
Deployment Status
  ID          = 3dac1f9d-7470-866c-db1e-2aa01276b34b               
  Version     = v49                                                
  Status      = running                                            
  Description = Deployment is running pending automatic promotion  
  Instances   = 1 desired, 1 placed, 0 healthy, 0 unhealthy

update: also rollback fail

Deployment Status
  ID          = 5ed41b44-0e7c-7035-85e8-900912029f54                                                                                   
  Version     = v87                                                                                                                    
  Status      = failed                                                                                                                 
  Description = Failed due to unhealthy allocations - not rolling back to stable job version 87 as current job has same specification  
  Instances   = 1 desired, 1 placed, 0 healthy, 1 unhealthy
5

mention @kurt

6

We’re looking at this now.

1 Like
7

This should be fixed now. A network issue on one of the hosts in HKG was preventing consul from syncing state, which then blocked VMs on other HKG hosts.

8

Yes it’s fixed now. :+1:

9

Oops, previous deployment was successful, but I started a new deployment and it run for a long time.

Deployment Status
  ID          = 6502ae90-d925-bc55-b6ad-95d7109ea83c         
  Version     = v52                                          
  Status      = running                                      
  Description = Deployment is running                        
  Instances   = 1 desired, 0 placed, 0 healthy, 0 unhealthy

update: It’s done.