Possible issues in `ord` region

1

We are currently seeing delays/unresponsiveness with managing applications in the ord region. This includes deploys, restarts, and scaling.

Some requests seem to eventually go through. For example, we were able to update to a new region (den) which seemed to push through some of the pending stop statuses.

This appears to be a region-specific issue as opposed to a wider issue as interactions within our new region seem to be more responsive. When attempting to set the region back to ord, we are running into a failed deployment and the instance getting stuck in a “stop” state.

Instances
ID      	PROCESS	VERSION	REGION	DESIRED	STATUS 	HEALTH CHECKS	RESTARTS	CREATED    
4da4496a	app    	103 ⇡  	den   	run    	pending	             	0       	5m26s ago 	
2dd43ec8	app    	101    	ord   	stop   	pending	             	0       	10m45s ago

I have explicitly set the region back to den and, after a few minutes in limbo with ord, the ord instance eventually went away again and we are able to connect to our application.

Within den interactions such as restarts are all working as expected.

We are now safely deployed, but I wanted to post to see if this is a region issue or something else (and to have an idea of when we can go back to ord).

Update 1 The database instance for this application is also in ord and it seems to be more responsive to scale commands, not sure if this rules out an ord region-wide issue.

1 Like
2

We’re running into the same issue with deployments in ord.

3

We’re looking into this. A single server in the ord region is currently a bit overloaded with deployment operations, so only some deployments in the region may be affected and not others.

3 Likes
4

This issue should be cleared up by now, sorry for the inconvenience!

1 Like
5

Thank you for the insights and updates, @wjordan!

We have switched our app back to ord and everything appears to be responsive again :+1:

For future reference–was this the appropriate method for asking the question, or is there a better practice you all would prefer?