We’ve had luck with autoscaling our Rails app, but not today.
We noticed it stuck at our minimum (2) and bumped the min up to 6. We have since changed the min to 4, and it is still scaled to 6.
Earlier today, we noticed external connection timeouts and discovered we were at our connection hard_limit. We assumed this would trigger a scale up, but it didn’t.
Looking at history, this happened yesterday too, but we hadn’t realized.
Hey there.
If the auto scale minimum numbers are set as you expected, maybe you could have a look at the regions?
Running the command below should show you what regions you have selected for deployment
fly regions list
If you have only one region, then the scale may not do anything if it can’t be deployed at the same region. Then it should look for backup regions. However, backup regions are disabled by default.
We got this feedback from support, which I’m happy to share:
It looks like this unexpected behaviour was caused by a known bug in the Nomad autoscaler service we use to autoscale Nomad-based apps. This bug causes the service to freeze up requiring a manual restart, and has happened to our platform only a few times over the last year. We’re assessing the alerting we have set up on the autoscaler service so that we can catch these faster in future, preventing this situation.
We’ve restarted the service so autoscaling should be working properly again. Sorry for the inconvenience
Right. Thanks for sharing!
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.