failure to restart after scaling vms due to inability to connect to db

1

I decided to scale up our apps’ VMs today.

Among others, I increased the scale of dev-www-rhg-db-20220314 and dev-www-rhg. The latter did not come back up. It crash-looped with a failure to connect to the DB:

failed to connect to host=dev-www-rhg-db-20220314.internal user=postgres database=: failed to receive message (unexpected EOF)

…which had also just gotten resized.

flyctl restart dev-www-rhg eventually fixed the problem. We’re OK now, and it didn’t hurt too much b/c it was just dev (not prod), but I wanted to report the bug. Please let me know if there’s more info I can/should provide.

2

Also possibly of note, the first thing I did to try to fix was flyctl deploy; that got stuck and after about a half an hour I gave up and did the sensible thing (flyctl restart). Deployments now work again.

3

And if you’re digging into this and notice that the app is now crashing a bunch, that’s just me trying to figure out how to get checks and monitoring working. :slight_smile: