URGENT: Primary replica won't spin up!

Our systems are all down. We’ve tried everything but no success. What’s happening there?

The primary simply won’t work.

1 Like

Hi @emilianomon and @morelli_enabler,

I’m tagging you both here because it looks like you are in the same company, is that correct? My name is John; I’m on the Infra team here at Fly.io. There were a few Fly Postgres clusters that had some type of bug triggered by a routine maintenance action we run against our host server fleet. We’re still trying to diagnose what this bug/interaction was; it’s quite obscure. There will be more forthcoming on that; right now, I’m focusing on helping out anyone who was impacted.

While the timing of your problem suggests that you were impacted, I’ve taken a look at some of our backend info and the structure of your Postgres cluster does not match what we’ve seen with the others who were impacted today. Whether related or not, I can help you get back in operation. Can you clarify the name of the Postgres app that has the data you want restored?

Hey @john-fly , we are migrate to another service but we can try one more time with you

the service name: zaia-postgres-ha-prod

This it happened more than once time. Last two were in our staging env.

@john-fly that delay is a REAL problem…our platform is down because that and our customer support is full…

Hi Tiago,

I think the best thing would be to restore your data to a new Fly Postgres cluster. You can do the following to get your data onto a new cluster and point your existing apps to it:

# Create a new Postgres app from one of your existing app's volumes
# (Do this once)
fly pg create --initial-cluster-size 3 --fork-from zaia-postgres-ha-prod:vol_4506j67el8z38z1r -n zaia-postgres-ha-prod-restored
# Repeat from here for every front-end app that connects to the database
# Remove the old database config from app
fly secrets unset -a zaia-your-frontend-app --stage DATABASE_URL
# Add the config for the new database
# NOTE: The below will need adjustment,
# I'm assuming you'll use a user named "restore_user" and the database you
# want to read from is named "prod"
fly pg attach -a zaia-your-frontend-app --database-user restore_user --database-name prod zaia-postgres-ha-prod-restored
fly secrets deploy -a zaia-your-frontend-app

We’ve had to migrate the db from your service. Unfortunately.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.