PostgresDB stuck on 'pending' for an hour+, can't view logs

I used flyctl launch to attempt to deploy an existing rails app. After selecting the options, I was provided the postgres credentials, which I noted. The deploy appeared to stall.

I ctrl-c’ed and tried launch again. This time the rails app went through the Docker build and deploy, then crashed when it attempted to create and migrate the db, as it couldn’t connect to the postgres instance.

I then used fly apps list, which showed the db was still pending, for over an hour. I tried restarting. Still stuck in pending. Tried scaling up. Still stuck in pending.

Not a great ‘speedrun’ so far to test out this deploy…

hey there-- welcome! happy to do what I can to help make the rest of your initial runthrough much smoother.

If you’re not immediately sure where to start with troubleshooting or get stuck, this checklist is super handy:

For this particular issue, you might wish to re-create the pg app with LOG_LEVEL=debug prepended (eg LOG_LEVEL=debug fly pg create, which will give you a clearer idea of where things are getting held up. Let us know if you have any questions about all of this!

1 Like

I should perhaps add that particular checklist may not help in this case, given this appears to be a fault with the database (rather than with the rails app). That checklist is more for app issues.

What I would do here is split up deploying the app and deploying the database (if that’s possible). And that may help narrow down the issue. Rather than doing both at once and have e.g the app fail because the database failed.

So first run fly launch to get the app deployed. When it asks if you want a database, say no. Assuming it’s able to deploy without a database (like, you don’t have a healthcheck that will only pass with one), that gets that done.

Then, as @eli says, try LOG_LEVEL=debug fly pg create.

Since you can create the two separately and then subsequently attach the database to the app. Take a look at Postgres on Fly · Fly Docs

That provides the app with a DATABASE_URL secret which is how many apps expect it to be provided.

It may be a temporary issue but the debug output will help figure it out.

1 Like

I tried what @greg recommended. I destroyed the two apps created by the Rails helper. Deleted the Dockerfile and fly config.

But after running $ LOG_LEVEL=debug fly pg create it’s been issuing the same status for going on 15 minutes now:

DEBUG <-- 200 (123.58ms) {"data":{"app":{"deploymentStatus":{"id":"e30d60da-09ca-6e01-0078-0301e3cfa878","inProgress":true,"status":"running","successful":false,"description":"Deployment is running","version":0,"desiredCount":1,"placedCount":0,"healthyCount":0,"unhealthyCount":0,"allocations":[]}}}}

Over and over. Thoughts?

Can you give it one more try? I think you deleted that most recent one right as I was digging into it. Everything looks ok, I’m wondering if the deploy status isn’t getting updated properly.

Yes, I just ran it again. This time it repeated the same call and response cycle for about a minute and then ended successfully. Any idea what’s going on?

Vaguely. It seems like Nomad (we use this for orchestration) isn’t always detecting the volume on the specific host you’re getting assigned, so it doesn’t think it can boot the Postgres VM. This last one went through, probably because I was staring at it.

You should be good to go now, sorry for the difficulties here. We’ll keep an eye on this host but that specific issue won’t affect anything once the DB is up and running.

Ok, thanks. I’ve now attached the created app to the postgres instance with:

flyctl postgres attach --app <app-name> --postgres-app <postgres-app-name>

But it still shows the app-name as pending. I restarted it and it remains unchanged. Running fly logs -a <app-name just hangs. Thoughts?

Currently running fly deploy. Let’s see if that works…

Deploy is correct. The <app-name> never got deployed, it was stuck waiting on Postgres stuff to happen.

Alright, we’re up and running. Thanks all. (Now to debug my asset pipeline errors…)