Production and staging apps down after being forcibly migrated to v2 platform without warning

Questions / Help
1

I received emails yesterday informing me that our apps had been migrated to v2. Looking at the logs, it seems that neither of them can connect to the fly postgres instances that they were using before. I’m guessing that I need to upgrade those db instances as well, but I’m trying to figure out how. The CLI command for upgrading doesn’t seem to take an -a argument to point at the app.

I’m just quite shocked that this migration was forced without any warning. We’ve been developing for a long time with minimal resources and we’re about to scale up and release in a month or so. This episode is making me consider switching to one of your competitors. All I wanted was an email so I could have gone through this process with development and staging instances first.

2

Initially, we gave notice to people in this forum post. In retrospect, it didn’t have the visibility that we had expected, and we’ve started sending out warning emails with ample time before further migrations.

migrate-to-v2 doesn’t take the -a flag, but you can do it with fly config save -a <app_name> && fly migrate-to-v2. (note that this will write a fly.toml file in the current directory for the postgres app, so you might want to do it in an empty directory)
If you want to, you can do that to migrate your database manually, but you shouldn’t have to - Apps V2 apps can connect to postgres running on V1 totally fine.

Now, let’s get your app running. Here’s some general debugging steps based on other issues we’ve seen:

  1. I’d like you to try to re-set any secrets you might’ve set for the apps. This should’ve been fixed, but we’ve seen an issue involving stale secrets values being used during migrations, as well as some secrets not being transferred correctly.
  2. I also want you to double-check your connection string. If it ends in .flycast, make sure there’s no region code in the URL. (e.g. my-db.flycast is good, ord.my-db.flycast is not)

If that doesn’t help, I’d like you to send an email to support@fly.io and we can take a closer look.

3

Thanks for replying quickly over the weekend.

I don’t know why you expected a forum post to have the necessary visibility. If your product is working well, people won’t be on the forums regularly.

The fly dashboard shows that the health checks for our apps are failing. Noticing this and rolling back the migration would have been a good move, if possible.

It also would have been good if this migration had taken place on our staging app first. Perhaps you could consider having tags for apps to denote whether they are production or not, to help you do this in the future.

If the migration caused secrets to be updated, then this is probably the cause. We have a spring boot app. For the benefit of anyone else who might be using this stack, I’m posting here rather than emailing. I set this up a while ago and I also migrated from Heroku so I’m struggling to remember what I did. I believe that the postgres JDBC driver requires the following convention:

DATABASE_URL=jdbc:postgresql://<host>:<port>/<dbname>?user=<username>&password=<password>

I also think that &sslmode=disable needs to go in there. However, I’ve tried resetting the secret without any success. I’ve also tried creating a brand new postgres cluster, abandoning the data we had, and using those details in the DATABASE_URL. No joy.

2 Likes
4

Creating a new postgres cluster output the following

Postgres cluster <cluster-name> created
  Username:    postgres
  Password:    [redacted]
  Hostname:    <cluster-name>.internal
  Flycast:     
  Proxy port:  5432
  Postgres port:  5433
  Connection string: postgres://<username>:<password>@<cluster-name>.flycast:5432

In order to convert this to the format required by the JDBC driver, I need the dbname. How can I confirm what this is?

5

It is a good thing to do but you don’t need to upgrade the postgres instances to get anything back.

Try running fly postgres attach POSGRES_APPNAME -a YOUR_APPNAME , that will print the datbase uri to convert to the JDBC format your app uses.

i.e.:

~$ fly pg attach -a myapp dangra-db1
Checking for existing attachments
Registering attachment
Creating database
Creating user

Postgres cluster dangra-db1 is now attached to myapp
The following secret was added to myapp:
  DATABASE_URL=postgres://myapp:xxxxx@dangra-db1.flycast:5432/myapp?sslmode=disable

and then use fly secrets set DATABASE_URL=jdbc://... to update the secret your app uses to connect to postgres db

1 Like
6

@dangra Thanks for the help.

When I execute the attach command for the existing database I get this.

Error: no 6pn ips found for <database> app

When I execute the same attach command for the new database that I created in my previous message, I gett a different password from the one I redacted above. I don’t understand why that happened. Any ideas? The good news is that the app now connects to the new database, but I cannot connect it to the previous one.

7

Reading the docs on fly postgres attach I see that it creates a new user (and password).

I still don’t understand why setting the DATABASE_URL directly using the postgres username and password didn’t work, though.

8

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.