I accidentally created a failing migration using Prisma which actually worked on my development but failed in prod, I fixed the migration and deployed again but now I get this error:
migrate found failed migrations in the target database, new migrations will not be applied. Read more about how to resolve migration issues in a production database: https://pris.ly/d/migrate-resolve
Now, the machine is offline because of this error, so I cannot even follow the steps from the Prisma docs.
How do I resolve this? I’m not 100% sure if deleting the failing migration would actually work since that migration is marked as failed in the prod DB. Any help is appreciated, thank you.
Have you tried https://fly.io/docs/flyctl/console/?
Yes, it says this:
`
╰─ fly ssh console --app xman ─╯
Error: app xman has no started VMs.
It may be unhealthy or not have been deployed yet.
Try the following command to verify:
fly status
`
Try fly console instead of fly ssh console. That will create a new machine for the life of your session.
That did work. But there’s no DB in the location specified by my dockerfile
root@5683ddd0ad6038:/myapp# cd /litefs/data
root@5683ddd0ad6038:/litefs/data# ls -la
total 8
drwxr-xr-x 2 root root 4096 Nov 28 20:09 .
drwxr-xr-x 3 root root 4096 Nov 28 20:09 ..
root@5683ddd0ad6038:/litefs/data# cd -
DockerFile
ENV LITEFS_DIR="/litefs/data"
ENV DATABASE_FILENAME="sqlite.db"
ENV DATABASE_PATH="$LITEFS_DIR/$DATABASE_FILENAME"
ENV DATABASE_URL="file:$DATABASE_PATH"
Running Prisma migrate status gives me this
root@5683ddd0ad6038:/myapp# npx prisma migrate status
Prisma schema loaded from prisma/schema.prisma
Datasource "db": SQLite database "sqlite.db" at "file:/litefs/data/sqlite.db"
Error: P1003: Database sqlite.db does not exist at /litefs/data/sqlite.db
I assume this is because there’s no DB connected to this machine?
Ah, I didn’t realize that you were using litefs. fly console won’t run the ENTRYPOINT/CMD from your dockerfile, and it is a byproduct of running those that sets up litefs. You will need to run that, but do so in a way that gets you back to a shell prompt.
Ah yes, sorry, forgot to mention before that I’m using litefs. I’m using epic-stack as the base template for this app and I haven’t changed anything in dockerfile. How can I create a new instance of this app to debug it?
Assuming your Dockerfile looks like this:
… then that’s the command you would want to run. That, in turn will run the following:
If you run fly console with no parameters, it will run the image you last deployed. If, instead, you pass --dockerfile Dockerfile, it will build a new image and run that. That might be the quickest way to proceed here: temporarily adjust litefs.yml and/or package.json. I’m guessing that you don’t want to run migrate deploy or npm start, but perhaps /usr/bin/bash might be useful.
Thank you very much for looking into this @rubys!
So, I should remove this part from litefs.yml
- cmd: npx prisma migrate deploy
if-candidate: true
- cmd: npm start
and run fly console --dockerfile Dockerfile which will build the app but won’t run it? I think the health checks wouldn’t also pass, would that make any difference? Should I just skip the npx prisma migrate deploy command and not the npm start to start the app that will pass the health checks?
Disclaimer: I haven’t tried these specific instructions myself as I don’t have an epic stack app handy.
My suggestion is to replace those lines with:
cmd: /usr/bin/bash
Then run fly console --dockerfile Dockerfile.
This will build an image, launch a machine, and shell you into it. Once there, run litefs mount, which should mount the database and then as a final command will return you to a bash shell.
From there, you should be able to run commands and access the database.
If you remove (or comment out) the whole exec section of the litefs.yml then LiteFS will start up and just run without trying to start anything else. It’ll continue running until you kill the litefs process or the instance itself. Then you should be able to ssh into the instance and work with the database without your npx application server trying to run as well.
I did do that but after running litefs mount, I’m constantly getting these message
level=INFO msg="cannot become primary, local node has no cluster ID and \"consul\" lease already initialized with cluster ID LFSC0E7BDBE40B20025D"
level=INFO msg="cannot find primary, retrying: no primary"
I have my primary_region set to yyz and I ran the console command with setting that as well
fly console --region yyz --dockerfile ./other/Dockerfile
Still getting this message and I cannot stop it.
Do I run litefs mount after? because after pushing the docker image to fly, there’s no DB for prisma to work with.
and my primary region is set to yyz too
root@17811122f5d348:/myapp# echo $PRIMARY_REGION
yyz
LiteFS tries to avoid connecting to a cluster with a different cluster ID but you can also get in the situation if the cluster ID that’s persisted to disk is removed. There’s some recovery options in the docs here: Disaster Recovery from LiteFS Cloud · Fly Docs
We’re also looking at removing this feature because it tends to be more of a pain than it’s worth.
1 Like
What is the best way to get litefs to mount SQLite? I don’t want to lose data from my primary machine but cannot get that up and running. And does running litefs mount mounts the primary DB or spins up a new DB?
The easiest fix is to change your Consul key in litefs.yml and redeploy: Disaster Recovery from LiteFS Cloud · Fly Docs
Okay, I changed the lease key and ran fly console with the region and dockefile command and then ran litefs mount. But now how can I SSH into this machine? Also, is there a way to destroy or stop machines by their ids or something?
This is how I fixed it:
TLDR:
Run the app without exec section in litefs.yml file and then re-deploy the app. Remove the problematic prisma migration from the prod db and migration SQL file. Re deploy the app with the exec section in litefs.yml file.
Steps:
- comment exec section in
litefs.yml - deploy the app
- ssh into it
- litefs export the db
- sftp download the db to the computer to backup
- make a copy of this db
- remove the last entry (failing entry) from
_prisma_migrationstable from the copied db - sftp upload the copied db to the app
- litefs import this db
- remove the problematic prisma-generated migration SQL file from the source code
- un-comment exec section in
litefs.yml - commit these changes and re-deploy the app
Note: steps 6 to 9 can be omitted by running SQL to remove the problematic entry from _prisma_migrations table right in the SSH session.
1 Like
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.