Help with Prisma migrations in prod

[arpitdalal]

I accidentally broke a migration in prod that worked okay in dev because I didn't have conflicting data in dev. The migration was flagged as failed in the prod DB, then I fixed the schema, created another migration, and deployed the app but still got the same error that no new migrations will be applied. I tried to deploy an older version of the app that didn't have this said migration but that deployment failed too saying that no new migrations will be applied. The fly app machine is offline, so I cannot connect to it and follow the steps from Prisma docs. I don't think deleting the migration generated by Prisma would help because that migration is flagged as failed in prod DB. Any help is appreciated, thanks you
Here's the error:

migrate found failed migrations in the target database, new migrations will not be applied. Read more about how to resolve migration issues in a production database: https://pris.ly/d/migrate-resolve

[kentcdodds]

Dang, I'm not sure what the best course of action would be in this scenario. The first thing I would try is to make it so my app can start up so I can download the database and back it up (especially if you don't have the LiteFS cloud backups enabled). From there you could actually fix it all locally and then upload it and maybe your job would be done.

To get the app started, you could deploy a temporary app that's just a simple node server that passes health checks (like a hello world express app would be enough). Once it's up and running, then you should be able to get in there and do stuff.

There's probably a better way to look into a volume when the app's not running, but I'm unaware of what that might be.

[swalker326]

I've been in this exact spot before, I was fortunate that I was still in development and none of the data on my app was real data. So i ended up deleting the app and recreating the app with the same name. Not sure if this is applicable for you or if you need the data in your app.

Its really annoying and there has gotta be a way to start the instance and skip running migrations or just don't run a start command. All I needed was the instance running so I could ssh into it and fix it. I just wasn't able to figure it out.

[kentcdodds]

I'll bet there's something that we could do to make doing this exact thing really easy. Just need to think about it a little bit.

[arpitdalal]

This is how I fixed it:

TLDR:
Run the app without exec section in litefs.yml file and then re-deploy the app. Remove the problematic prisma migration from the prod db and migration SQL file. Re deploy the app with the exec section in litefs.yml file.

Steps:

1. comment exec section in litefs.yml
2. deploy the app
3. ssh into it
4. litefs export the db
5. sftp download the db to the computer to backup
6. make a copy of this db
7. remove the last entry (failing entry) from _prisma_migrations table from the copied db
8. sftp upload the copied db to the app
9. litefs import this db
10. remove the problematic prisma-generated migration SQL file from the source code
11. un-comment exec section in litefs.yml
12. commit these changes and re-deploy the app

Note: steps 6 to 9 can be omitted by running SQL to remove the problematic entry from _prisma_migrations table right in the SSH session.

[kentcdodds]

Thanks for sharing that! Would you be interested in adding something about this to the docs?

[arpitdalal]

No problem and sure.

[arpitdalal]

Created this