Migrations

[fwbrasil]

Decide if Quill should have a migration module. If not, document the recommended library for migrations.

[b0c1]

I prefer typed migration (activate framework like...?)

[cvogt]

@lastland's Scala Forklift is a framework agnostic tool to manage versions and apply migrations written in Scala. Worth checking out as a potential solution to one piece of the puzzle.

lastland/scala-forklift#14

https://github.com/lastland/scala-forklift

[fwbrasil]

@cvogt thanks for the pointer! :) I didn't know about forklift.

[cvogt]

@fwbrasil It was developed last summer. @lastland is a smart and dedicated guy, I am sure he is willing to help with the integration should this be considered.

[cvogt]

interesting about forklift is also that it has a notion of working while developing in a world of branches ... switching the schema when switching the branch

[lastland]

@cvogt Thanks for the reference!

@fwbrasil Would be glad if I could help on this issue. 😄

Now Forklift makes use of slick-codegen to maintain database models in the code. I'm not very sure yet how this works with quill. Perhaps need a code generator for quill? (it seems quite easy, since it only need to generate a case class for each schema)

And I haven't found any support for alter table operations in quill. Or did I miss something?

[fwbrasil]

@lastland, quill doesn't have support for DDLs. Let me have a look at your library in the envening so we can discuss it further.

[cvogt]

@lastland I think a starting point for forklift with quill would be untyped plain sql and no code generation.

[lastland]

Hi, I have just implemented a prototype to demonstrate how to use Forklift to do database migrations on Quill: https://github.com/lastland/scala-forklift-quill

This demo still contains some hard code and unimplemented features, but I believe it's sufficient to show how it works.

Please refer to README to see how to play with it. I have also discussed a challenge there.

Feel free to reach me if you have any questions.

[fwbrasil]

@lastland Thanks for putting together the example project, I'm sorry for taking so long to give you feedback on this. I was focusing on releasing the 0.2.0 version.

Quick question about forklift: does it support reverting the schema to previous versions (similar to ruby's up/down mechanism)?

[cvogt]

@fwbrasil What's your use case for down migrations? I know some frameworks offer them (I don't know about forklift), but how important are they in reality?

[fwbrasil]

@cvogt Activate provides this feature and it is very convenient, specially for very large schemas that can't be easily recreated from scratch.

[lastland]

@fwbrasil Hi, Forklift does not currently support down operations. I don't know much about the down operations' real-world use cases, but I imagine (correct me if I'm wrong) the most use cases are:

1. Revert the previous migration(s) permanently.
2. Use "revert to" to get to a specific version of database and write some code, then probably "update" back.

For the first case one can define another migration. For the second case I think one can use the Forklift's git integration (mentioned by @cvogt before) to keep different db versions in different git branches.

The git integration is not demonstrated in the example code. The basic idea is to store some information to recover the db with the git branches. The information can be a copy of the original database, a dump file, a sequence of migrations, or any other information sufficient to recover the database provided by the developer or some other libraries. You can find a bit more information here. PS: Forklift's current git integration features are only experimental and quite slow, but we plan to further explore this direction in the future.

Does this make sense?

[cvogt]

@fwbrasil right, but in what situation would you want to go back to an older version?

[fwbrasil]

@fwbrasil right, but in what situation would you want to go back to an older version?

• Running a local database with the latest version, I can go back to an older revision, fix an issue, and switch back to master.
• New version has a severe bug and needs to be rolled back, the database schema also needs to be rolled back.

[fwbrasil]

For the first case one can define another migration. For the second case I think one can use the Forklift's git integration (mentioned by @cvogt before) to keep different db versions in different git branches.

I see, it seems to be an alternative. I'd consider up/down migrations a better solution for both scenarios though.

[cvogt]

Here are a few thoughts:

• the so far stated use cases for down migrations were development scenarios, which give us a much easier time because we don't have live data to retain. use cases for down migrations (instead of undoing up migrations) with live data are thinkable, but seem much much rarer
• for development scenarios, where live data is irrelevant, reapplying up migrations from the beginning of the time basically gives you "down" migrations for free at the expense of performance
• real down migrations are probably easy to integrate into forklift as it already has the concept of switching between database states and down is purely a performance optimization for it
• a up migration undoing a previous up migration IS a down migration only that it will appear in history
• in the very general model forklift follows, where migrations can be literally arbitrary scala code, down migrations cannot be derived from up migrations and may actually be impossible, if the up migration was destructive
• for some use cases up and down migrations can be generated from a single declarative description. @nafg's slick-migration-api project allows that for some of DDL AFAIK

[hlegius]

Just curious: is it yet in progress? thanks! 🥇

[guizmaii]

I'd advise people to use something like flyway or liquibase. We don't plan to implement this feature in Quill.