How to deal with problematic upgrades

[tailhook]

This discusses approaches for: geldata/gel-cli#976

Local glossary:

• shared database -- production, backup, staging, etc. databases that contain valuable data
• dev database -- local, staging, etc. databases that can be revered or loose data on ocassion

(yes staging is in some sense a shared database, and in some sense dev database)

Upgrade Check

edgedb migration upgrade-check
edgedb migration upgrade-check --watch
edgedb migration upgrade-check --to-testing

Not edgedb watch because:

1. We have to have a command to run the check once (this is a primary command to run before upgrading, --watch is only needed once you did the check and figured out that something is wrong and are actively editing the schema)
2. It has to have all kinds of version aguments --to-testing, --to-nightly, --to-channel, which don't make sense for watch

What it does:

1. Spawns a teporary instance
2. Applies the latest schema and checks whether it's okay
3. Applies all the migrations and checks if it's okay

So it has one of the two outcomes:

You're all set, feel free to upgrade

You're schema is compatible, but migrations are not. Please do the squash of the schema before upgrading.

(the messages are rough, we should provide actual instructions for the latter)

Not-watch mode can also indicate a schema error:

The schema is not compatible to the EdgeDB 3.0. Please fix the errors above.
hint: Use `edgedb migration upgrade-check --watch` for faster feedback loop.

If upgrade check is clean without squashing, user may proceed the upgrade without further work. So the following text only discusses failure scenario

Here are three possible ways to implement failure workflow:

No-Bookkeeping Workflow

edgedb migration create --squash

1. Requires no schema changes comparing to current migrations
2. Replaces all revisions to 0001.edgeql that records previous one as
   annotation:

   CREATE MIGRATION ... {
       SET squashed_from := "m1askdfjasdihslkjvdahfowerafahd";
   }
   

In this case, edgedb migrate will apply to the database following these
guidelines:

1. Only first migration can contain squashed_from
2. If the first (squashed) revision in the database, migration proceeds normally
3. If squashed_from revision is the latest, it's replaced with the new first revision
4. If squashed_from revision is not in the database or not the latest one, it's the fatal error

From user point of view full process looks like this:

1. Run migration update-check --watch and fix the schema
2. Run (normal) migration create
3. Push all the revisions to all shared databases
4. Run migration create --squash
5. Do dev database upgrade and check your app
6. Push that to all shared databases
7. Do shared database upgrade

Fixup Workflow

edgedb migration create --squash

Does two things:

1. Replaces all revisions into 0001.edgeql
2. And creates a fixup file that contains a difference from the latest migration in the filesystem to schema:

   dbschema/fixups/<old_revision>-<new_revision>.edgeql
   

   Fixup file is created via normal (interactive) migration construction process.

In this case, edgedb migrate when encounters a revision that is not in the database and can't be upgraded to:

1. Looks for a fixup file <db_revision>-<file_revision>.edgeql
2. If that fixup exists applies migration and does rebase
3. If fixup for 00001.edgeq does not exist, repeat process for the next revisions

From user point of view full process looks like this:

1. Run migration update-check --watch and fix the schema
2. Run migration create --squash
3. Do dev database upgrade and check your app
4. Eventually push your change do the shared databases
5. Do update of shared databases once migrations are applied

Advantages of this workflow:

1. Squashing can be done right away (with the no bookkeeping workflow you have to deploy migrations to all the shared databases before you can do squash)
2. Fixups from multiple revisions can coexist
3. Fixup files could be kept indefinitely, to restore from backups (although, this only works for backups having exactly the same latest revision)
4. Can fixup arbitrary revision, i.e. shared database that was deployed from a branch, or to recover from DDL (comparing to Diamond Workflow)
5. Fixups can contain data migrations specific to this upgrade

Diamond Workflow

This allows keeping multiple revision hierarchies.

edgedb migration create --squash

1. Requires no schema changes comparing to current migrations
2. Creates a combined revision:

   dbschema/migrations/00001-00265.edgeql
   

In this case, edgedb migrate:

1. Applies all revisions up to 265
2. Rebases to use 00001-00265.edgeql instead
3. Proceeds next revisions normally
4. New instances skip fine-grained revisions

From user point of view full process looks like this:

1. Run migration update-check --watch and fix the schema
2. Run migration create --squash
3. Do dev database upgrade and check your app
4. Eventually push your change do the shared databases
5. Do update of shared databases once migrations are applied

Advantages of this workflow:

1. Squashing can be done right away (with the no bookkeeping workflow you have to deploy migrations to all the shared databases before you can do squash)
2. Old revision files could be kept indefinitely, to restore from backups
3. Revisions from git branches keep their numbers (although have to be rebased anyways)

Cons:

1. Fine-grained revision that are left could contain code that is invalid in the new edgedb. And that is confusingly stored in the git (even though, we can allow deleting those files, users will unlikely be doing that).

Conclusion

My choice is "Fixup Workflow" for now. As it's the most versatile, and is also easier to implement that Diamond workflow, and also equivalent in convenience.

The following scenario is fine under Fixup workflow and particularly bad under both of the alternatives:

1. Schema is updated and squashed
2. Some dev instances are upgraded (staging, local)
3. Bug is found in the application that requires changing schema again
4. No bookkeping: best way to proceed is revert git and redo process again, dropping/restoring from backup all the databases that has already adopded squashed schema
5. Fixup: squash the schema again and provide fixup both from both parents.

Note: No-Bookkeeping Workflow may look like a subset of the Fixup Workflow. But that depends on how bookkeeping of "empty fixups" is done. In No-bookeeping workflow that is recorded in the squash migration. I think that storing that in fixup file is more robust, as modifying an annotation of squashed revision changes its hash.