Migrate forms to use offline entities

[matthew-white]

One issue around offline entities that we've discussed is what would happen if a Central project has a mix of forms such that some forms use offline entities, while others don't. That could end up being a very confusing experience to data collectors. For example, imagine a project that has four forms and two entity lists: form A updates entity list X, form B also updates X, form C updates entity list Y, and form D reads from both X and Y (but doesn't update them). There are a number of possibilities then with the potential for confusion:

• Form A makes offline updates, but form B makes non-offline updates, so A's updates to entity list X are persisted across forms/submissions on the client, but B's updates are not. X could end up in a weird state on the client.
• Forms A and B both make offline updates to entity list X, but form C makes non-offline updates to entity list Y. In that case, form D will use the latest offline updates from X, but not the latest local updates from Y (only the latest server update).

The solution to this issue that we've discussed is to migrate all existing forms that modify (or even use?) entities. The idea is to automatically convert forms to the latest entities spec (2024.1) so that they are all capable of making offline changes. If they are all converted at the same time, then clients shouldn't end up with a mix of forms, avoiding this issue.

Once forms are migrated, we will need to check going forward that all form definitions that are uploaded (for new or existing forms) are capable of making offline changes. Specifically, we should reject form definitions that specify an entities spec before 2024.1. That shouldn't be an issue for users using XLSForms (things should work seamlessly), because pyxform will automatically specify 2024.1.

In the future, users may wish to opt out of offline entities. We're thinking that that's something that would be configured on the client (perhaps via a QR code setting) and not something that the XForm would determine. In other words, even if we migrate forms now, that doesn't close off the possibility of users opting out of offline entities in the future.

I think we should convert both the current form def (the most recently published def) and the form draft. That's what we do when we convert forms when enabling managed encryption. Central will autogenerate the form version string as part of the migration. If there is an XLSForm on the form def, Central should carry it forward to the migrated form def (related: getodk/central-backend#401).

This might not be needed, but I personally think that it'd be nice if Frontend were to show a message above the forms table when the forms table includes a form that has been recently migrated. The message would inform users that their form definitions have been changed and will need to be re-downloaded to clients. If the autogenerated form version string includes a particular prefix or suffix, then Frontend could check for that prefix/suffix and a timestamp in the last X days when determining whether to show the message.

One thing to note is that even if Central migrates all forms at once, there are rare cases where a client could end up with a mix of forms. That could happen if the form download process is interrupted partway through such that only some forms are updated. If a client does end up with a mix of forms, then it's possible that a submission would specify a non-offline entity update that would result in an entity processing error on the server. That should be a rare case, but once we implement #688, such errors would be surfaced in Frontend.

Another thing to think about is forms that used to be allowed in Central, but are not anymore. For example, it's no longer possible to upload a form definition without a <meta> field, but a server could have an old form definition without a <meta> field. It's possible that such a form definition would cause the migration to fail, so we should check that case. Could we just skip forms like that or bypass requirements like the existence of a <meta> field? How does that work currently when enabling managed encryption? Another example of a form that used to be allowed is a form definition that includes an <entity> field without any action attribute (neither create nor update). Such form definitions were never valid according to the entities spec and would be rejected on upload today, but at some point, it was possible to upload such a form.

[matthew-white]

Currently, I'm thinking that the only thing that the database migration should do is add events to the audit log like upgrade.process.form and upgrade.process.form.draft. That way, it'd be the worker rather than the database migration that would ultimately convert forms to the latest entities spec. In order to convert forms, we'll need to call something like Forms.createVersion(), but as noted in getodk/central-backend#1084, it's best for database migrations not to rely on application code. We could use a worker to call Forms.createVersion() instead.

This strategy was used previously to backfill Enketo IDs for forms and form drafts.

Another thing to think about is forms that used to be allowed in Central, but are not anymore. … It's possible that such a form definition would cause the migration to fail.

I think this is another advantage of using a worker. If a worker tries to create a new version of a form, and it fails for some reason, then that'd be handled in the usual way and won't bring anything down. On the other hand, if a database migration fails, then the server won't be able to complete the upgrade to .2.