Concerns about data synchronization and the full launch of NewCantus

[jacobdgm]

I still have doubts/questions about how we are going to handle data synchronization between OldCantus and NewCantus around the time of the full launch.

The crux of the matter is that it takes ~2 days to run our data synchronization scripts, and while we're running the synchronization process, one of two things will happen: 1. we find a way to prevent people from adding data to OldCantus during that time, so that we can make sure not a single change made on OldCantus is missed, or 2. some changes made to the data in OldCantus will be lost.

• In situation 1., I'm sure there are a number of students hired as RAs to enter data into CantusDB (I was once one such RA), who are depending on the income from this job to pay rent and tuition, for whom we would create difficulties if we had to tell them not to work for several days.

• In situation 2., a bunch of work will be lost. Not only is this taxpayer money that we're wasting by paying people to do the same work twice, but it's possible that the lost data will not all be in one chunk (e.g. if anyone is skipping around while working on a manuscript and we lose a day's worth of changes, we will create random gaps in manuscripts rather than erroneous records in a contiguous block of folios), making it even more difficult to correct these lacunae following the full launch.

Probably, this boils down to "let's ask Debra and see what she thinks is the best approach." Perhaps this has already happened, and news of her instructions hasn't yet reached me. But I just want to make sure we've weighed the likely effects for all relevant stakeholders, whichever approach and launch timeline we choose.

Ultimately, it would be easy for me to say, "we'll just assume that the data on OldCantus won't change during the synchronization process, and when this assumption proves to be false, that someone else's problem/responsibility and not mine". But many people depend on Cantus Database, and I (and all of us!) have a responsibility to do what we can to get this right. If the launch process doesn't go smoothly, it will shine poorly on me, @lucasmarchd01, @fujinaga and Debra, on the DDMAL lab, and on Cantus Database as a whole, and has the potential to create hardship for a bunch of people.

Tagging relevant people: @fujinaga, @annamorphism, @lucasmarchd01; @jwang44 may have some insights too.

[jacobdgm]

Here are a couple of possible solutions the concerns outlined above (thanks to @dchiller for his thoughts on these):

1. One of the main difficulties outlined above is that it takes so long to run the synchronization scripts: for each of the ~10^6 records in the database, we make a call to one of OldCantus's APIs. Even if we can complete and process ten queries per second, this amounts to more than 24 hours.
   • I can imagine a data synchronization process where we get all the relevant data from a database dump (I don't understand why this approach wasn't taken in the first place, so maybe there are difficulties associated with this approach? @jwang44?) rather than making a million calls to an API. If we set this up, I can imagine the full data synchronization process completing within a few hours rather than a few days. But a new process will take time to write and (importantly!) to test.
2. We can change when we plan to synchronize the data. If we do it on a weekend, or (for example) during the first week of September when many university-related people will be busy running/attending orientation week activities, we can likely minimize the number of people making changes to the old database.
3. If we give people more lead time, we can give people time to make plans accounting for the transition. For example, we can tell people, "on {these days}, do whatever you would on Cantus Database, but enter your changes into a spreadsheet so they can be easily copied into CantusDB once the transition is complete". This would be a good idea to do, whether or not we implement bullet points 1. and/or 2. above. But we don't have much time to do this for a first-week-of-July launch.

[annamorphism]

A 2-day stoppage is doable--there is a holiday around the beginning of July in both Canada and the US, after all, and it's not a very large number of RAs affected (i.e. who would need to be notified). So a sync that happens over the course of a (long) weekend is within reason. (This--your situation 1--was discussed in a conversation with Team Cantus at some point, but I don't have it in writing). If it has to be down for longer than that for subsequent testing, though, I think it might be worth looking into your Point 1 in order to streamline matters (personal opinion, not representing any other Cantus stakeholders.).

[jwang44]

Yes, getting chants from a database dump would be great, but it's difficult. If you take a look at the mysql database, you will notice that the fields associated with the chant model are scattered in dozens of separate tables (1-2 tables for each field if I remember correctly). It is possible to somehow collect them with an automated approach, but it would likely be very messy. The json API, as far as I know, is the only place where all fields are in one place.

Suggestion:
Instead of running the sync_chant script as-is, you can adjust the script to only fetch the chants that have been updated since your last sync. The script fetches chants by ID. I assume the vast majority of chants in the database would stay the same. Only a few of them (maybe 200) will be updated during the full-launch transition period. If you can figure out which objects (including but not limited to chants) have been updated during the transition period, you can fetch only those objects and add/update them in NewCantus.

[annamorphism]

Ah, excellent. I can tell in OldCantus's admin interface which 180 items (sources and chants) have been updated since Monday, for example, and with a little ingenuity it should be possible to get their IDs.

[jacobdgm]

If you take a look at the mysql database, you will notice that the fields associated with the chant model are scattered in dozens of separate tables (1-2 tables for each field if I remember correctly). It is possible to somehow collect them with an automated approach, but it would likely be very messy. The json API, as far as I know, is the only place where all fields are in one place.

This is useful information, and explains things - thank you!

Ah, excellent. I can tell in OldCantus's admin interface which 180 items (sources and chants) have been updated since Monday, for example, and with a little ingenuity it should be possible to get their IDs.

If we have the IDs of these sources and chants, we should be able to run an adapted version of the script to update these. It will mean a second step in the synchronization process (rather than just running the script once), but this has the potential to alleviate time-to-run-the-synchronization concerns. Tagging @lucasmarchd01, who's been working with the synchronization scripts.