Storing MEI files for using in Cantus Ultimus production

[dchiller]

@fujinaga @JoyfulGen @annamorphism @yinanazhou @martha-thomae

I'm opening this discussion about how we will use MEI files in Cantus Ultimus production by tagging some people that I think might have insight into the best way to do this; feel free to tag others or mute as needed.

Big question: Where will production-ready MEI files be stored so that they can be used as part of OMR search for Cantus Ultimus?

Some sub-questions, with more background below.

1. When a folio is corrected in Neon at the end of an OMR process, where does the resulting mei file go currently? (there are some on the Neon repository (deployment/server/samples/mei/) and some on the ddmal.music.mcgill.ca website, but neither seem like there are actively updated)
2. When a folio is corrected in Neon at the end of an OMR process, where does the resulting Neon manifest go? (again, there are some on the Neon repository (deployment/server/samples/manifests/) but they don't seem totally up to date)
3. For 1-2, is that how it is envisioned working long-term or just for now?
4. What manuscript has the most complete version of MEI files?

Background:

I'm beginning to work on re-implementing OMR search at long last in Cantus Ultimus. Yay! Soon we'll have a place to see and use OMR workflow results in action. The last working version of OMR search on Cantus Ultimus that I have access to seems to store the mei files in the Cantus Ultimus repository and then in the vm file system. I don't think it makes sense to continue in this vein for a few reasons:

1. It doesn't really make sense to have the mei files sequestered away in the Cantus Ultimus repository when they are potentially useful for other purposes outside of Cantus Ultimus
2. As the number of manuscripts served and the number of mei files grows, it feels a bit unwieldy as a management solution.
3. I imagine there is already some place that corrected mei files are going and I know that that place is not the Cantus Ultimus repository, so it seems like in reconfiguring Cantus Ultimus to use the mei files it would just make sense to hook into what already exists.

Additionally, Nestor and I last summer initiated the production_mei_files repository (https://github.com/DDMAL/production_mei_files), the idea behind which was to store mei files ready for serving in production, as well as some validation utilities. This is an option, and it does have the benefit of allowing for version control of the mei files, but does not necessarily have to be the solution.

[dchiller]

Ah it looks like the answer to 1 & 2 is: https://github.com/DDMAL/e2e-omr-resources/tree/main/resulting_mei_files

[fujinaga]

For your question No. 3: I think Salzinnes is the most complete MEI file.
Note that we need to incorporate a mapping file (used in the workflow) for each manuscript to be able to search by the name of neumes.

[martha-thomae]

Answering your questions:

1. Yes, that is the site where the files have been stored: https://github.com/DDMAL/e2e-omr-resources/tree/main/resulting_mei_files. For Salzinnes, they should be in the reviewed folder. For Einsiedeln, I don't know as those are still in progress, but @JoyfulGen would know (maybe the ones in the reviewed_once folder?)
2. I am not sure what that is or how to obtain those. @yinanazhou ?
3. Just for now, I made the resulting_mei_files folder to store the MEI files somewhere. I thought that this repo was ok right now given that all the resources to run the OMR workflow for the various manuscripts are here already, so we could put the output of the workflow runs here as well.
4. Yes, it would Salzinnes.

Additionally, Nestor and I last summer initiated the production_mei_files repository (https://github.com/DDMAL/production_mei_files), the idea behind which was to store mei files ready for serving in production, as well as some validation utilities. This is an option, and it does have the benefit of allowing for version control of the mei files, but does not necessarily have to be the solution.

That option sounds good to me.

[dchiller]

Just for now, I made the resulting_mei_files folder to store the MEI files somewhere. I thought that this repo was ok right now given that all the resources to run the OMR workflow for the various manuscripts are here already, so we could put the output of the workflow runs here as well.

Could we just use this then? Do you think there is a good reason to have a separate repo like production_mei_files?

[dchiller]

I guess it is always good to have a back-up, but maybe that can still be the Google Drive and OwnCloud versions? Although, whose account are they on and are they accessible after people leave, etc.

[martha-thomae]

Could we just use this then? Do you think there is a good reason to have a separate repo like production_mei_files?

Yes, I think we could use them from here, no need to have a separate repo.

I guess it is always good to have a back-up, but maybe that can still be the Google Drive and OwnCloud versions? Although, whose account are they on and are they accessible after people leave, etc.

Yes, of course. I only asked this to make sure we were still keeping copies. I agree to keep Google Drive and OwnCloud as backup and not make another repo for this.

They are both under the ddmal lab manager account, credentials should be in the wiki. The ddmal lab manager account in gmail has a two-factor authentication thing set up to Ichiro's phone.

[dchiller]

Perfect! I think this all makes sense.

[martha-thomae]

One question, @JoyfulGen and @annamorphism, do we have a second and third copy of these files stored somewhere? I remember setting something up on OwnCloud(?). Are we still using this?

(Sorry, @dchiller, this is not 100% related to your question, I am just wondering if we are still keeping copies just in case something happens.)

[JoyfulGen]

One question, @JoyfulGen and @annamorphism, do we have a second and third copy of these files stored somewhere? I remember setting something up on OwnCloud(?). Are we still using this?

Yes! There's a copy of the files in Google Drive https://drive.google.com/drive/u/2/folders/1uKqbjHATAssLbi048G8Tm_nKDzGfpOQk. There should also be one in OwnCloud, but I can't find it... @annamorphism please help?