Staff rotation may cause mismatched bounding box

[yinanazhou]

When Neon gets the OMR output file, Verovio renders the neume position based on the pitch of the neume (to make sure the pitch of the neume match with how it is displayed on the rendered staff), instead of the exact position in the score according to the bounding box. This may cause a mismatched bounding box in Cantus Ultimus when the staff is extremely rotated/wavy.

@annamorphism @JoyfulGen, please feel free to correct me if I'm not being clear.

[JoyfulGen]

@yinanazhou it's very clear! Was a solution for this discussed, finally? Or is this what you're thinking about now?

[yinanazhou]

@JoyfulGen I don't think we have a solution for now.

[annamorphism]

I think it would be good to find a few pages with this rotated staff issue and see what happens to those MEI files outside Neon (i.e. in Cantus Ultimus). @JoyfulGen if you could come up with a few particularly curvy/wavy staff files that you've corrected, maybe @dchiller could think about how much of a problem they create.

[dchiller]

To be honest, even after these multiple conversations at lab meeting, I still feel slightly confused about what the issue is (I suspect this is because I'm a relative newcomer to MEI and have only ever experienced Neon through presentations).

So an example would be helpful! And maybe someone else could try to explain again? But also maybe best to wait for an example.

[JoyfulGen]

Hello! Here's an example from the Neon perspective (as far as I understand the situation):

As you can see, the last staff of Einsie 190v starts pretty straight, but then it gets all curvy and fun:

If I align the Neon staff very strictly with the beginning of the folio staff, you can see that the curve of the folio staff means that Neon is way off at the end:

The pitches of neume components in Neon are calculated by where they fall on the Neon staff. So in a situation like this one, if I want the pitches to be correct, I can't put the neume components exactly where they are on the page. I have to put them on the equivalent line or space on the Neon staff, which means that their true position is misaligned with the original folio image:

However! I realized when investigating this that extreme cases like this one don't often happen, because it's usually possible to fuss a little bit and make the staff align relatively well all the way across. It's not really perfect anymore, but it's close enough all the way through. This looks like that:

Even in my compromise situation, however, the position of the custos is off by a full step. As I understand it, the concern is that the MEI will have the correct pitch, but not the correct absolute position, which will be a problem in Cantus Ultimus when a user will ask for a certain element to be highlighted on the folio image itself.

Here's the video of the demo, if you want to witness the full fussing:

Curvy.staff.demo.mov

@annamorphism, @yinanazhou, @dchiller, did I get that right?

[dchiller]

Thanks for this example @JoyfulGen ! I'm getting it bit by bit!

Even in my compromise situation, however, the position of the custos is off by a full step. As I understand it, the concern is that the MEI will have the correct pitch, but not the correct absolute position, which will be a problem in Cantus Ultimus when a user will ask for a certain element to be highlighted on the folio image itself.

So, if I'm understanding correctly:

1. The pitch information that is contained in the corrected MEI file that is produced by Neon is determined by the position of the rendered neume components relative to the Neon staff lines.
2. In this wavy case, you've been able to fiddle with the Neon staff lines so that both (a) the Neon staff lines and the neume heads are close enough to what's in the image so you aren't having to do too many mental gymnastics around placement when you are correcting and (b) the Neon-rendered note heads and staff lines are in position relative to each other that the pitches in the resulting MEI are correct.
3. EXCEPT: in this extreme case, all that finagling doesn't quite get the custos on the correct pitch.
4. At this time, the bounding box information (absolute position in the image) is derived from the rendered staff lines and note heads.

Is that correct?

And the thing that we have been discussing with Ich is that he would like 4 to be different (that the bounding box information come from the OMR and not from the Neon corrections?

One last question for now: In this case, would it solve the problem (not saying we should do this, but just trying to make sure I understand) for you to do this sort of imperfect staff placement -- things in Neon are a little off from the image but not so bad and all the pitches are correct -- and then just have the custos be the one thing where it looks really off from the image but is at the "right pitch"? In other words, if you did the "I have to place this thing far away from what's in the image so that neon will think it's at the correct pitch" for the custos only? If you did, it would be the case that everything in the MEI is "correct" except to bounding box for that custos?

[annamorphism]

Thanks @JoyfulGen for this great example!
@dchiller regarding your last question: I think in this case, it happens to affect a custos the most because the curve or the staff line (in reality) is such that it can be approximately pretty well by some straight line in Neon for most of its length until the very right hand edge. This would probably be the most common "curvy line" situation on a verso, since the page is mostly flat until it starts dropping off into the binding (dropping into the z-axis, as it were, which reads on the image as the staff line shooting upward or downward with a much higher d/dx). The fatter the book, the more this curve might extend into the staff line--so it could affect some pitched neumes as well as custodes (e.g. arguably the bounding boxes for the very last neume in Gen's example is also a bit off, even after finagling.) And conceivably on a recto you'd have a flipped scenario, where you can get a pretty good linear approximation of the staff for all but the very beginning of it, so you'd get clefs that were slightly displaced. (I'm sure there's some sort of way of expressing all this as a projection of a function on a manifold, but I don't want to think about it right now...)

So I guess the question for me is, what degree of tolerance is acceptable?
e.g. In the pre-"fussing" example, a very tiny box calculated on the position of the "neon neumes" would miss much of the "page neume"

but a box that had about a note-head worth of space around it would still catch the notes in question (they would just be off-center)

[dchiller]

And conceivably on a recto you'd have a flipped scenario, where you can get a pretty good linear approximation of the staff for all but the very beginning of it, so you'd get clefs that were slightly displaced.

Thanks! And clefs are presumably ever more important to have correct in the Neon renderings, because they affect the calculated pitch of any following notes.

So I guess the question for me is, what degree of tolerance is acceptable?

I guess this depends on what the use of the bounding boxes is for. In CU, I already think we draw the boxes too tightly (because you can hardly see what you are looking at in the search results without a little space -- something your two images show very well). But are there use cases for the MEI where the bounding box position of the notehead very specifically is critical?

[annamorphism]

I mean, I can imagine questions that could draw on both an accurate sense of position on the page (bounding boxes) and some sort of musical information from the MEI--e.g. "what is the usual distance scribe A draws her custos from the last note, compared to scribe B?" To my mind they aren't really use cases we're set up to answer, more like exploratory questions you could ask if you happened to have that data; I wouldn't really expect to do them within CU.
I think the main use case for CU is just search, yes? Would you want it if you were to (say) have a "singalong" as we had for Salzinnes in the exhibition (highlighting neumes as they are played)? Is there some other use I'm not thinking of?

[dchiller]

Yeah, I mean, I think in some broad sense Neon should handle wavy staff lines. It's supposed to correct MEI from OMR, so it's contrary to purpose that it would introduce errors to bounding boxes.

At the same time, it sounds like this situation is rare enough and the applications for very exact bounding boxes unclear enough that this is a very minor issue. To your point Anna, do we really want Neon to be in the business of doing the math of lines and manifolds? Probably not.

For CU, as I understand it the use case is to be able to draw a box around the element in the image so that someone can find it more easily on a huge page of neumes... this is true for search, it's true for singalong, etc. So as far as CU goes, I think just expanding bounding boxes a little bit is the best bet.

[JoyfulGen]

Just fyi, in answer to this question:

One last question for now: In this case, would it solve the problem (not saying we should do this, but just trying to make sure I understand) for you to do this sort of imperfect staff placement -- things in Neon are a little off from the image but not so bad and all the pitches are correct -- and then just have the custos be the one thing where it looks really off from the image but is at the "right pitch"? In other words, if you did the "I have to place this thing far away from what's in the image so that neon will think it's at the correct pitch" for the custos only? If you did, it would be the case that everything in the MEI is "correct" except to bounding box for that custos?

This is what I've been doing so far! Extreme cases where the staff is so curved that even fussing doesn't align the neumes are quite rare; most of the time it's really just the custos that's off. In Einsie, I would say that is actually quite common; the absolute position of most of the final custodes of verso folios is off by a step.