investigation into how many Cantus IDs in CD don't exist in CI

[jacobdgm]

working notes:

there exist ~30000 unique Cantus IDs in Cantus Database.

Cantus Index says it has ~60000 Cantus IDs. I'm surprised that it has twice as many as CD, but I guess the other databases in CN deal with other, quite distinct, repertoires.

[annamorphism]

very interested in the answer to this question! if there are more than 10 there's probably sort of problem with how IDs are being used that needs to be looked into...

[jacobdgm]

how important is it that this question be answered, and on what timeline?

We don't have a list of Cantus IDs present on CI (perhaps you're aware of one somewhere, @annamorphism? Eventually, Jan's One API To Rule Them All will provide this, but we don't have it yet). There are two ways I can think of to test our list against theirs:

• check https://cantusindex.org/id/{Cantus ID} for each Cantus ID in our list and see if we get an actual page or a 404 error. I've been trying this, and a back-of-the-envelope calculation indicates it will take about 5 hours, requesting about 100 pages/minute
• try to scrape all 1188 pages of https://cantusindex.org/chants to compile a list of IDs on CI, and then compare it to our list (this would take much less time to execute, but I'm not sure it's worth the time to set up.)

[annamorphism]

I don't think it's particularly important--if it comes up on the way to solving something else, or there is a simple way of doing it, it would be interesting. I'm not surprised that there are a lot more IDs in CI than in CD (though I didn't realize it was 30000 more!), but there really shouldn't be many in the other direction. (I would say "not any" except I know of two anomalous IDs for placeholder chants, but there shouldn't be any others I don't know about...)