Update links to CantusDB (cantus.waterloo.ca --> cantusdatabase.org) #765

dchiller · 2023-08-10T17:56:53Z

This PR changes all urls for CantusDB in the Cantus Ultimus code base from cantus.waterloo.ca to cantusdatabase.org. Also uses https rather than http for all requests to CantusDB.

Although the /genre page redirects to /genres on NewCantus, this PR changes the url in helpers/scrapers/genre.py to go directly to /genres rather than through the redirect.

Closes #761 along with DDMAL/cantus-staticpages#5

dchiller · 2023-08-10T18:56:22Z

It looks like the failing test is because we made the change to the /genre(s) call in anticipation of the redirect from /genre to /genres getting up on CantusDB (which is understandably isn't yet). I think we can just hold off on merging this PR until that gets up on production.

Significant changes since approval

app/public/cantusdata/helpers/source_importer.py

jacobdgm

Important necessary change: we can't use /node/ paths.

jacobdgm · 2023-09-14T19:39:06Z

nginx/public/node/frontend/public/js/app/manuscript-detail/folio/ChantRecordView.js

@@ -244,7 +244,7 @@ export default Marionette.ItemView.extend({
 		var formattedVolpiano = parseVolpianoSyllables(text, volpiano);
 		this.model.set('volpiano', formattedVolpiano);
 		var cdb_uri = this.model.get('cdb_uri');
-		this.model.set({ 'cdb_link_url': 'https://cantus.uwaterloo.ca/node/' + cdb_uri });
+		this.model.set({ 'cdb_link_url': 'https://cantusdatabase.org/node/' + cdb_uri });


/node/ paths are deprecated. They'll redirect to their proper Chant/Source/Sequence/etc. Detail page if they were created in OldCantus, but /node/ will not work for any objects initially created in NewCantus. Can we use https://cantusdatabase.org/chant/ here?

Also, using a string literal would improve readability here.

Can we use https://cantusdatabase.org/chant/ here?

I think so....let me write through it to make sure. Every source not in the Bower database will have chants associated with them, rather than sequences. Therefore, ever chant associated with a non-Bower source will be accessible at /chant/<chant-id>. Since Cantus Ultimus only imports the sources that are returned by json_source_exposrt API, it will only import non-Bower sources. --> We can use /chant/ for any chant in Cantus Ultimus. Sound right?

cantusdatabase.org/json-sources does return sources from the Bower segment, so that premise doesn't work. If CU is going to have sources that are part of the Bower database, we may need to attach a flag to each source indicating whether it contains Chants or Sequences.

@annamorphism, do we expect sources from the Bower segment to end up in CU? Do any sources from Bower currently exist in CU?

cantusdatabase.org/json-sources does return sources from the Bower segment, so that premise doesn't work

The weird thing is it doesn't look like CU does have sources from the Bower segment, and we import any source returned by json-sources that doesn't give us a 403 Forbidden response. We end up with 207 sources, which is exactly the number that are public in the non-Bower segment on CantusDB. (https://cantusdatabase.org/sources/?segment=4063). Maybe there is somewhere where we are filtering out the Bower sources unbeknownst to me...

Also, some sources are both in the Bower segment and not in the Bower segment?

I don't think this is possible. Can you provide an example?

https://cantusdatabase.org/source/123779 and https://cantusdatabase.org/source/666647?

These look the "same" to me. 123779 is Bower and 666647 is not.

I probably should have clarified that by "source" here I meant an actual manuscript source and not a source object. I have no example of a source object (with a unique ID) somehow being both.

That is odd! Can confirm that e.g. source 123931 (Graz 1584) is in the json-export and not the CU mansucript list....
As to Jacob's question: it will be a long while before anything Bower-related is CU-ready, though it would certainly be an interesting project. (For starters, somebody would have to go hunt down which sources have image links...and clean up the "Notes" and the notation stuff...) There is one source that exists (separately catalogued) in both segments (123779 and 666647 are the same manuscript, Einsiedeln 121), so you could put it in (in theory anyway) and see what that would entail.
Anyway as a project I would say this is several years out, so while it would be good to be able to add Bower sources, it can be left as "somebody can add a flag for this later" and not too much deeper than that, I think.

looks like @dchiller cross-posted with the same example I was digging up. The two catalogs (which is basically what /source is for CantusDB) are in different places, and look different--the Bower one is only interested in the sequences and not the mass chants, and has different stuff associated: https://cantusdatabase.org/sequence/620743 and https://cantusdatabase.org/chant/671413 both describe the same chant but one has a "feast" and the other has a "rubric".

Hm... I was under the impression that the sources in CU were hand-picked, based on how complete the texts/volpiano are, whether there are images, etc. But I gather now that this isn't the case....

Yeah, I'll clarify. There are a dozen or so sources currently "public" on Cantus Ultimus. You can see the list here: https://cantus.simssa.ca/manuscripts/. These are a small subset of source on CantusDB that have IIIF-conforming image sources and that met some other somewhat ill-defined criteria (eg. did it have "good" volpiano coverage, as you mention) that Néstor and I developed last summer to narrow down a "first go" list of manuscripts.

Eventually, we want to serve as many manuscripts as we can from CantusDB. And, as in #783, when we find new IIIF sources for manuscripts in CantusDB, we want to start using CU as the viewer, as we just did with GB-Ob Can Lit 202. So we "import" a whole bunch of sources from Cantus Ultimus which are then not "public" until we have the IIIF manifest, we spend the time mapping them, etc.

So in this conversation when I say sources "on" Cantus Ultimus, I mean ones that have been imported (ie. that are in the latter much larger category).

To sum up, @jacobdgm @annamorphism

There are no imminent plans to serve Bower sources on CU, so we can go ahead and assume all chants on CU will be "chants" in CantusDB and not "sequences"...therefore we can make the offending link that started this discussion into /chant.

In the course of this discussion, we've learned that Cantus Ultimus currently imports only non-Bower sources. This is fine (per 1), but I don't know why this is when it is trying to import all sources returned in the json-sources API. As a separate item, I will investigate this to make sure that behaviour is well-documented and not a cause or effect of unintentional side effects.

dchiller · 2023-09-18T18:05:02Z

@jacobdgm @annamorphism Some details from my investigations:

cantusdatabase.org/json-sources does return sources from the Bower segment, so that premise doesn't work

This is different from OldCantus, where json-sources did not return sources from the Bower segment. (None of the Bower source id's I've searched for have turned up in response from cantus.uwaterloo.ca/json-sources). So this would explain why Bower sources are not currently imported into Cantus Ultimus.

And it turns out that with the changes in this PR, we would start importing Bower manuscripts because of that difference. I had tested that manuscripts were being correctly imported, but I guess hadn't looked that the same number of manuscripts had been imported.

Was that change in behaviour of json-sources intentional? If so, I can update our import process to include a screen on Bower sources. If not, we could make a CantusDB issue and not worry about it here.

jacobdgm · 2023-09-19T17:42:04Z

Was that change in behaviour of json-sources intentional?

Not to my knowledge, no.

If so, I can update our import process to include a screen on Bower sources. If not, we could make a CantusDB issue and not worry about it here.

I'll open this issue. Good that we caught this!

dchiller · 2023-09-19T19:04:34Z

I'll open this issue. Good that we caught this!

Ok... I could totally have seen the current behaviour being correct...there's nothing about the name json-sources that suggests it should only return CANTUS segment sources. But now new Cantus will be just like Old Cantus so that's good. Thanks for this!

dchiller added 2 commits August 10, 2023 11:53

Replace cantus.uwaterloo urls with cantusdatabase.org

86ba34c

Update link to NewCantus genres page

203c56b

dchiller requested review from jacobdgm and lucasmarchd01 August 10, 2023 17:57

jacobdgm previously approved these changes Aug 10, 2023

View reviewed changes

Merge branch 'main' into i-761-change-links-to-NewCantus

583e6be

dchiller mentioned this pull request Sep 13, 2023

json_node_export API is different in NewCantus DDMAL/CantusDB#1031

Closed

Update source importer and genre scraper for NewCantus

22d4205

dchiller requested a review from jacobdgm September 14, 2023 18:19

jacobdgm reviewed Sep 14, 2023

View reviewed changes

app/public/cantusdata/helpers/source_importer.py Outdated Show resolved Hide resolved

jacobdgm requested changes Sep 14, 2023

View reviewed changes

dchiller added 2 commits September 18, 2023 12:23

Remove print statement from SourceImporter

0d9089d

Update /node/ --> /chant/

3b4d234

dchiller requested a review from jacobdgm September 18, 2023 16:25

jacobdgm mentioned this pull request Sep 19, 2023

json-sources API should only return sources in the Cantus segment. DDMAL/CantusDB#1054

Closed

jacobdgm approved these changes Sep 19, 2023

View reviewed changes

dchiller merged commit d836bf0 into DDMAL:main Sep 19, 2023
2 checks passed

dchiller deleted the i-761-change-links-to-NewCantus branch September 19, 2023 19:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update links to CantusDB (cantus.waterloo.ca --> cantusdatabase.org) #765

Update links to CantusDB (cantus.waterloo.ca --> cantusdatabase.org) #765

dchiller commented Aug 10, 2023 •

edited

Loading

dchiller commented Aug 10, 2023

jacobdgm left a comment

jacobdgm Sep 14, 2023

jacobdgm Sep 14, 2023

dchiller Sep 15, 2023

jacobdgm Sep 18, 2023

dchiller Sep 18, 2023

dchiller Sep 18, 2023

annamorphism Sep 18, 2023

annamorphism Sep 18, 2023

dchiller Sep 18, 2023

dchiller Sep 18, 2023 •

edited

Loading

dchiller commented Sep 18, 2023

jacobdgm commented Sep 19, 2023

dchiller commented Sep 19, 2023

Update links to CantusDB (cantus.waterloo.ca --> cantusdatabase.org) #765

Update links to CantusDB (cantus.waterloo.ca --> cantusdatabase.org) #765

Conversation

dchiller commented Aug 10, 2023 • edited Loading

dchiller commented Aug 10, 2023

jacobdgm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dchiller Sep 18, 2023 • edited Loading

Choose a reason for hiding this comment

dchiller commented Sep 18, 2023

jacobdgm commented Sep 19, 2023

dchiller commented Sep 19, 2023

dchiller commented Aug 10, 2023 •

edited

Loading

dchiller Sep 18, 2023 •

edited

Loading