Create test files for additional annotation metadata

[ukemi]

Now that the annotations are passing the QC checks (#63), we should start thinking about the additional metadata that MGI would like to import with their annotations. There are six types of metadata that we would like to have imported, probably as comments on the GO-CAM annotations.

• contributor
• Created by date
• modified by date
• created by person/group
• modified by person/group
• any has_participant relations that MGI has as current annotation extensions
• curator notes
• dual taxon id

What is the best way to proceed from here? I could make a test file for @dustine32 with just a few annotations. Then I can put a work order in for @loricorbani to generate a GPAD with this information from our database. Of course some of this will also be dependent on how we decide to handle annotation history.

[goodb]

It would be useful to make some example go-cam models that have the annotations like you want them. e.g. are all of these model level annotations or should some of them be added as annotations to node or relations within the models? We should also try to make use of controlled terminology for annotation properties (e.g. PROV, skos, or our own little ontology) - we don't want everything stuffed into comments if we want to be able to query it later.

[ukemi]

@goodb these are all excellent questions that we might want to discuss at one of the Wednesday meetings. In fact, one of the MGI curators was just asking me last Friday where he should be putting the comments in Noctua that he normally attaches to an annotation in the MGI editorial interface. As far as I know, we never made a decision on this. All of these notes are annotation-level notes, but the curator wanted to know if it was ok to just attach them to the main node or if he had to copy them to every piece of evidence etc. I will tag some of his current models because he has already been adding them. We can look at those on a call and decide if we think he is doing it correctly. They are definitely not at the model level. They are usually notes to himself as to why he made the annotation. ping @vanaukenk
For the has_participant annotation extensions that we are dropping, I would certainly want to be able to query them in the future. I'm not so sure about the curators notes. They are usually used when reviewing annotations.

[ukemi]

@loricorbani and I have made the first addition to the MGI gpad file available here:
http://www.informatics.jax.org/downloads/custom/mgi.gpa
This will be the home for our one-time-load TEST file for imports.

This GPAD1.0 file contains 'contributor' information when an MGI curator has either created or modified and annotation in MGI. This information is not usually edited by a curator, but is displayed on the Noctua home page in the list of models and shown as un-editable contributor annotations in the model itself.
Ping @dustine32 @kltm @goodb @vanaukenk
Once we have tested this addition, @loricorbani and I will continue to add additional metadata properties to our GPAD file. Next we will add created_by and modified_by data, which are also not usually editable by curators.

[dustine32]

@ukemi Fantastic! @dougli1sqrd has done a lot of work setting up separate branches of pipeline and the go-site metadata for the import load. Now, with this import GPAD URL, we can start iterating on handling the additional properties and history. And the results will be much easier to manage!

[dustine32]

@ukemi @loricorbani Would it be difficult to gzip the file at this location?
http://www.informatics.jax.org/downloads/custom/mgi.gpa

Similar to the regular annotation files deposited to the GO pipeline. Two reasons: faster download (obviously) and also I think the GO pipeline will croak if it's not gzipped. Thanks!

[kltm]

@dougli1sqrd Will it still croak? I thought it didn't croak anymore.

[dustine32]

Sorry @kltm I think I was referring to when it happened before but didn't realize it was fixed. So, only the one reason for gzipping it (bandwidth) left. Thanks!

[dougli1sqrd]

This was an issue in past times. The downloading logic is more robust now and in general will unzip if zipped, and keep uncompressed if found that way.

(As an aside, this is functionality that also extends to the downloader which we should probably be utilizing, but we are not, as far as the Jenkinsfile goes)

But ontobio handles both at the moment.

[dustine32]

Thanks @dougli1sqrd for confirming! No change necessary then @ukemi @loricorbani

[ukemi]

@dustine32 , we are going to try to update this file on Fridays. The update file will contain not only additional properties/metadata as we move forward, but the progress that I am making cleaning up the Shex errors in #79. Is there any way that you can give me a status report once in a while as a sanity check that I am making progress?

[leemdi]

I gzipped the file:
http://www.informatics.jax.org/downloads/custom/mgi.gpa.gz
http://www.informatics.jax.org/downloads/custom/mgi.gpa

[leemdi]

new file 03/27/2020:
http://www.informatics.jax.org/downloads/custom/mgi.gpa.gz
http://www.informatics.jax.org/downloads/custom/mgi.gpa

[dustine32]

Thanks @loricorbani ! Finally got to test the contributor parsing and it works great!

Example line from 03/27/2020 import GPAD:

MGI     MGI:1920971     part_of GO:0048471      MGI:MGI:3794006|PMID:18163442   ECO:0000314                     20081211        MGI     part_of(EMAPA:17972),part_of(CL:0000018)        contributor=http://orcid.org/0000-0003-3394-9805

Translated assertion in this model:

Evidence properties show contributor from line:

The contributor ORCID even links out to Dmitry:

So contributor should be good now.

[ukemi]

Awesome! I am having trouble getting to your personal server. Do they show up on the menu page too? The next step is for us to add the former MGI curators. @hdrabkin is making sure they are in users.yaml, then I will forward them to @loricorbani and we create a new file. One step at a time....