Pipeline currently mis-maps IEAs around GOlr, likely related to owltools or its environment

[kltm]

There are 128276 more annotations on release than were on the candidate. Looking at AmiGO, it seems like "evidence used in automatic assertion" (ECO:0000501) has entirely disappeared. Strangely, a very small numberof IEA annotations did manage to find their way in:

https://amigo-staging.geneontology.io/amigo/search/annotation?q=IEA*&fq=-evidence_subset_closure_label:%22genetic%20interaction%20evidence%20used%20in%20manual%20assertion%22&fq=-evidence_subset_closure_label:%22biological%20aspect%20of%20ancestor%20evidence%20used%20in%20manual%20assertion%22&sfq=document_category:%22annotation%22

That's weird.
I'm putting a pin in that for now.

That leaves us with two paths for data to be dropped, assuming that we are at issue and not upstream: ontobio and owltools.

Checking around for a data set let's arbitrarily select the MGI GAF as an example, specifically the IEAs:

snapshot | release
src: 73326 | 73829
valid: 73318 | 73821

So, that is a small change in the src/valid numbers. In AmiGO, filtering for just species "mouse" annotations, there are actually /more/ in the snapshot...so hm.

My guess for the moment is that there is something going on in either owltools or the docker mechanism around it (possibly cached old files) that is causing a problem here. I'll do a bit more digging to try and get at what's going on.

[kltm]

GOlr build step runs with>

java \
    -Xms$LOADER_MEM \
    -Xmx$LOADER_MEM \
    -DentityExpansionLimit=8172000 \
    -Djava.awt.headless=true \
    -jar /srv/amigo/java/lib/owltools-runner-all.jar  \
    $ONTOLOGIES \
    --log-info \
    --solr-config /srv/amigo/metadata/ont-config.yaml \
    --merge-support-ontologies \
    --merge-imports-closure \
    --remove-subset-entities upperlevel \
    --remove-disjoints \
    --silence-elk \
    --reasoner elk \
    --solr-taxon-subset-name amigo_grouping_subset \
    --solr-eco-subset-name go_groupings \
    --solr-url http://localhost:8080/solr/ \
    --solr-log /tmp/golr_timestamp.log \
    --solr-load-ontology \
    --solr-load-ontology-general  \
    --read-panther /tmp/panther_data/ \
    --solr-load-gafs $GAFS \
    --solr-load-panther \
    --solr-load-panther-general \
    --solr-optimize

The only explicit ECO reference is --solr-eco-subset-name go_groupings.

[kltm]

Noting:
biolink/ontobio#589
evidenceontology/evidenceontology@2399858#diff-8026d95055ee94ddd11aee9f5a7b6509f2153da1defe2b9c5dad33e597beb78d

IEA ECO:0000501 switched to ECO:0007669

[kltm]

The go_groupings subset seems to have not changed with the above diff.

That said, looking at the older eco-basic.obo
https://github.com/evidenceontology/evidenceontology/blob/b6fa5152b9063bcf1ae01c573e35302578156f9f/eco-basic.obo
and newer
https://github.com/evidenceontology/evidenceontology/blob/master/eco-basic.obo
versions of ECO, it seems that what changed here it that "IEA" as a synonym has been changed. Oddly (to me at least), other structures are untouched.

Starting to dive into owltools a little...
https://github.com/owlcollab/owltools/blob/9faa4f42b761839a26e8c8096cd24044e2bdcfc7/OWLTools-Annotation/src/main/java/owltools/gaf/eco/EcoMapper.java#L24
makes use of the (I assume proper) http://purl.obolibrary.org/obo/eco/gaf-eco-mapping.txt
The code doing the mapping work appears to be:
https://github.com/owlcollab/owltools/blob/master/OWLTools-Annotation/src/main/java/owltools/gaf/EcoTools.java

TODO: May need to update http://wiki.geneontology.org/index.php/Inferred_from_Electronic_Annotation_(IEA)

[kltm]

Raw access to "IEA" annotations in production AmiGO:
http://amigo.geneontology.org/amigo/search/annotation?q=*:*&fq=evidence_subset_closure:%22ECO:0000501%22&sfq=document_category:%22annotation%22
No analog in amigo-staging:
https://amigo-staging.geneontology.io/amigo/search/annotation?q=*:*&fq=evidence_subset_closure:%22ECO:0000501%22&sfq=document_category:%22annotation%22
https://amigo-staging.geneontology.io/amigo/search/annotation?q=*:*&fq=evidence_subset_closure:%22ECO:0007669%22&sfq=document_category:%22annotation%22
This makes me wonder if GO:0007769 ("computational evidence used in automatic assertion") is not in our evidence subset and mechanism around that is causing our issues.
Oddly, this doesn't show up outside the closure either:
https://amigo-staging.geneontology.io/amigo/search/annotation?q=*:*&fq=evidence_closure:%22ECO:0007669%22&sfq=document_category:%22annotation%22

[kltm]

Getting a little more cute with AmiGO, we can see that the annotations are more or less there:
http://amigo.geneontology.org/amigo/search/annotation?q=*:*&fq=evidence_type:%22IEA%22&sfq=document_category:%22annotation%22
https://amigo-staging.geneontology.io/amigo/search/annotation?q=*:*&fq=evidence_type:%22IEA%22&sfq=document_category:%22annotation%22
But that the closures, at least, are not getting created properly for them. Looking directly at GOlr:

http://golr.geneontology.org/solr/select?defType=edismax&qt=standard&indent=on&wt=json&rows=10&start=0&fl=bioentity,bioentity_name,qualifier,annotation_class,annotation_extension_json,assigned_by,taxon,evidence_closure,evidenve_closure_label,evidence_subset_closure,evidence_subset_closure_label,evidence_type,evidence_with,panther_family,type,bioentity_isoform,reference,date,bioentity_label,annotation_class_label,taxon_label,panther_family_label,score&facet=true&facet.mincount=1&facet.sort=count&json.nl=arrarr&facet.limit=25&hl=true&hl.simple.pre=%3Cem%20class=%22hilite%22%3E&hl.snippets=1000&fq=evidence_type:%22IEA%22&q=*:*
http://golr-staging.geneontology.io/solr/select?defType=edismax&qt=standard&indent=on&wt=json&rows=10&start=0&fl=bioentity,bioentity_name,qualifier,annotation_class,annotation_extension_json,assigned_by,taxon,evidence_closure,evidenve_closure_label,evidence_subset_closure,evidence_subset_closure_label,evidence_type,evidence_with,panther_family,type,bioentity_isoform,reference,date,bioentity_label,annotation_class_label,taxon_label,panther_family_label,score&facet=true&facet.mincount=1&facet.sort=count&json.nl=arrarr&facet.limit=25&hl=true&hl.simple.pre=%3Cem%20class=%22hilite%22%3E&hl.snippets=1000&fq=evidence_type:%22IEA%22&q=*:*
they seem to have been created empty in the loader and are not appearing in the document.

[kltm]

Attempt at quick recreation of possible issue, after doing some small updates of owltools ECO tests (see owlcollab/owltools#321); unfortunately, testing cleanly with fixes:
mvn clean -Dtest="EcoMapperTest" -DfailIfNoTests=false test
mvn clean -Dtest="TraversingEcoMapperTest" -DfailIfNoTests=false test

I'm having a little trouble figuring out where to go next. Maybe need to dig in more around the GAF document loader?

[kltm]

Noting that the far upstream origin of this issue is likely evidenceontology/evidenceontology#251

[kltm]

I can now hit a problem spot with:
mvn clean -Dtest="EcoToolsTest" -DfailIfNoTests=false test
The test returns 0/nothing for IEA, instead of, basically, anything.
The issue seems to be with getClassesForGoCode in:

https://github.com/owlcollab/owltools/blob/d58e87f4be4b38fd1c1607b6777054768034fbdf/OWLTools-Annotation/src/main/java/owltools/gaf/EcoTools.java#L124

Appearing in the GOlr loader here:
https://github.com/owlcollab/owltools/blob/d58e87f4be4b38fd1c1607b6777054768034fbdf/OWLTools-Solr/src/main/java/owltools/solrj/GafSolrDocumentLoader.java#L629
https://github.com/owlcollab/owltools/blob/d58e87f4be4b38fd1c1607b6777054768034fbdf/OWLTools-Solr/src/main/java/owltools/solrj/GafSolrDocumentLoader.java#L658

[kltm]

@cmungall @balhoff, with the context right above here (#251 (comment)), I was wondering if you could easily spot what might be going wrong? I can start working through getClassesForGoCode, I could also switch to using getClassesForCode, which seems to be giving results that are more inline with what is expected (unsure why these choices were made). Any thoughts here?

[kltm]

@balhoff @cmungall To resummarize where I'm at:

• The GOlr loader uses the function getClassesForGoCode (https://github.com/owlcollab/owltools/blob/d58e87f4be4b38fd1c1607b6777054768034fbdf/OWLTools-Annotation/src/main/java/owltools/gaf/EcoTools.java#L124)
• I can simulate the likely problem with the loader in tests with this function on the owltools branch issue-pipeline-251-iea-golr-load with mvn clean -Dtest="EcoToolsTest" -DfailIfNoTests=false test, PRed here to see differences https://github.com/owlcollab/owltools/pull/321/files
• There is a very similar function getClassesForCode that I test (https://github.com/owlcollab/owltools/blob/d58e87f4be4b38fd1c1607b6777054768034fbdf/OWLTools-Annotation/src/test/java/owltools/gaf/eco/TraversingEcoMapperTest.java#L116) that does not seem to have this problem.

With that, my questions are, in order of utility towards fixing this:

• Why is getClassesForGoCode apparently broken with this ontology change?
• What is the difference between getClassesForGoCode and getClassesForCode
• Is getClassesForCode a safe replacement for getClassesForGoCode for our ECO use case?
  • If so, why does getClassesForGoCode exist at all?

[kltm]

Talking with @pgaudet @mgiglio99 et al. this morning, the current plan is to: have eco / mapping for IEA to revert to its previous state for the time being, get a release out, and continue to move forward on finding a fix with an eye for the next release out.

[kltm]

I believe we've cleared other issues blocking the pipeline and am restarting the release pipeline.

[kltm]

I'm assuming this is closed for now.