CoNLL-U to displaCy treebank conversion

[GruffPrys]

CoNLL-U is the format used by the Universal Dependencies initiative to annotate dependency treebanks (http://universaldependencies.org/format.html). A large number of treebanks for many different languages are available in the CoNLL-U format. However, currently displaCy cannot parse the CoNLL format to visualize those trees, and it would be nice if it could.

The following code converts a single sentence from the CoNLL format to the json format used by displaCy:

import json
from collections import OrderedDict

conll_u_string = """1	As	as	SCONJ	IN	_	4	mark	_	_
2	the	the	DET	DT	Definite=Def|PronType=Art	3	det	_	_
3	year	year	NOUN	NN	Number=Sing	4	nsubj	_	_
4	progresses	progress	VERB	VBZ	Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin	13	advcl	_	SpaceAfter=No
5	,	,	PUNCT	,	_	13	punct	_	_
6	the	the	DET	DT	Definite=Def|PronType=Art	13	mark	_	_
7	more	more	ADV	RBR	_	13	advmod	_	_
8	impressive	impressive	ADJ	JJ	Degree=Pos	13	xcomp	_	_
9	your	you	PRON	PRP$	Person=2|Poss=Yes|PronType=Prs	10	nmod:poss	_	_
10	ideas	idea	NOUN	NNS	Number=Plur	13	nsubj	_	_
11	and	and	CCONJ	CC	_	12	cc	_	_
12	works	work	NOUN	NNS	Number=Plur	10	conj	_	_
13	become	become	VERB	VBP	Mood=Ind|Tense=Pres|VerbForm=Fin	0	root	_	SpaceAfter=No
14	.	.	PUNCT	.	_	8	punct	_	_"""


def set_arrow_direction(word_line):
    """
    Sets the orientation of the arrow that notes the directon of the dependency
    between the two units.

    """
    if int(word_line["id"]) > int(word_line["head"]):
        word_line["dir"] = "right"
    elif int(word_line["id"]) < int(word_line["head"]):
        word_line["dir"] = "left"
    return word_line

def convert2zero_based_numbering(word_line_field):
    "CONLL-U numbering starts at 1, displaCy's at 0..."
    word_line_field = str(int(word_line_field) - 1)
    return word_line_field

def get_start_and_end(word_line):
    """
    Displacy's 'start' value is the lowest value amongst the ID and HEAD values,
    and the 'end' is always the highest. 'Start' and 'End' have nothing to do
    with dependency which is indicated by the arrow direction, not the line
    direction.
    """
    word_line["start"] = min([int(word_line["id"]), int(word_line["head"])])
    word_line["end"] = max([int(word_line["id"]), int(word_line["head"])])
    return word_line
                           
def conll_u_string2displacy_json(conll_u_sent_string): 
    """
    Converts a single CONLL-U formatted sentence to the displaCy json format.
    CONLL-U specification: http://universaldependencies.org/format.html
    """
    conll_u_lines = [line for line in conll_u_sent_string.split("\n") \
                     if line[0].isnumeric()]

    displacy_json = {"arcs": [], "words": []}
    for tabbed_line in conll_u_lines:

        word_line = OrderedDict()
        word_line["id"], word_line["form"], word_line["lemma"], \
        word_line["upostag"], word_line["xpostag"], word_line["feats"], \
        word_line["head"], word_line["deprel"], word_line["deps"], \
        word_line["misc"] = tabbed_line.split("\t")

        word_line["id"] = convert2zero_based_numbering(word_line["id"])
        word_line["head"] = convert2zero_based_numbering(word_line["head"])       
        
        if word_line["deprel"] != "root":
            word_line = get_start_and_end(word_line)
            word_line = set_arrow_direction(word_line)
            displacy_json["arcs"].append({"dir": word_line["dir"],
                                          "end": word_line["end"],
                                          "label": word_line["deprel"],
                                          "start": word_line["start"]})
            
        displacy_json["words"].append({"tag": word_line["upostag"],
                                       "text": word_line["form"]})

    displacy_json = (json.dumps(displacy_json, indent=4))
    return displacy_json

displacy_json = conll_u_string2displacy_json(conll_u_string)
print (displacy_json)

Example output:

``` { "arcs": [ { "label": "mark", "start": 0, "dir": "left", "end": 3 }, { "label": "det", "start": 1, "dir": "left", "end": 2 }, { "label": "nsubj", "start": 2, "dir": "left", "end": 3 }, { "label": "advcl", "start": 3, "dir": "left", "end": 12 }, { "label": "punct", "start": 4, "dir": "left", "end": 12 }, { "label": "mark", "start": 5, "dir": "left", "end": 12 }, { "label": "advmod", "start": 6, "dir": "left", "end": 12 }, { "label": "xcomp", "start": 7, "dir": "left", "end": 12 }, { "label": "nmod:poss", "start": 8, "dir": "left", "end": 9 }, { "label": "nsubj", "start": 9, "dir": "left", "end": 12 }, { "label": "cc", "start": 10, "dir": "left", "end": 11 }, { "label": "conj", "start": 9, "dir": "right", "end": 11 }, { "label": "punct", "start": 7, "dir": "right", "end": 13 } ], "words": [ { "tag": "SCONJ", "text": "As" }, { "tag": "DET", "text": "the" }, { "tag": "NOUN", "text": "year" }, { "tag": "VERB", "text": "progresses" }, { "tag": "PUNCT", "text": "," }, { "tag": "DET", "text": "the" }, { "tag": "ADV", "text": "more" }, { "tag": "ADJ", "text": "impressive" }, { "tag": "PRON", "text": "your" }, { "tag": "NOUN", "text": "ideas" }, { "tag": "CCONJ", "text": "and" }, { "tag": "NOUN", "text": "works" }, { "tag": "VERB", "text": "become" }, { "tag": "PUNCT", "text": "." } ] } ```

Apologies that this isn't submitted as a Pull Request - I've not been able to get Spacy 2.0 Alpha up and running on my Windows machines. I've tested the output in displaCy using Jupyter and compared the resulting visualizations with the brat visualizations here:
http://bionlp-www.utu.fi/dep_search/

For example:

So far, they seem to match:

However, further testing is advisable.

CoNLL-U for the above examples:

1	Thursday	Thursday	PROPN	NNP	Number=Sing	2	nsubj	_	_
2	works	work	VERB	VBZ	Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin	0	root	_	_
3	for	for	ADP	IN	_	4	case	_	_
4	me	I	PRON	PRP	Case=Acc|Number=Sing|Person=1|PronType=Prs	2	obl	_	SpaceAfter=No
5	.	.	PUNCT	.	_	2	punct	_	_

[GruffPrys]

Things to consider:

• CoNLL-U files are treebank files and often contain significantly more than the single sentence that displaCy accepts. Is this an issue? Should displaCy have a means of displaying a sequence of many hundreds of trees?
• For my needs (discussing theoretical annotation issues), brat's ability to cope with a partially annotated tree is useful as I may only want to annotate a single dependency when drawing attention to a particular issue. This may not be a priority for spaCy of course.

[ines]

Ah cool, thank you so much for posting your code here! 💯 This actually looks much more compact than I had imagined – so I suppose we could integrate this with v2 pretty easily.

CoNLL-U files are treebank files and often contain significantly more than the single sentence that displaCy accepts. Is this an issue? Should displaCy have a means of displaying a sequence of many hundreds of trees?

Theoretically, displaCy can render whole documents with hundreds of sentences in one go (anything that's in a Doc). However, for long documents, this can easily get messy, as it'll all be rendered as part of one, scrollable graphic. If you use displacy.serve or displacy.render with Page=True, you can pass in a variable number of docs and the output will be a complete HTML page, with one displaCy visualisation per doc. The same works for dictionaries describing the words and arcs. So the output doesn't have to be a standalone graphic, this approach should work, too.

docs1 = [Doc, Doc, Doc, Doc]              # Doc objects
docs2 = [dict(), dict(), dict(), dict()]  # dicts describing words/arcs

# start the web server and serve the HTML
displacy.serve(*docs1) 
displacy.serve(*docs2, manual=True)

# get the markup to export to a file etc.
html = displacy.render(*docs1, page=True)
html = displacy.render(*docs2, page=True, manual=True) 

For my needs (discussing theoretical annotation issues), brat's ability to cope with a partially annotated tree is useful as I may only want to annotate a single dependency when drawing attention to a particular issue. This may not be a priority for spaCy of course.

This should be no problem for displaCy – at least if you use the "manual" mode with words/arcs dict, or pass in a modified Doc with only partial annotations. displaCy will render whichever words and arcs you tell it to. (Even complete nonsense – which was super helpful btw when we were training the models for v2 alpha.)

The JS version of displaCy had a nice feature for assigning custom styles to arcs (e.g. "style": "correct" for green etc.). Maybe I should bring this back in the built-in version. This would also let you visualise two different annotation options in one tree, similar to this example from one of our old blog posts:

But I'm not sure if this would make sense for your use case. Anyway, thanks again for the great work – I'll start integrating it on the v2 branch and keep you updated!

[GruffPrys]

Ah cool, thank you so much for posting your code here!

No problem - least I could do considering you're sharing such a great piece of software. :)

Thanks for addressing the other considerations - sounds good! I'll have another go at getting the nightly to work, even if it means running it in Docker.

I should probably submit an issue about the Visual Studio compiler issues I've been having when installing as I saw someone else posting a similar error message recently.

[GruffPrys]

BTW this would be very useful for my purposes:

feature for assigning custom styles to arcs (e.g. "style": "correct" for green etc.). Maybe I should bring this back in the built-in version.

You should definitely bring that back :)

[GruffPrys]

One other thing (sorry!) - there's a flaw in my code. It should convert the CoNLL-U null value _ to whatever equivalent displaCy uses.

Come to think of it, that's probably why my attempts to render partial annotations didn't work.

[GruffPrys]

Small tweak of the main function to allow for incomplete annotations to be rendered by displaCy:

def conll_u_string2displacy_json(conll_u_sent_string): 
    """
    Converts a single CONLL-U formatted sentence to the displaCy json format.
    CONLL-U specification: http://universaldependencies.org/format.html
    """
    conll_u_lines = [line for line in conll_u_sent_string.split("\n") \
                     if line[0].isnumeric()]

    displacy_json = {"arcs": [], "words": []}
    for tabbed_line in conll_u_lines:

        word_line = OrderedDict()
        word_line["id"], word_line["form"], word_line["lemma"], \
        word_line["upostag"], word_line["xpostag"], word_line["feats"], \
        word_line["head"], word_line["deprel"], word_line["deps"], \
        word_line["misc"] = tabbed_line.split("\t")

        word_line["id"] = convert2zero_based_numbering(word_line["id"])
        if word_line["head"] != "_":
            word_line["head"] = convert2zero_based_numbering(word_line["head"])       
        
        if word_line["deprel"] != "root" and word_line["head"] != "_":
            word_line = get_start_and_end(word_line)
            word_line = set_arrow_direction(word_line)
            displacy_json["arcs"].append({"dir": word_line["dir"],
                                          "end": word_line["end"],
                                          "label": word_line["deprel"],
                                          "start": word_line["start"]})
            
        displacy_json["words"].append({"tag": word_line["upostag"],
                                       "text": word_line["form"]})

    displacy_json = (json.dumps(displacy_json, indent=4))
    return displacy_json

It works by not creating or adding arcs where the HEAD is "_" .