[Feature]: Add DANSK corpus

[alanakbik]

Problem statement

The DANSK NER corpus should be added to Flair: https://huggingface.co/datasets/chcaa/dansk-ner

Solution

A NER_DANISH_DANSK corpus object

Additional Context

No response

[stefan-it]

Hi @alanakbik ,

I played a bit with converting it into IOB2 - here's my current draft so far:

from datasets import load_dataset

ds_name = "chcaa/dansk-ner"
ds_split = "train"

ds = load_dataset(ds_name)[ds_split]

f_out = open(f"{ds_split}.tsv", "wt")

for example in ds:
    if len(example["spans"]["incorrect_spans"]) > 0:
        print("Found potential bad example:", example)
        continue

    for sent in example["sents"]:
        for token in example["tokens"]:
            token_start_pos = token["start"]
            token_end_pos = token["end"]
            token_text = example["text"][token_start_pos:token_end_pos]

            # Avoid empty tokens
            if not token_text.strip():
                continue
            
            entity = "O"
            
            for ent in example["ents"]:
                if token_start_pos >= ent["start"] and token_end_pos <= ent["end"]:
                    entity = ent["label"]
    
                    if token_start_pos == ent["start"]:
                        entity = f"B-{entity}"
                    else:
                        entity = f"I-{entity}"
                    break

            f_out.write(f"{token_text}\t{entity}\n")
        f_out.write(f"\n")
f_out.close()

Output looks ok:

Nyheder	O
Keen	B-ORGANIZATION
Revel	B-PRODUCT
III	I-PRODUCT
1159,00	B-MONEY
Kr	I-MONEY
Disponible	O
størrelser	O
Nyheder	O
Keen	B-ORGANIZATION
Revel	B-PRODUCT
III	I-PRODUCT
1159,00	B-MONEY
Kr	I-MONEY
-20	B-PERCENT
%	I-PERCENT

Konkurrence	O
badedragter	O
til	O
kvinder	O

juli	B-DATE
18	I-DATE
Skrevet	O
af	O
:	O
Morten	B-PERSON
Mønnike-Hald	I-PERSON
Lørdag	B-DATE
/	O
søndag	B-DATE
i	O
sidste	B-DATE
uge	I-DATE
,	O
løb	O
årets	O
udgave	O
af	O
B52	B-ORGANIZATION
/	I-ORGANIZATION
AFC	I-ORGANIZATION
REMA	B-EVENT
1000	I-EVENT
Cup	I-EVENT
af	O
stablen	O
.	O

[alanakbik]

Closed by #3639