We provide sentential paraphrase detection train, test datasets as well as BERT-based models for the Armenian language.
To sentences in the dataset are taken from Hetq and Panarmenian news articles.
To generate paraphrase for the sentences, we used back translation from Armenian to English. We repeated the step twice, after which the generated paraphrases were manually reviewed. Invalid sentences were filtered out, while the rest were labelled as either paraphrase, near paraphrase or non-paraphrase (with 1, 0, -1 labels respectively). The sentences are divided into train and test sets. Test examples were reviewed by 3 different annotators. In addition, to increase the number of non-paraphrase pairs, we padded the dataset with automatically generated negative examples, including pairs of consecutive sentences and random pairs.
| Number of examples | Total | Paraphrase | Non-paraphrase (near paraphrase) |
|---|---|---|---|
| Train | 4233 | 1339 | 2683 (211) |
| Test | 1682 | 1021 | 448 (213) |
We finetuned Multilingual BERT on several training sets, including the proposed ARPA dataset, and evaluated their performance on our test set. During training and evaluation, near paraphrase and non-paraphrase pairs were combined into one class. The results are provided below:
| BERT Model | Train set | F1 | Acc. |
|---|---|---|---|
| Multilingual BERT | ARPA train set | 84.27 | 78.06 |
| Multilingual BERT | Paraphraser.ru train set machine-translated into Armenian | 83.81 | 77.09 |
| Multilingual BERT | MRPC train set machine-translated into Armenian | 80.07 | 69.87 |
| Multilingual BERT | All of the above combined | 84 | 77.6 |
The model trained on ARPA is available for use, and can be downloaded using this link.
For more details about the models and dataset construction, refer to the paper.