Added it-en, updated ru-en (#22)

* Add script to pull models from snakepit * Update ruen * Update ruen * Add iten * Update evaluation results [skip ci] Co-authored-by: CircleCI evaluation job <ci-models-evaluation@firefox-translations>
mozilla · Aug 11, 2021 · acfbae6 · acfbae6
1 parent a0af849
commit acfbae6
Show file tree

Hide file tree

Showing 26 changed files with 66 additions and 23 deletions.
diff --git a/evaluation/dev/img/avg.png b/evaluation/dev/img/avg.png
diff --git a/evaluation/dev/img/it-en.png b/evaluation/dev/img/it-en.png
diff --git a/evaluation/dev/img/ru-en.png b/evaluation/dev/img/ru-en.png
diff --git a/evaluation/dev/it-en/mtedx_test.bergamot.en.bleu b/evaluation/dev/it-en/mtedx_test.bergamot.en.bleu
@@ -0,0 +1 @@
+35.5
diff --git a/evaluation/dev/it-en/mtedx_test.google.en.bleu b/evaluation/dev/it-en/mtedx_test.google.en.bleu
@@ -0,0 +1 @@
+35.9
diff --git a/evaluation/dev/it-en/mtedx_test.microsoft.en.bleu b/evaluation/dev/it-en/mtedx_test.microsoft.en.bleu
@@ -0,0 +1 @@
+36.4
diff --git a/evaluation/dev/it-en/wmt09.bergamot.en.bleu b/evaluation/dev/it-en/wmt09.bergamot.en.bleu
@@ -0,0 +1 @@
+33.1
diff --git a/evaluation/dev/it-en/wmt09.google.en.bleu b/evaluation/dev/it-en/wmt09.google.en.bleu
@@ -0,0 +1 @@
+35.4
diff --git a/evaluation/dev/it-en/wmt09.microsoft.en.bleu b/evaluation/dev/it-en/wmt09.microsoft.en.bleu
@@ -0,0 +1 @@
+35.8
diff --git a/evaluation/dev/results.md b/evaluation/dev/results.md
@@ -57,21 +57,21 @@ Both absolute and relative differences in BLEU scores between Bergamot and other
 
 ## avg
 
-| Translator/Dataset | ru-en | en-ru | pt-en |
-| --- | --- | --- | --- |
-| bergamot | 32.29 | 28.51 | 39.70 |
-| google | 36.59 (+4.30, +13.32%) | 33.75 (+5.24, +18.37%) | 38.90 (-0.80, -2.02%) |
-| microsoft | 37.03 (+4.74, +14.69%) | 33.83 (+5.31, +18.63%) | 40.90 (+1.20, +3.02%) |
+| Translator/Dataset | ru-en | en-ru | it-en | pt-en |
+| --- | --- | --- | --- | --- |
+| bergamot | 33.74 | 28.51 | 34.30 | 39.70 |
+| google | 36.59 (+2.84, +8.43%) | 33.75 (+5.24, +18.37%) | 35.65 (+1.35, +3.94%) | 38.90 (-0.80, -2.02%) |
+| microsoft | 37.03 (+3.29, +9.75%) | 33.83 (+5.31, +18.63%) | 36.10 (+1.80, +5.25%) | 40.90 (+1.20, +3.02%) |
 
 ![Results](img/avg.png)
 
 ## ru-en
 
 | Translator/Dataset | mtedx_test | wmt19 | wmt17 | wmt14 | wmt15 | wmt16 | wmt13 | wmt18 | wmt20 |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| bergamot | 22.80 | 37.60 | 35.90 | 35.60 | 32.40 | 31.90 | 28.80 | 31.20 | 34.40 |
-| google | 25.00 (+2.20, +9.65%) | 42.40 (+4.80, +12.77%) | 41.50 (+5.60, +15.60%) | 41.20 (+5.60, +15.73%) | 37.50 (+5.10, +15.74%) | 36.60 (+4.70, +14.73%) | 31.40 (+2.60, +9.03%) | 36.00 (+4.80, +15.38%) | 37.70 (+3.30, +9.59%) |
-| microsoft | 26.10 (+3.30, +14.47%) | 42.60 (+5.00, +13.30%) | 41.60 (+5.70, +15.88%) | 41.70 (+6.10, +17.13%) | 37.80 (+5.40, +16.67%) | 37.60 (+5.70, +17.87%) | 31.20 (+2.40, +8.33%) | 36.90 (+5.70, +18.27%) | 37.80 (+3.40, +9.88%) |
+| bergamot | 24.20 | 39.30 | 37.90 | 38.00 | 33.70 | 33.40 | 29.50 | 32.30 | 35.40 |
+| google | 25.00 (+0.80, +3.31%) | 42.40 (+3.10, +7.89%) | 41.50 (+3.60, +9.50%) | 41.20 (+3.20, +8.42%) | 37.50 (+3.80, +11.28%) | 36.60 (+3.20, +9.58%) | 31.40 (+1.90, +6.44%) | 36.00 (+3.70, +11.46%) | 37.70 (+2.30, +6.50%) |
+| microsoft | 26.10 (+1.90, +7.85%) | 42.60 (+3.30, +8.40%) | 41.60 (+3.70, +9.76%) | 41.70 (+3.70, +9.74%) | 37.80 (+4.10, +12.17%) | 37.60 (+4.20, +12.57%) | 31.20 (+1.70, +5.76%) | 36.90 (+4.60, +14.24%) | 37.80 (+2.40, +6.78%) |
 
 ![Results](img/ru-en.png)
 
@@ -85,6 +85,16 @@ Both absolute and relative differences in BLEU scores between Bergamot and other
 
 ![Results](img/en-ru.png)
 
+## it-en
+
+| Translator/Dataset | mtedx_test | wmt09 |
+| --- | --- | --- |
+| bergamot | 35.50 | 33.10 |
+| google | 35.90 (+0.40, +1.13%) | 35.40 (+2.30, +6.95%) |
+| microsoft | 36.40 (+0.90, +2.54%) | 35.80 (+2.70, +8.16%) |
+
+![Results](img/it-en.png)
+
 ## pt-en
 
 | Translator/Dataset | mtedx_test |

diff --git a/evaluation/dev/ru-en/mtedx_test.bergamot.en.bleu b/evaluation/dev/ru-en/mtedx_test.bergamot.en.bleu
@@ -1 +1 @@
-22.8
+24.2
diff --git a/evaluation/dev/ru-en/wmt13.bergamot.en.bleu b/evaluation/dev/ru-en/wmt13.bergamot.en.bleu
@@ -1 +1 @@
-28.8
+29.5
diff --git a/evaluation/dev/ru-en/wmt14.bergamot.en.bleu b/evaluation/dev/ru-en/wmt14.bergamot.en.bleu
@@ -1 +1 @@
-35.6
+38.0
diff --git a/evaluation/dev/ru-en/wmt15.bergamot.en.bleu b/evaluation/dev/ru-en/wmt15.bergamot.en.bleu
@@ -1 +1 @@
-32.4
+33.7
diff --git a/evaluation/dev/ru-en/wmt16.bergamot.en.bleu b/evaluation/dev/ru-en/wmt16.bergamot.en.bleu
@@ -1 +1 @@
-31.9
+33.4
diff --git a/evaluation/dev/ru-en/wmt17.bergamot.en.bleu b/evaluation/dev/ru-en/wmt17.bergamot.en.bleu
@@ -1 +1 @@
-35.9
+37.9
diff --git a/evaluation/dev/ru-en/wmt18.bergamot.en.bleu b/evaluation/dev/ru-en/wmt18.bergamot.en.bleu
@@ -1 +1 @@
-31.2
+32.3
diff --git a/evaluation/dev/ru-en/wmt19.bergamot.en.bleu b/evaluation/dev/ru-en/wmt19.bergamot.en.bleu
@@ -1 +1 @@
-37.6
+39.3
diff --git a/evaluation/dev/ru-en/wmt20.bergamot.en.bleu b/evaluation/dev/ru-en/wmt20.bergamot.en.bleu
@@ -1 +1 @@
-34.4
+35.4
diff --git a/models/dev/iten/lex.50.50.iten.s2t.bin.gz b/models/dev/iten/lex.50.50.iten.s2t.bin.gz
diff --git a/models/dev/iten/model.iten.intgemm.alphas.bin.gz b/models/dev/iten/model.iten.intgemm.alphas.bin.gz
diff --git a/models/dev/iten/vocab.iten.spm.gz b/models/dev/iten/vocab.iten.spm.gz
diff --git a/models/dev/ruen/lex.50.50.ruen.s2t.bin.gz b/models/dev/ruen/lex.50.50.ruen.s2t.bin.gz
diff --git a/models/dev/ruen/model.ruen.intgemm.alphas.bin.gz b/models/dev/ruen/model.ruen.intgemm.alphas.bin.gz
diff --git a/models/dev/ruen/vocab.ruen.spm.gz b/models/dev/ruen/vocab.ruen.spm.gz
diff --git a/scripts/pull_models.sh b/scripts/pull_models.sh
@@ -0,0 +1,18 @@
+#!/bin/bash
+
+# Downloads models from snakepit cluster
+
+SRC=ru
+TRG=en
+EXPERIMENT=all_mtdata_and_opus2
+ROOT_DIR="bergamot-training/models"
+
+SNAKEPIT_DIR="${ROOT_DIR}/${SRC}-${TRG}/${EXPERIMENT}/exported"
+LOCAL_DIR="../models/dev/${SRC}${TRG}"
+
+rm -rf "${LOCAL_DIR}"
+mkdir "${LOCAL_DIR}"
+
+pit pull home "${SNAKEPIT_DIR}/lex.50.50.${SRC}${TRG}.s2t.bin.gz" "${LOCAL_DIR}"
+pit pull home "${SNAKEPIT_DIR}/model.${SRC}${TRG}.intgemm.alphas.bin.gz" "${LOCAL_DIR}"
+pit pull home "${SNAKEPIT_DIR}/vocab.${SRC}${TRG}.spm.gz" "${LOCAL_DIR}"