Add enpt (#23)

eu9ene · CircleCI evaluation job · web-flow · commit 5ff404f0af67 · 2021-08-26T12:02:52.000-07:00
* Add enpt

* Update readme

* Update evaluation results [skip ci]

Co-authored-by: CircleCI evaluation job &lt;ci-models-evaluation@firefox-translations&gt;
diff --git a/README.md b/README.md
@@ -25,7 +25,7 @@ Use [Firefox Translations training pipeline](https://github.com/mozilla/firefox-
 
 ## Training data
 
-Do not use [SacreBLEU](https://github.com/mjpost/sacrebleu) datasets as a part of training data, otherwise evaluation will not be correct.
+Do not use [SacreBLEU](https://github.com/mjpost/sacrebleu) or [Flores](https://github.com/facebookresearch/flores) datasets as a part of training data, otherwise evaluation will not be correct.
 
 To see SacreBLEU datasets run `sacrebleu --list`.
 
@@ -57,11 +57,10 @@ If you want to run it with `bergamot` only, remove mentions of those variables f
 - Czech <-> English
 ## Dev
 -  Russian <-> English
--  Portuguese -> English
+-  Portuguese <-> English
 -  Italian -> English
 ## Upcoming
 -  German -> English
 -  French <-> English
 -  Polish <-> English
--  Portuguese <- English
 -  Italian <- English
diff --git a/evaluation/dev/en-pt/flores-dev.bergamot.pt.bleu b/evaluation/dev/en-pt/flores-dev.bergamot.pt.bleu
@@ -0,0 +1 @@
+46.9
diff --git a/evaluation/dev/en-pt/flores-dev.google.pt.bleu b/evaluation/dev/en-pt/flores-dev.google.pt.bleu
@@ -0,0 +1 @@
+56.4
diff --git a/evaluation/dev/en-pt/flores-dev.microsoft.pt.bleu b/evaluation/dev/en-pt/flores-dev.microsoft.pt.bleu
@@ -0,0 +1 @@
+49.8
diff --git a/evaluation/dev/en-pt/flores-test.bergamot.pt.bleu b/evaluation/dev/en-pt/flores-test.bergamot.pt.bleu
@@ -0,0 +1 @@
+46.2
diff --git a/evaluation/dev/en-pt/flores-test.google.pt.bleu b/evaluation/dev/en-pt/flores-test.google.pt.bleu
@@ -0,0 +1 @@
+55.7
diff --git a/evaluation/dev/en-pt/flores-test.microsoft.pt.bleu b/evaluation/dev/en-pt/flores-test.microsoft.pt.bleu
@@ -0,0 +1 @@
+50.7
diff --git a/evaluation/dev/en-ru/flores-dev.bergamot.ru.bleu b/evaluation/dev/en-ru/flores-dev.bergamot.ru.bleu
@@ -0,0 +1 @@
+28.2
diff --git a/evaluation/dev/en-ru/flores-dev.google.ru.bleu b/evaluation/dev/en-ru/flores-dev.google.ru.bleu
@@ -0,0 +1 @@
+33.6
diff --git a/evaluation/dev/en-ru/flores-dev.microsoft.ru.bleu b/evaluation/dev/en-ru/flores-dev.microsoft.ru.bleu
@@ -0,0 +1 @@
+33.2
diff --git a/evaluation/dev/en-ru/flores-test.bergamot.ru.bleu b/evaluation/dev/en-ru/flores-test.bergamot.ru.bleu
@@ -0,0 +1 @@
+27.7
diff --git a/evaluation/dev/en-ru/flores-test.google.ru.bleu b/evaluation/dev/en-ru/flores-test.google.ru.bleu
@@ -0,0 +1 @@
+33.6
diff --git a/evaluation/dev/en-ru/flores-test.microsoft.ru.bleu b/evaluation/dev/en-ru/flores-test.microsoft.ru.bleu
@@ -0,0 +1 @@
+33.0
diff --git a/evaluation/dev/img/avg.png b/evaluation/dev/img/avg.png
diff --git a/evaluation/dev/img/en-pt.png b/evaluation/dev/img/en-pt.png
diff --git a/evaluation/dev/img/en-ru.png b/evaluation/dev/img/en-ru.png
diff --git a/evaluation/dev/img/it-en.png b/evaluation/dev/img/it-en.png
diff --git a/evaluation/dev/img/pt-en.png b/evaluation/dev/img/pt-en.png
diff --git a/evaluation/dev/img/ru-en.png b/evaluation/dev/img/ru-en.png
diff --git a/evaluation/dev/it-en/flores-dev.bergamot.en.bleu b/evaluation/dev/it-en/flores-dev.bergamot.en.bleu
@@ -0,0 +1 @@
+30.7
diff --git a/evaluation/dev/it-en/flores-dev.google.en.bleu b/evaluation/dev/it-en/flores-dev.google.en.bleu
@@ -0,0 +1 @@
+33.4
diff --git a/evaluation/dev/it-en/flores-dev.microsoft.en.bleu b/evaluation/dev/it-en/flores-dev.microsoft.en.bleu
@@ -0,0 +1 @@
+33.3
diff --git a/evaluation/dev/it-en/flores-test.bergamot.en.bleu b/evaluation/dev/it-en/flores-test.bergamot.en.bleu
@@ -0,0 +1 @@
+29.9
diff --git a/evaluation/dev/it-en/flores-test.google.en.bleu b/evaluation/dev/it-en/flores-test.google.en.bleu
@@ -0,0 +1 @@
+33.3
diff --git a/evaluation/dev/it-en/flores-test.microsoft.en.bleu b/evaluation/dev/it-en/flores-test.microsoft.en.bleu
@@ -0,0 +1 @@
+32.7
diff --git a/evaluation/dev/pt-en/flores-dev.bergamot.en.bleu b/evaluation/dev/pt-en/flores-dev.bergamot.en.bleu
@@ -0,0 +1 @@
+45.9
diff --git a/evaluation/dev/pt-en/flores-dev.google.en.bleu b/evaluation/dev/pt-en/flores-dev.google.en.bleu
@@ -0,0 +1 @@
+49.5
diff --git a/evaluation/dev/pt-en/flores-dev.microsoft.en.bleu b/evaluation/dev/pt-en/flores-dev.microsoft.en.bleu
@@ -0,0 +1 @@
+49.8
diff --git a/evaluation/dev/pt-en/flores-test.bergamot.en.bleu b/evaluation/dev/pt-en/flores-test.bergamot.en.bleu
@@ -0,0 +1 @@
+44.7
diff --git a/evaluation/dev/pt-en/flores-test.google.en.bleu b/evaluation/dev/pt-en/flores-test.google.en.bleu
@@ -0,0 +1 @@
+48.9
diff --git a/evaluation/dev/pt-en/flores-test.microsoft.en.bleu b/evaluation/dev/pt-en/flores-test.microsoft.en.bleu
@@ -0,0 +1 @@
+48.6
diff --git a/evaluation/dev/results.md b/evaluation/dev/results.md
@@ -57,50 +57,60 @@ Both absolute and relative differences in BLEU scores between Bergamot and other
 
 ## avg
 
-| Translator/Dataset | ru-en | en-ru | it-en | pt-en |
-| --- | --- | --- | --- | --- |
-| bergamot | 33.74 | 28.51 | 34.30 | 39.70 |
-| google | 36.59 (+2.84, +8.43%) | 33.75 (+5.24, +18.37%) | 35.65 (+1.35, +3.94%) | 38.90 (-0.80, -2.02%) |
-| microsoft | 37.03 (+3.29, +9.75%) | 33.83 (+5.31, +18.63%) | 36.10 (+1.80, +5.25%) | 40.90 (+1.20, +3.02%) |
+| Translator/Dataset | en-pt | ru-en | en-ru | it-en | pt-en |
+| --- | --- | --- | --- | --- | --- |
+| bergamot | 46.55 | 33.37 | 28.40 | 32.30 | 43.43 |
+| google | 56.05 (+9.50, +20.41%) | 36.53 (+3.15, +9.45%) | 33.72 (+5.32, +18.73%) | 34.50 (+2.20, +6.81%) | 45.77 (+2.33, +5.37%) |
+| microsoft | 50.25 (+3.70, +7.95%) | 36.87 (+3.50, +10.49%) | 33.68 (+5.28, +18.59%) | 34.55 (+2.25, +6.97%) | 46.43 (+3.00, +6.91%) |
 
 ![Results](img/avg.png)
 
+## en-pt
+
+| Translator/Dataset | flores-test | flores-dev |
+| --- | --- | --- |
+| bergamot | 46.20 | 46.90 |
+| google | 55.70 (+9.50, +20.56%) | 56.40 (+9.50, +20.26%) |
+| microsoft | 50.70 (+4.50, +9.74%) | 49.80 (+2.90, +6.18%) |
+
+![Results](img/en-pt.png)
+
 ## ru-en
 
-| Translator/Dataset | mtedx_test | wmt19 | wmt17 | wmt14 | wmt15 | wmt16 | wmt13 | wmt18 | wmt20 |
-| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| bergamot | 24.20 | 39.30 | 37.90 | 38.00 | 33.70 | 33.40 | 29.50 | 32.30 | 35.40 |
-| google | 25.00 (+0.80, +3.31%) | 42.40 (+3.10, +7.89%) | 41.50 (+3.60, +9.50%) | 41.20 (+3.20, +8.42%) | 37.50 (+3.80, +11.28%) | 36.60 (+3.20, +9.58%) | 31.40 (+1.90, +6.44%) | 36.00 (+3.70, +11.46%) | 37.70 (+2.30, +6.50%) |
-| microsoft | 26.10 (+1.90, +7.85%) | 42.60 (+3.30, +8.40%) | 41.60 (+3.70, +9.76%) | 41.70 (+3.70, +9.74%) | 37.80 (+4.10, +12.17%) | 37.60 (+4.20, +12.57%) | 31.20 (+1.70, +5.76%) | 36.90 (+4.60, +14.24%) | 37.80 (+2.40, +6.78%) |
+| Translator/Dataset | mtedx_test | wmt19 | wmt17 | flores-dev | flores-test | wmt14 | wmt15 | wmt16 | wmt13 | wmt18 | wmt20 |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| bergamot | 24.20 | 39.30 | 37.90 | 31.90 | 31.50 | 38.00 | 33.70 | 33.40 | 29.50 | 32.30 | 35.40 |
+| google | 25.00 (+0.80, +3.31%) | 42.40 (+3.10, +7.89%) | 41.50 (+3.60, +9.50%) | 37.00 (+5.10, +15.99%) | 35.50 (+4.00, +12.70%) | 41.20 (+3.20, +8.42%) | 37.50 (+3.80, +11.28%) | 36.60 (+3.20, +9.58%) | 31.40 (+1.90, +6.44%) | 36.00 (+3.70, +11.46%) | 37.70 (+2.30, +6.50%) |
+| microsoft | 26.10 (+1.90, +7.85%) | 42.60 (+3.30, +8.40%) | 41.60 (+3.70, +9.76%) | 36.20 (+4.30, +13.48%) | 36.10 (+4.60, +14.60%) | 41.70 (+3.70, +9.74%) | 37.80 (+4.10, +12.17%) | 37.60 (+4.20, +12.57%) | 31.20 (+1.70, +5.76%) | 36.90 (+4.60, +14.24%) | 37.80 (+2.40, +6.78%) |
 
 ![Results](img/ru-en.png)
 
 ## en-ru
 
-| Translator/Dataset | wmt16 | wmt15 | wmt18 | wmt14 | wmt17 | wmt20 | wmt13 | wmt19 |
-| --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| bergamot | 28.80 | 29.70 | 26.50 | 35.30 | 31.60 | 21.00 | 25.20 | 30.00 |
-| google | 34.00 (+5.20, +18.06%) | 35.00 (+5.30, +17.85%) | 34.10 (+7.60, +28.68%) | 43.40 (+8.10, +22.95%) | 37.30 (+5.70, +18.04%) | 26.40 (+5.40, +25.71%) | 27.60 (+2.40, +9.52%) | 32.20 (+2.20, +7.33%) |
-| microsoft | 33.80 (+5.00, +17.36%) | 35.60 (+5.90, +19.87%) | 33.20 (+6.70, +25.28%) | 44.10 (+8.80, +24.93%) | 38.10 (+6.50, +20.57%) | 26.00 (+5.00, +23.81%) | 27.10 (+1.90, +7.54%) | 32.70 (+2.70, +9.00%) |
+| Translator/Dataset | wmt16 | wmt15 | flores-dev | wmt18 | wmt14 | wmt17 | wmt20 | wmt13 | wmt19 | flores-test |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| bergamot | 28.80 | 29.70 | 28.20 | 26.50 | 35.30 | 31.60 | 21.00 | 25.20 | 30.00 | 27.70 |
+| google | 34.00 (+5.20, +18.06%) | 35.00 (+5.30, +17.85%) | 33.60 (+5.40, +19.15%) | 34.10 (+7.60, +28.68%) | 43.40 (+8.10, +22.95%) | 37.30 (+5.70, +18.04%) | 26.40 (+5.40, +25.71%) | 27.60 (+2.40, +9.52%) | 32.20 (+2.20, +7.33%) | 33.60 (+5.90, +21.30%) |
+| microsoft | 33.80 (+5.00, +17.36%) | 35.60 (+5.90, +19.87%) | 33.20 (+5.00, +17.73%) | 33.20 (+6.70, +25.28%) | 44.10 (+8.80, +24.93%) | 38.10 (+6.50, +20.57%) | 26.00 (+5.00, +23.81%) | 27.10 (+1.90, +7.54%) | 32.70 (+2.70, +9.00%) | 33.00 (+5.30, +19.13%) |
 
 ![Results](img/en-ru.png)
 
 ## it-en
 
-| Translator/Dataset | mtedx_test | wmt09 |
-| --- | --- | --- |
-| bergamot | 35.50 | 33.10 |
-| google | 35.90 (+0.40, +1.13%) | 35.40 (+2.30, +6.95%) |
-| microsoft | 36.40 (+0.90, +2.54%) | 35.80 (+2.70, +8.16%) |
+| Translator/Dataset | mtedx_test | flores-dev | flores-test | wmt09 |
+| --- | --- | --- | --- | --- |
+| bergamot | 35.50 | 30.70 | 29.90 | 33.10 |
+| google | 35.90 (+0.40, +1.13%) | 33.40 (+2.70, +8.79%) | 33.30 (+3.40, +11.37%) | 35.40 (+2.30, +6.95%) |
+| microsoft | 36.40 (+0.90, +2.54%) | 33.30 (+2.60, +8.47%) | 32.70 (+2.80, +9.36%) | 35.80 (+2.70, +8.16%) |
 
 ![Results](img/it-en.png)
 
 ## pt-en
 
-| Translator/Dataset | mtedx_test |
-| --- | --- |
-| bergamot | 39.70 |
-| google | 38.90 (-0.80, -2.02%) |
-| microsoft | 40.90 (+1.20, +3.02%) |
+| Translator/Dataset | mtedx_test | flores-dev | flores-test |
+| --- | --- | --- | --- |
+| bergamot | 39.70 | 45.90 | 44.70 |
+| google | 38.90 (-0.80, -2.02%) | 49.50 (+3.60, +7.84%) | 48.90 (+4.20, +9.40%) |
+| microsoft | 40.90 (+1.20, +3.02%) | 49.80 (+3.90, +8.50%) | 48.60 (+3.90, +8.72%) |
 
 ![Results](img/pt-en.png)
diff --git a/evaluation/dev/ru-en/flores-dev.bergamot.en.bleu b/evaluation/dev/ru-en/flores-dev.bergamot.en.bleu
@@ -0,0 +1 @@
+31.9
diff --git a/evaluation/dev/ru-en/flores-dev.google.en.bleu b/evaluation/dev/ru-en/flores-dev.google.en.bleu
@@ -0,0 +1 @@
+37.0
diff --git a/evaluation/dev/ru-en/flores-dev.microsoft.en.bleu b/evaluation/dev/ru-en/flores-dev.microsoft.en.bleu
@@ -0,0 +1 @@
+36.2
diff --git a/evaluation/dev/ru-en/flores-test.bergamot.en.bleu b/evaluation/dev/ru-en/flores-test.bergamot.en.bleu
@@ -0,0 +1 @@
+31.5
diff --git a/evaluation/dev/ru-en/flores-test.google.en.bleu b/evaluation/dev/ru-en/flores-test.google.en.bleu
@@ -0,0 +1 @@
+35.5
diff --git a/evaluation/dev/ru-en/flores-test.microsoft.en.bleu b/evaluation/dev/ru-en/flores-test.microsoft.en.bleu
@@ -0,0 +1 @@
+36.1
diff --git a/evaluation/prod/cs-en/flores-dev.bergamot.en.bleu b/evaluation/prod/cs-en/flores-dev.bergamot.en.bleu
@@ -0,0 +1 @@
+35.3
diff --git a/evaluation/prod/cs-en/flores-dev.google.en.bleu b/evaluation/prod/cs-en/flores-dev.google.en.bleu
@@ -0,0 +1 @@
+38.6
diff --git a/evaluation/prod/cs-en/flores-dev.microsoft.en.bleu b/evaluation/prod/cs-en/flores-dev.microsoft.en.bleu
@@ -0,0 +1 @@
+40.0
diff --git a/evaluation/prod/cs-en/flores-test.bergamot.en.bleu b/evaluation/prod/cs-en/flores-test.bergamot.en.bleu
@@ -0,0 +1 @@
+35.3
diff --git a/evaluation/prod/cs-en/flores-test.google.en.bleu b/evaluation/prod/cs-en/flores-test.google.en.bleu
@@ -0,0 +1 @@
+39.0
diff --git a/evaluation/prod/cs-en/flores-test.microsoft.en.bleu b/evaluation/prod/cs-en/flores-test.microsoft.en.bleu
@@ -0,0 +1 @@
+40.3
diff --git a/evaluation/prod/en-cs/flores-dev.bergamot.cs.bleu b/evaluation/prod/en-cs/flores-dev.bergamot.cs.bleu
@@ -0,0 +1 @@
+30.1
diff --git a/evaluation/prod/en-cs/flores-dev.google.cs.bleu b/evaluation/prod/en-cs/flores-dev.google.cs.bleu
@@ -0,0 +1 @@
+34.1
diff --git a/evaluation/prod/en-cs/flores-dev.microsoft.cs.bleu b/evaluation/prod/en-cs/flores-dev.microsoft.cs.bleu
@@ -0,0 +1 @@
+33.5
diff --git a/evaluation/prod/en-cs/flores-test.bergamot.cs.bleu b/evaluation/prod/en-cs/flores-test.bergamot.cs.bleu
@@ -0,0 +1 @@
+30.2
diff --git a/evaluation/prod/en-cs/flores-test.google.cs.bleu b/evaluation/prod/en-cs/flores-test.google.cs.bleu
@@ -0,0 +1 @@
+34.4
diff --git a/evaluation/prod/en-cs/flores-test.microsoft.cs.bleu b/evaluation/prod/en-cs/flores-test.microsoft.cs.bleu
@@ -0,0 +1 @@
+34.0
diff --git a/evaluation/prod/en-de/flores-dev.bergamot.de.bleu b/evaluation/prod/en-de/flores-dev.bergamot.de.bleu
@@ -0,0 +1 @@
+37.6
diff --git a/evaluation/prod/en-de/flores-dev.google.de.bleu b/evaluation/prod/en-de/flores-dev.google.de.bleu
@@ -0,0 +1 @@
+43.7
diff --git a/evaluation/prod/en-de/flores-dev.microsoft.de.bleu b/evaluation/prod/en-de/flores-dev.microsoft.de.bleu
@@ -0,0 +1 @@
+44.0
diff --git a/evaluation/prod/en-de/flores-test.bergamot.de.bleu b/evaluation/prod/en-de/flores-test.bergamot.de.bleu
@@ -0,0 +1 @@
+37.1
diff --git a/evaluation/prod/en-de/flores-test.google.de.bleu b/evaluation/prod/en-de/flores-test.google.de.bleu
@@ -0,0 +1 @@
+42.3
diff --git a/evaluation/prod/en-de/flores-test.microsoft.de.bleu b/evaluation/prod/en-de/flores-test.microsoft.de.bleu
@@ -0,0 +1 @@
+42.9
diff --git a/evaluation/prod/en-es/flores-dev.bergamot.es.bleu b/evaluation/prod/en-es/flores-dev.bergamot.es.bleu
@@ -0,0 +1 @@
+25.9
diff --git a/evaluation/prod/en-es/flores-dev.google.es.bleu b/evaluation/prod/en-es/flores-dev.google.es.bleu
@@ -0,0 +1 @@
+30.5
diff --git a/evaluation/prod/en-es/flores-dev.microsoft.es.bleu b/evaluation/prod/en-es/flores-dev.microsoft.es.bleu
@@ -0,0 +1 @@
+28.4
diff --git a/evaluation/prod/en-es/flores-test.bergamot.es.bleu b/evaluation/prod/en-es/flores-test.bergamot.es.bleu
@@ -0,0 +1 @@
+26.2
diff --git a/evaluation/prod/en-es/flores-test.google.es.bleu b/evaluation/prod/en-es/flores-test.google.es.bleu
@@ -0,0 +1 @@
+30.4
diff --git a/evaluation/prod/en-es/flores-test.microsoft.es.bleu b/evaluation/prod/en-es/flores-test.microsoft.es.bleu
@@ -0,0 +1 @@
+28.5
diff --git a/evaluation/prod/en-et/flores-dev.bergamot.et.bleu b/evaluation/prod/en-et/flores-dev.bergamot.et.bleu
@@ -0,0 +1 @@
+25.6
diff --git a/evaluation/prod/en-et/flores-dev.google.et.bleu b/evaluation/prod/en-et/flores-dev.google.et.bleu
@@ -0,0 +1 @@
+30.2
diff --git a/evaluation/prod/en-et/flores-dev.microsoft.et.bleu b/evaluation/prod/en-et/flores-dev.microsoft.et.bleu
@@ -0,0 +1 @@
+28.6
diff --git a/evaluation/prod/en-et/flores-test.bergamot.et.bleu b/evaluation/prod/en-et/flores-test.bergamot.et.bleu
@@ -0,0 +1 @@
+25.7
diff --git a/evaluation/prod/en-et/flores-test.google.et.bleu b/evaluation/prod/en-et/flores-test.google.et.bleu
@@ -0,0 +1 @@
+29.0
diff --git a/evaluation/prod/en-et/flores-test.microsoft.et.bleu b/evaluation/prod/en-et/flores-test.microsoft.et.bleu
@@ -0,0 +1 @@
+29.2
diff --git a/evaluation/prod/es-en/flores-dev.bergamot.en.bleu b/evaluation/prod/es-en/flores-dev.bergamot.en.bleu
@@ -0,0 +1 @@
+27.5
diff --git a/evaluation/prod/es-en/flores-dev.google.en.bleu b/evaluation/prod/es-en/flores-dev.google.en.bleu
@@ -0,0 +1 @@
+30.5
diff --git a/evaluation/prod/es-en/flores-dev.microsoft.en.bleu b/evaluation/prod/es-en/flores-dev.microsoft.en.bleu
@@ -0,0 +1 @@
+30.3
diff --git a/evaluation/prod/es-en/flores-test.bergamot.en.bleu b/evaluation/prod/es-en/flores-test.bergamot.en.bleu
@@ -0,0 +1 @@
+26.8
diff --git a/evaluation/prod/es-en/flores-test.google.en.bleu b/evaluation/prod/es-en/flores-test.google.en.bleu
@@ -0,0 +1 @@
+30.3
diff --git a/evaluation/prod/es-en/flores-test.microsoft.en.bleu b/evaluation/prod/es-en/flores-test.microsoft.en.bleu
@@ -0,0 +1 @@
+29.9
diff --git a/evaluation/prod/et-en/flores-dev.bergamot.en.bleu b/evaluation/prod/et-en/flores-dev.bergamot.en.bleu
@@ -0,0 +1 @@
+33.5
diff --git a/evaluation/prod/et-en/flores-dev.google.en.bleu b/evaluation/prod/et-en/flores-dev.google.en.bleu
@@ -0,0 +1 @@
+38.3
diff --git a/evaluation/prod/et-en/flores-dev.microsoft.en.bleu b/evaluation/prod/et-en/flores-dev.microsoft.en.bleu
@@ -0,0 +1 @@
+37.4
diff --git a/evaluation/prod/et-en/flores-test.bergamot.en.bleu b/evaluation/prod/et-en/flores-test.bergamot.en.bleu
@@ -0,0 +1 @@
+32.7
diff --git a/evaluation/prod/et-en/flores-test.google.en.bleu b/evaluation/prod/et-en/flores-test.google.en.bleu
@@ -0,0 +1 @@
+37.0
diff --git a/evaluation/prod/et-en/flores-test.microsoft.en.bleu b/evaluation/prod/et-en/flores-test.microsoft.en.bleu
@@ -0,0 +1 @@
+37.0
diff --git a/evaluation/prod/img/avg.png b/evaluation/prod/img/avg.png
diff --git a/evaluation/prod/img/cs-en.png b/evaluation/prod/img/cs-en.png
diff --git a/evaluation/prod/img/en-cs.png b/evaluation/prod/img/en-cs.png
diff --git a/evaluation/prod/img/en-de.png b/evaluation/prod/img/en-de.png
diff --git a/evaluation/prod/img/en-es.png b/evaluation/prod/img/en-es.png
diff --git a/evaluation/prod/img/en-et.png b/evaluation/prod/img/en-et.png
diff --git a/evaluation/prod/img/es-en.png b/evaluation/prod/img/es-en.png
diff --git a/evaluation/prod/img/et-en.png b/evaluation/prod/img/et-en.png
diff --git a/evaluation/prod/results.md b/evaluation/prod/results.md
@@ -59,78 +59,78 @@ Both absolute and relative differences in BLEU scores between Bergamot and other
 
 | Translator/Dataset | cs-en | en-et | et-en | en-es | en-cs | es-en | en-de |
 | --- | --- | --- | --- | --- | --- | --- | --- |
-| bergamot | 29.51 | 25.20 | 30.90 | 34.53 | 23.80 | 33.87 | 30.32 |
-| google | 31.33 (+1.82, +6.18%) | 26.60 (+1.40, +5.56%) | 32.10 (+1.20, +3.88%) | 36.17 (+1.63, +4.73%) | 25.58 (+1.78, +7.47%) | 34.57 (+0.70, +2.07%) | 31.63 (+1.31, +4.31%) |
-| microsoft | 32.42 (+2.92, +9.88%) | 27.60 (+2.40, +9.52%) | 34.10 (+3.20, +10.36%) | 35.53 (+1.00, +2.90%) | 26.83 (+3.03, +12.73%) | 33.74 (-0.13, -0.38%) | 32.13 (+1.81, +5.96%) |
+| bergamot | 30.34 | 25.50 | 32.37 | 32.41 | 24.65 | 32.38 | 31.20 |
+| google | 32.40 (+2.06, +6.80%) | 28.60 (+3.10, +12.16%) | 35.80 (+3.43, +10.61%) | 34.74 (+2.32, +7.17%) | 26.73 (+2.09, +8.47%) | 33.64 (+1.27, +3.91%) | 33.05 (+1.85, +5.93%) |
+| microsoft | 33.53 (+3.19, +10.53%) | 28.47 (+2.97, +11.63%) | 36.17 (+3.80, +11.74%) | 33.76 (+1.35, +4.17%) | 27.75 (+3.11, +12.60%) | 32.93 (+0.56, +1.72%) | 33.54 (+2.34, +7.51%) |
 
 ![Results](img/avg.png)
 
 ## cs-en
 
-| Translator/Dataset | wmt08 | wmt17 | wmt10 | wmt12 | wmt11 | wmt14 | wmt15 | wmt16 | wmt13 | wmt18 | wmt09 | wmt20 |
-| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| bergamot | 24.50 | 30.20 | 28.20 | 26.50 | 28.10 | 35.00 | 32.00 | 33.40 | 30.30 | 31.30 | 27.60 | 27.00 |
-| google | 26.30 (+1.80, +7.35%) | 31.20 (+1.00, +3.31%) | 30.50 (+2.30, +8.16%) | 28.60 (+2.10, +7.92%) | 30.20 (+2.10, +7.47%) | 38.00 (+3.00, +8.57%) | 33.60 (+1.60, +5.00%) | 34.80 (+1.40, +4.19%) | 32.40 (+2.10, +6.93%) | 32.10 (+0.80, +2.56%) | 29.90 (+2.30, +8.33%) | 28.40 (+1.40, +5.19%) |
-| microsoft | 26.40 (+1.90, +7.76%) | 33.60 (+3.40, +11.26%) | 30.70 (+2.50, +8.87%) | 29.70 (+3.20, +12.08%) | 30.90 (+2.80, +9.96%) | 39.90 (+4.90, +14.00%) | 34.70 (+2.70, +8.44%) | 38.30 (+4.90, +14.67%) | 33.40 (+3.10, +10.23%) | 34.30 (+3.00, +9.58%) | 29.60 (+2.00, +7.25%) | 27.60 (+0.60, +2.22%) |
+| Translator/Dataset | wmt08 | wmt17 | wmt10 | flores-dev | flores-test | wmt12 | wmt11 | wmt14 | wmt15 | wmt16 | wmt13 | wmt18 | wmt09 | wmt20 |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| bergamot | 24.50 | 30.20 | 28.20 | 35.30 | 35.30 | 26.50 | 28.10 | 35.00 | 32.00 | 33.40 | 30.30 | 31.30 | 27.60 | 27.00 |
+| google | 26.30 (+1.80, +7.35%) | 31.20 (+1.00, +3.31%) | 30.50 (+2.30, +8.16%) | 38.60 (+3.30, +9.35%) | 39.00 (+3.70, +10.48%) | 28.60 (+2.10, +7.92%) | 30.20 (+2.10, +7.47%) | 38.00 (+3.00, +8.57%) | 33.60 (+1.60, +5.00%) | 34.80 (+1.40, +4.19%) | 32.40 (+2.10, +6.93%) | 32.10 (+0.80, +2.56%) | 29.90 (+2.30, +8.33%) | 28.40 (+1.40, +5.19%) |
+| microsoft | 26.40 (+1.90, +7.76%) | 33.60 (+3.40, +11.26%) | 30.70 (+2.50, +8.87%) | 40.00 (+4.70, +13.31%) | 40.30 (+5.00, +14.16%) | 29.70 (+3.20, +12.08%) | 30.90 (+2.80, +9.96%) | 39.90 (+4.90, +14.00%) | 34.70 (+2.70, +8.44%) | 38.30 (+4.90, +14.67%) | 33.40 (+3.10, +10.23%) | 34.30 (+3.00, +9.58%) | 29.60 (+2.00, +7.25%) | 27.60 (+0.60, +2.22%) |
 
 ![Results](img/cs-en.png)
 
 ## en-et
 
-| Translator/Dataset | wmt18 |
-| --- | --- |
-| bergamot | 25.20 |
-| google | 26.60 (+1.40, +5.56%) |
-| microsoft | 27.60 (+2.40, +9.52%) |
+| Translator/Dataset | flores-dev | flores-test | wmt18 |
+| --- | --- | --- | --- |
+| bergamot | 25.60 | 25.70 | 25.20 |
+| google | 30.20 (+4.60, +17.97%) | 29.00 (+3.30, +12.84%) | 26.60 (+1.40, +5.56%) |
+| microsoft | 28.60 (+3.00, +11.72%) | 29.20 (+3.50, +13.62%) | 27.60 (+2.40, +9.52%) |
 
 ![Results](img/en-et.png)
 
 ## et-en
 
-| Translator/Dataset | wmt18 |
-| --- | --- |
-| bergamot | 30.90 |
-| google | 32.10 (+1.20, +3.88%) |
-| microsoft | 34.10 (+3.20, +10.36%) |
+| Translator/Dataset | flores-dev | flores-test | wmt18 |
+| --- | --- | --- | --- |
+| bergamot | 33.50 | 32.70 | 30.90 |
+| google | 38.30 (+4.80, +14.33%) | 37.00 (+4.30, +13.15%) | 32.10 (+1.20, +3.88%) |
+| microsoft | 37.40 (+3.90, +11.64%) | 37.00 (+4.30, +13.15%) | 34.10 (+3.20, +10.36%) |
 
 ![Results](img/et-en.png)
 
 ## en-es
 
-| Translator/Dataset | wmt08 | wmt11 | wmt09 | wmt10 | wmt13 | wmt12 |
-| --- | --- | --- | --- | --- | --- | --- |
-| bergamot | 29.00 | 37.90 | 29.90 | 36.70 | 34.80 | 38.90 |
-| google | 30.00 (+1.00, +3.45%) | 39.90 (+2.00, +5.28%) | 30.90 (+1.00, +3.34%) | 38.80 (+2.10, +5.72%) | 36.90 (+2.10, +6.03%) | 40.50 (+1.60, +4.11%) |
-| microsoft | 29.90 (+0.90, +3.10%) | 39.10 (+1.20, +3.17%) | 30.70 (+0.80, +2.68%) | 37.80 (+1.10, +3.00%) | 35.70 (+0.90, +2.59%) | 40.00 (+1.10, +2.83%) |
+| Translator/Dataset | wmt08 | wmt11 | wmt09 | flores-dev | wmt10 | wmt13 | wmt12 | flores-test |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| bergamot | 29.00 | 37.90 | 29.90 | 25.90 | 36.70 | 34.80 | 38.90 | 26.20 |
+| google | 30.00 (+1.00, +3.45%) | 39.90 (+2.00, +5.28%) | 30.90 (+1.00, +3.34%) | 30.50 (+4.60, +17.76%) | 38.80 (+2.10, +5.72%) | 36.90 (+2.10, +6.03%) | 40.50 (+1.60, +4.11%) | 30.40 (+4.20, +16.03%) |
+| microsoft | 29.90 (+0.90, +3.10%) | 39.10 (+1.20, +3.17%) | 30.70 (+0.80, +2.68%) | 28.40 (+2.50, +9.65%) | 37.80 (+1.10, +3.00%) | 35.70 (+0.90, +2.59%) | 40.00 (+1.10, +2.83%) | 28.50 (+2.30, +8.78%) |
 
 ![Results](img/en-es.png)
 
 ## en-cs
 
-| Translator/Dataset | wmt12 | wmt19 | wmt14 | wmt17 | wmt20 | wmt15 | wmt08 | wmt11 | wmt18 | wmt09 | wmt10 | wmt13 | wmt16 |
-| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| bergamot | 18.80 | 27.10 | 28.70 | 23.40 | 32.70 | 25.40 | 19.00 | 20.70 | 22.70 | 20.80 | 20.90 | 23.40 | 25.80 |
-| google | 20.70 (+1.90, +10.11%) | 27.20 (+0.10, +0.37%) | 31.20 (+2.50, +8.71%) | 24.70 (+1.30, +5.56%) | 35.50 (+2.80, +8.56%) | 26.80 (+1.40, +5.51%) | 20.50 (+1.50, +7.89%) | 23.00 (+2.30, +11.11%) | 24.40 (+1.70, +7.49%) | 22.60 (+1.80, +8.65%) | 22.40 (+1.50, +7.18%) | 25.20 (+1.80, +7.69%) | 28.30 (+2.50, +9.69%) |
-| microsoft | 22.90 (+4.10, +21.81%) | 27.20 (+0.10, +0.37%) | 31.90 (+3.20, +11.15%) | 25.60 (+2.20, +9.40%) | 34.10 (+1.40, +4.28%) | 27.40 (+2.00, +7.87%) | 22.60 (+3.60, +18.95%) | 25.30 (+4.60, +22.22%) | 24.90 (+2.20, +9.69%) | 25.00 (+4.20, +20.19%) | 24.30 (+3.40, +16.27%) | 27.70 (+4.30, +18.38%) | 29.90 (+4.10, +15.89%) |
+| Translator/Dataset | wmt12 | flores-dev | wmt19 | wmt14 | wmt17 | wmt20 | wmt15 | wmt08 | wmt11 | wmt18 | wmt09 | wmt10 | flores-test | wmt13 | wmt16 |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| bergamot | 18.80 | 30.10 | 27.10 | 28.70 | 23.40 | 32.70 | 25.40 | 19.00 | 20.70 | 22.70 | 20.80 | 20.90 | 30.20 | 23.40 | 25.80 |
+| google | 20.70 (+1.90, +10.11%) | 34.10 (+4.00, +13.29%) | 27.20 (+0.10, +0.37%) | 31.20 (+2.50, +8.71%) | 24.70 (+1.30, +5.56%) | 35.50 (+2.80, +8.56%) | 26.80 (+1.40, +5.51%) | 20.50 (+1.50, +7.89%) | 23.00 (+2.30, +11.11%) | 24.40 (+1.70, +7.49%) | 22.60 (+1.80, +8.65%) | 22.40 (+1.50, +7.18%) | 34.40 (+4.20, +13.91%) | 25.20 (+1.80, +7.69%) | 28.30 (+2.50, +9.69%) |
+| microsoft | 22.90 (+4.10, +21.81%) | 33.50 (+3.40, +11.30%) | 27.20 (+0.10, +0.37%) | 31.90 (+3.20, +11.15%) | 25.60 (+2.20, +9.40%) | 34.10 (+1.40, +4.28%) | 27.40 (+2.00, +7.87%) | 22.60 (+3.60, +18.95%) | 25.30 (+4.60, +22.22%) | 24.90 (+2.20, +9.69%) | 25.00 (+4.20, +20.19%) | 24.30 (+3.40, +16.27%) | 34.00 (+3.80, +12.58%) | 27.70 (+4.30, +18.38%) | 29.90 (+4.10, +15.89%) |
 
 ![Results](img/en-cs.png)
 
 ## es-en
 
-| Translator/Dataset | wmt08 | mtedx_test | wmt10 | wmt12 | wmt11 | wmt13 | wmt09 |
-| --- | --- | --- | --- | --- | --- | --- | --- |
-| bergamot | 27.30 | 36.80 | 35.80 | 38.30 | 34.30 | 35.20 | 29.40 |
-| google | 28.30 (+1.00, +3.66%) | 35.40 (-1.40, -3.80%) | 37.00 (+1.20, +3.35%) | 38.80 (+0.50, +1.31%) | 35.20 (+0.90, +2.62%) | 35.70 (+0.50, +1.42%) | 31.60 (+2.20, +7.48%) |
-| microsoft | 26.80 (-0.50, -1.83%) | 37.60 (+0.80, +2.17%) | 35.40 (-0.40, -1.12%) | 37.80 (-0.50, -1.31%) | 33.70 (-0.60, -1.75%) | 35.30 (+0.10, +0.28%) | 29.60 (+0.20, +0.68%) |
+| Translator/Dataset | wmt08 | mtedx_test | wmt10 | flores-dev | flores-test | wmt12 | wmt11 | wmt13 | wmt09 |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| bergamot | 27.30 | 36.80 | 35.80 | 27.50 | 26.80 | 38.30 | 34.30 | 35.20 | 29.40 |
+| google | 28.30 (+1.00, +3.66%) | 35.40 (-1.40, -3.80%) | 37.00 (+1.20, +3.35%) | 30.50 (+3.00, +10.91%) | 30.30 (+3.50, +13.06%) | 38.80 (+0.50, +1.31%) | 35.20 (+0.90, +2.62%) | 35.70 (+0.50, +1.42%) | 31.60 (+2.20, +7.48%) |
+| microsoft | 26.80 (-0.50, -1.83%) | 37.60 (+0.80, +2.17%) | 35.40 (-0.40, -1.12%) | 30.30 (+2.80, +10.18%) | 29.90 (+3.10, +11.57%) | 37.80 (-0.50, -1.31%) | 33.70 (-0.60, -1.75%) | 35.30 (+0.10, +0.28%) | 29.60 (+0.20, +0.68%) |
 
 ![Results](img/es-en.png)
 
 ## en-de
 
-| Translator/Dataset | iwslt17 | wmt20 | wmt08 | wmt12 | wmt19 | wmt10 | wmt11 | wmt09 | wmt13 | wmt17 | wmt16 | wmt18 | wmt14 | wmt15 |
-| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| bergamot | 25.70 | 34.10 | 23.40 | 24.00 | 41.50 | 25.80 | 23.00 | 22.80 | 27.60 | 30.80 | 38.20 | 45.60 | 29.50 | 32.50 |
-| google | 28.90 (+3.20, +12.45%) | 36.50 (+2.40, +7.04%) | 23.70 (+0.30, +1.28%) | 24.70 (+0.70, +2.92%) | 43.50 (+2.00, +4.82%) | 26.50 (+0.70, +2.71%) | 24.10 (+1.10, +4.78%) | 23.60 (+0.80, +3.51%) | 28.80 (+1.20, +4.35%) | 31.50 (+0.70, +2.27%) | 38.60 (+0.40, +1.05%) | 47.80 (+2.20, +4.82%) | 30.90 (+1.40, +4.75%) | 33.70 (+1.20, +3.69%) |
-| microsoft | 28.20 (+2.50, +9.73%) | 36.10 (+2.00, +5.87%) | 24.00 (+0.60, +2.56%) | 25.30 (+1.30, +5.42%) | 43.80 (+2.30, +5.54%) | 27.20 (+1.40, +5.43%) | 23.70 (+0.70, +3.04%) | 23.90 (+1.10, +4.82%) | 28.80 (+1.20, +4.35%) | 33.10 (+2.30, +7.47%) | 40.50 (+2.30, +6.02%) | 48.70 (+3.10, +6.80%) | 32.20 (+2.70, +9.15%) | 34.30 (+1.80, +5.54%) |
+| Translator/Dataset | flores-test | iwslt17 | wmt20 | wmt08 | wmt12 | wmt19 | wmt10 | wmt11 | flores-dev | wmt09 | wmt13 | wmt17 | wmt16 | wmt18 | wmt14 | wmt15 |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| bergamot | 37.10 | 25.70 | 34.10 | 23.40 | 24.00 | 41.50 | 25.80 | 23.00 | 37.60 | 22.80 | 27.60 | 30.80 | 38.20 | 45.60 | 29.50 | 32.50 |
+| google | 42.30 (+5.20, +14.02%) | 28.90 (+3.20, +12.45%) | 36.50 (+2.40, +7.04%) | 23.70 (+0.30, +1.28%) | 24.70 (+0.70, +2.92%) | 43.50 (+2.00, +4.82%) | 26.50 (+0.70, +2.71%) | 24.10 (+1.10, +4.78%) | 43.70 (+6.10, +16.22%) | 23.60 (+0.80, +3.51%) | 28.80 (+1.20, +4.35%) | 31.50 (+0.70, +2.27%) | 38.60 (+0.40, +1.05%) | 47.80 (+2.20, +4.82%) | 30.90 (+1.40, +4.75%) | 33.70 (+1.20, +3.69%) |
+| microsoft | 42.90 (+5.80, +15.63%) | 28.20 (+2.50, +9.73%) | 36.10 (+2.00, +5.87%) | 24.00 (+0.60, +2.56%) | 25.30 (+1.30, +5.42%) | 43.80 (+2.30, +5.54%) | 27.20 (+1.40, +5.43%) | 23.70 (+0.70, +3.04%) | 44.00 (+6.40, +17.02%) | 23.90 (+1.10, +4.82%) | 28.80 (+1.20, +4.35%) | 33.10 (+2.30, +7.47%) | 40.50 (+2.30, +6.02%) | 48.70 (+3.10, +6.80%) | 32.20 (+2.70, +9.15%) | 34.30 (+1.80, +5.54%) |
 
 ![Results](img/en-de.png)
diff --git a/models/dev/enpt/lex.50.50.enpt.s2t.bin.gz b/models/dev/enpt/lex.50.50.enpt.s2t.bin.gz
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6e64062e58fa26a5f50b185fe28886b6c94d3bc89ba6ddc84e05b1665a92b4e0
+size 2411984
diff --git a/models/dev/enpt/model.enpt.intgemm.alphas.bin.gz b/models/dev/enpt/model.enpt.intgemm.alphas.bin.gz
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5075d4d30f2b0f7d81ced65d091d4acc9b26fa915129123b41f4189f797e0d0c
+size 13429592
diff --git a/models/dev/enpt/vocab.enpt.spm.gz b/models/dev/enpt/vocab.enpt.spm.gz
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac1d1efab07d5e6dca53a576deafd4d2c67a3ac19dcda38b685e787cc8dbb637
+size 406524

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+version https://git-lfs.github.com/spec/v1`
	`2`	`+oid sha256:6e64062e58fa26a5f50b185fe28886b6c94d3bc89ba6ddc84e05b1665a92b4e0`
	`3`	`+size 2411984`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+version https://git-lfs.github.com/spec/v1`
	`2`	`+oid sha256:5075d4d30f2b0f7d81ced65d091d4acc9b26fa915129123b41f4189f797e0d0c`
	`3`	`+size 13429592`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+version https://git-lfs.github.com/spec/v1`
	`2`	`+oid sha256:ac1d1efab07d5e6dca53a576deafd4d2c67a3ac19dcda38b685e787cc8dbb637`
	`3`	`+size 406524`