Update pten and move to prod (#39)

* Update pten and move to prod * Update model registry [skip ci] * Fix ci workflow * Update evaluation results [skip ci] Co-authored-by: CircleCI evaluation job <ci-models-evaluation@firefox-translations>
mozilla · Mar 16, 2022 · 8fc518d · 8fc518d
1 parent f22092f
commit 8fc518d
Show file tree

Hide file tree

Showing 43 changed files with 171 additions and 167 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -102,12 +102,6 @@ workflows:
   version: 2
   ci:
     jobs:
-      - deploy:
-          filters:
-            branches:
-              ignore: /.*/
-            tags:
-              only: /\d*\.\d*\.\d*/
       - evaluate:
           filters:
             branches:
@@ -116,9 +110,19 @@ workflows:
                 - /pull\/[0-9]+/
                 - main
       - generate-registry:
+          requires:
+            - evaluate
           filters:
             branches:
               # Forked pull requests have CIRCLE_BRANCH set to pull/XXX
               ignore:
                 - /pull\/[0-9]+/
                 - main
+  publish:
+    jobs:
+      - deploy:
+          filters:
+            branches:
+              ignore: /.*/
+            tags:
+              only: /\d*\.\d*\.\d*/
diff --git a/evaluation/dev/img/avg.png b/evaluation/dev/img/avg.png
diff --git a/evaluation/dev/img/en-fa.png b/evaluation/dev/img/en-fa.png
diff --git a/evaluation/dev/img/en-it.png b/evaluation/dev/img/en-it.png
diff --git a/evaluation/dev/img/en-ru.png b/evaluation/dev/img/en-ru.png
diff --git a/evaluation/dev/img/it-en.png b/evaluation/dev/img/it-en.png
diff --git a/evaluation/dev/img/ru-en.png b/evaluation/dev/img/ru-en.png
diff --git a/evaluation/dev/pt-en/flores-dev.bergamot.en.bleu b/evaluation/dev/pt-en/flores-dev.bergamot.en.bleu
diff --git a/evaluation/dev/pt-en/flores-dev.google.en.bleu b/evaluation/dev/pt-en/flores-dev.google.en.bleu
diff --git a/evaluation/dev/pt-en/flores-test.bergamot.en.bleu b/evaluation/dev/pt-en/flores-test.bergamot.en.bleu
diff --git a/evaluation/dev/pt-en/flores-test.google.en.bleu b/evaluation/dev/pt-en/flores-test.google.en.bleu
diff --git a/evaluation/dev/pt-en/mtedx_test.bergamot.en.bleu b/evaluation/dev/pt-en/mtedx_test.bergamot.en.bleu
diff --git a/evaluation/dev/pt-en/mtedx_test.google.en.bleu b/evaluation/dev/pt-en/mtedx_test.google.en.bleu
diff --git a/evaluation/dev/pt-en/mtedx_test.microsoft.en.bleu b/evaluation/dev/pt-en/mtedx_test.microsoft.en.bleu
diff --git a/evaluation/dev/results.md b/evaluation/dev/results.md
@@ -57,93 +57,73 @@ Both absolute and relative differences in BLEU scores between Bergamot and other
 
 ## avg
 
-| Translator/Dataset | fa-en | pt-en | ru-en | it-en | en-ru | en-it | en-fa | en-pt | is-en |
-| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| bergamot | 28.70 | 43.43 | 33.37 | 32.30 | 30.47 | 29.53 | 17.30 | 46.55 | 23.50 |
-| google | 36.05 (+7.35, +25.61%) | 45.77 (+2.33, +5.37%) | 36.53 (+3.15, +9.45%) | 34.50 (+2.20, +6.81%) | 33.72 (+3.25, +10.67%) | 29.47 (-0.07, -0.23%) | 27.70 (+10.40, +60.12%) | 56.05 (+9.50, +20.41%) | 34.95 (+11.45, +48.72%) |
-| microsoft | 36.15 (+7.45, +25.96%) | 46.43 (+3.00, +6.91%) | 36.87 (+3.50, +10.49%) | 34.55 (+2.25, +6.97%) | 33.68 (+3.21, +10.53%) | 32.20 (+2.67, +9.03%) | 20.50 (+3.20, +18.50%) | 50.25 (+3.70, +7.95%) | 34.90 (+11.40, +48.51%) |
+| Translator/Dataset | en-it | en-pt | ru-en | en-ru | en-fa | fa-en | is-en | it-en |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| bergamot | 29.53 | 46.55 | 33.37 | 30.47 | 17.30 | 28.70 | 23.50 | 32.30 |
+| google | 29.47 (-0.07, -0.23%) | 56.05 (+9.50, +20.41%) | 36.53 (+3.15, +9.45%) | 33.72 (+3.25, +10.67%) | 27.70 (+10.40, +60.12%) | 36.05 (+7.35, +25.61%) | 34.95 (+11.45, +48.72%) | 34.50 (+2.20, +6.81%) |
+| microsoft | 32.20 (+2.67, +9.03%) | 50.25 (+3.70, +7.95%) | 36.87 (+3.50, +10.49%) | 33.68 (+3.21, +10.53%) | 20.50 (+3.20, +18.50%) | 36.15 (+7.45, +25.96%) | 34.90 (+11.40, +48.51%) | 34.55 (+2.25, +6.97%) |
 
 ![Results](img/avg.png)
 
-## fa-en
+## en-it
 
-| Translator/Dataset | flores-dev | flores-test |
-| --- | --- | --- |
-| bergamot | 29.10 | 28.30 |
-| google | 36.70 (+7.60, +26.12%) | 35.40 (+7.10, +25.09%) |
-| microsoft | 36.50 (+7.40, +25.43%) | 35.80 (+7.50, +26.50%) |
+| Translator/Dataset | flores-test | flores-dev | wmt09 |
+| --- | --- | --- | --- |
+| bergamot | 29.20 | 28.40 | 31.00 |
+| google | 30.10 (+0.90, +3.08%) | 29.00 (+0.60, +2.11%) | 29.30 (-1.70, -5.48%) |
+| microsoft | 32.00 (+2.80, +9.59%) | 31.00 (+2.60, +9.15%) | 33.60 (+2.60, +8.39%) |
 
-![Results](img/fa-en.png)
+![Results](img/en-it.png)
 
-## pt-en
+## en-pt
 
-| Translator/Dataset | mtedx_test | flores-dev | flores-test |
-| --- | --- | --- | --- |
-| bergamot | 39.70 | 45.90 | 44.70 |
-| google | 38.90 (-0.80, -2.02%) | 49.50 (+3.60, +7.84%) | 48.90 (+4.20, +9.40%) |
-| microsoft | 40.90 (+1.20, +3.02%) | 49.80 (+3.90, +8.50%) | 48.60 (+3.90, +8.72%) |
+| Translator/Dataset | flores-test | flores-dev |
+| --- | --- | --- |
+| bergamot | 46.20 | 46.90 |
+| google | 55.70 (+9.50, +20.56%) | 56.40 (+9.50, +20.26%) |
+| microsoft | 50.70 (+4.50, +9.74%) | 49.80 (+2.90, +6.18%) |
 
-![Results](img/pt-en.png)
+![Results](img/en-pt.png)
 
 ## ru-en
 
-| Translator/Dataset | wmt17 | wmt18 | wmt14 | mtedx_test | wmt16 | wmt13 | wmt19 | wmt20 | flores-dev | flores-test | wmt15 |
+| Translator/Dataset | mtedx_test | wmt19 | wmt17 | flores-dev | flores-test | wmt14 | wmt15 | wmt16 | wmt13 | wmt18 | wmt20 |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| bergamot | 37.90 | 32.30 | 38.00 | 24.20 | 33.40 | 29.50 | 39.30 | 35.40 | 31.90 | 31.50 | 33.70 |
-| google | 41.50 (+3.60, +9.50%) | 36.00 (+3.70, +11.46%) | 41.20 (+3.20, +8.42%) | 25.00 (+0.80, +3.31%) | 36.60 (+3.20, +9.58%) | 31.40 (+1.90, +6.44%) | 42.40 (+3.10, +7.89%) | 37.70 (+2.30, +6.50%) | 37.00 (+5.10, +15.99%) | 35.50 (+4.00, +12.70%) | 37.50 (+3.80, +11.28%) |
-| microsoft | 41.60 (+3.70, +9.76%) | 36.90 (+4.60, +14.24%) | 41.70 (+3.70, +9.74%) | 26.10 (+1.90, +7.85%) | 37.60 (+4.20, +12.57%) | 31.20 (+1.70, +5.76%) | 42.60 (+3.30, +8.40%) | 37.80 (+2.40, +6.78%) | 36.20 (+4.30, +13.48%) | 36.10 (+4.60, +14.60%) | 37.80 (+4.10, +12.17%) |
+| bergamot | 24.20 | 39.30 | 37.90 | 31.90 | 31.50 | 38.00 | 33.70 | 33.40 | 29.50 | 32.30 | 35.40 |
+| google | 25.00 (+0.80, +3.31%) | 42.40 (+3.10, +7.89%) | 41.50 (+3.60, +9.50%) | 37.00 (+5.10, +15.99%) | 35.50 (+4.00, +12.70%) | 41.20 (+3.20, +8.42%) | 37.50 (+3.80, +11.28%) | 36.60 (+3.20, +9.58%) | 31.40 (+1.90, +6.44%) | 36.00 (+3.70, +11.46%) | 37.70 (+2.30, +6.50%) |
+| microsoft | 26.10 (+1.90, +7.85%) | 42.60 (+3.30, +8.40%) | 41.60 (+3.70, +9.76%) | 36.20 (+4.30, +13.48%) | 36.10 (+4.60, +14.60%) | 41.70 (+3.70, +9.74%) | 37.80 (+4.10, +12.17%) | 37.60 (+4.20, +12.57%) | 31.20 (+1.70, +5.76%) | 36.90 (+4.60, +14.24%) | 37.80 (+2.40, +6.78%) |
 
 ![Results](img/ru-en.png)
 
-## it-en
-
-| Translator/Dataset | wmt09 | mtedx_test | flores-dev | flores-test |
-| --- | --- | --- | --- | --- |
-| bergamot | 33.10 | 35.50 | 30.70 | 29.90 |
-| google | 35.40 (+2.30, +6.95%) | 35.90 (+0.40, +1.13%) | 33.40 (+2.70, +8.79%) | 33.30 (+3.40, +11.37%) |
-| microsoft | 35.80 (+2.70, +8.16%) | 36.40 (+0.90, +2.54%) | 33.30 (+2.60, +8.47%) | 32.70 (+2.80, +9.36%) |
-
-![Results](img/it-en.png)
-
 ## en-ru
 
-| Translator/Dataset | wmt15 | flores-dev | wmt13 | wmt16 | wmt20 | flores-test | wmt19 | wmt17 | wmt18 | wmt14 |
+| Translator/Dataset | wmt16 | wmt15 | flores-dev | wmt18 | wmt14 | wmt17 | wmt20 | wmt13 | wmt19 | flores-test |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| bergamot | 31.80 | 30.20 | 26.70 | 31.60 | 22.00 | 29.10 | 31.70 | 34.10 | 29.00 | 38.50 |
-| google | 35.00 (+3.20, +10.06%) | 33.60 (+3.40, +11.26%) | 27.60 (+0.90, +3.37%) | 34.00 (+2.40, +7.59%) | 26.40 (+4.40, +20.00%) | 33.60 (+4.50, +15.46%) | 32.20 (+0.50, +1.58%) | 37.30 (+3.20, +9.38%) | 34.10 (+5.10, +17.59%) | 43.40 (+4.90, +12.73%) |
-| microsoft | 35.60 (+3.80, +11.95%) | 33.20 (+3.00, +9.93%) | 27.10 (+0.40, +1.50%) | 33.80 (+2.20, +6.96%) | 26.00 (+4.00, +18.18%) | 33.00 (+3.90, +13.40%) | 32.70 (+1.00, +3.15%) | 38.10 (+4.00, +11.73%) | 33.20 (+4.20, +14.48%) | 44.10 (+5.60, +14.55%) |
+| bergamot | 31.60 | 31.80 | 30.20 | 29.00 | 38.50 | 34.10 | 22.00 | 26.70 | 31.70 | 29.10 |
+| google | 34.00 (+2.40, +7.59%) | 35.00 (+3.20, +10.06%) | 33.60 (+3.40, +11.26%) | 34.10 (+5.10, +17.59%) | 43.40 (+4.90, +12.73%) | 37.30 (+3.20, +9.38%) | 26.40 (+4.40, +20.00%) | 27.60 (+0.90, +3.37%) | 32.20 (+0.50, +1.58%) | 33.60 (+4.50, +15.46%) |
+| microsoft | 33.80 (+2.20, +6.96%) | 35.60 (+3.80, +11.95%) | 33.20 (+3.00, +9.93%) | 33.20 (+4.20, +14.48%) | 44.10 (+5.60, +14.55%) | 38.10 (+4.00, +11.73%) | 26.00 (+4.00, +18.18%) | 27.10 (+0.40, +1.50%) | 32.70 (+1.00, +3.15%) | 33.00 (+3.90, +13.40%) |
 
 ![Results](img/en-ru.png)
 
-## en-it
-
-| Translator/Dataset | flores-dev | flores-test | wmt09 |
-| --- | --- | --- | --- |
-| bergamot | 28.40 | 29.20 | 31.00 |
-| google | 29.00 (+0.60, +2.11%) | 30.10 (+0.90, +3.08%) | 29.30 (-1.70, -5.48%) |
-| microsoft | 31.00 (+2.60, +9.15%) | 32.00 (+2.80, +9.59%) | 33.60 (+2.60, +8.39%) |
-
-![Results](img/en-it.png)
-
 ## en-fa
 
-| Translator/Dataset | flores-dev | flores-test |
+| Translator/Dataset | flores-test | flores-dev |
 | --- | --- | --- |
-| bergamot | 17.20 | 17.40 |
-| google | 27.20 (+10.00, +58.14%) | 28.20 (+10.80, +62.07%) |
-| microsoft | 19.90 (+2.70, +15.70%) | 21.10 (+3.70, +21.26%) |
+| bergamot | 17.40 | 17.20 |
+| google | 28.20 (+10.80, +62.07%) | 27.20 (+10.00, +58.14%) |
+| microsoft | 21.10 (+3.70, +21.26%) | 19.90 (+2.70, +15.70%) |
 
 ![Results](img/en-fa.png)
 
-## en-pt
+## fa-en
 
-| Translator/Dataset | flores-test | flores-dev |
+| Translator/Dataset | flores-dev | flores-test |
 | --- | --- | --- |
-| bergamot | 46.20 | 46.90 |
-| google | 55.70 (+9.50, +20.56%) | 56.40 (+9.50, +20.26%) |
-| microsoft | 50.70 (+4.50, +9.74%) | 49.80 (+2.90, +6.18%) |
+| bergamot | 29.10 | 28.30 |
+| google | 36.70 (+7.60, +26.12%) | 35.40 (+7.10, +25.09%) |
+| microsoft | 36.50 (+7.40, +25.43%) | 35.80 (+7.50, +26.50%) |
 
-![Results](img/en-pt.png)
+![Results](img/fa-en.png)
 
 ## is-en
 
@@ -153,4 +133,14 @@ Both absolute and relative differences in BLEU scores between Bergamot and other
 | google | 35.50 (+11.90, +50.42%) | 34.40 (+11.00, +47.01%) |
 | microsoft | 35.10 (+11.50, +48.73%) | 34.70 (+11.30, +48.29%) |
 
-![Results](img/is-en.png)
+![Results](img/is-en.png)
+
+## it-en
+
+| Translator/Dataset | mtedx_test | flores-dev | flores-test | wmt09 |
+| --- | --- | --- | --- | --- |
+| bergamot | 35.50 | 30.70 | 29.90 | 33.10 |
+| google | 35.90 (+0.40, +1.13%) | 33.40 (+2.70, +8.79%) | 33.30 (+3.40, +11.37%) | 35.40 (+2.30, +6.95%) |
+| microsoft | 36.40 (+0.90, +2.54%) | 33.30 (+2.60, +8.47%) | 32.70 (+2.80, +9.36%) | 35.80 (+2.70, +8.16%) |
+
+![Results](img/it-en.png)
diff --git a/evaluation/prod/img/avg.png b/evaluation/prod/img/avg.png
diff --git a/evaluation/prod/img/cs-en.png b/evaluation/prod/img/cs-en.png
diff --git a/evaluation/prod/img/de-en.png b/evaluation/prod/img/de-en.png
diff --git a/evaluation/prod/img/en-bg.png b/evaluation/prod/img/en-bg.png
diff --git a/evaluation/prod/img/en-cs.png b/evaluation/prod/img/en-cs.png
diff --git a/evaluation/prod/img/en-de.png b/evaluation/prod/img/en-de.png
diff --git a/evaluation/prod/img/en-es.png b/evaluation/prod/img/en-es.png
diff --git a/evaluation/prod/img/en-et.png b/evaluation/prod/img/en-et.png
diff --git a/evaluation/prod/img/es-en.png b/evaluation/prod/img/es-en.png
diff --git a/evaluation/prod/img/et-en.png b/evaluation/prod/img/et-en.png
diff --git a/evaluation/prod/img/pt-en.png b/evaluation/prod/img/pt-en.png
diff --git a/evaluation/prod/pt-en/flores-dev.bergamot.en.bleu b/evaluation/prod/pt-en/flores-dev.bergamot.en.bleu
@@ -0,0 +1 @@
+47.8
diff --git a/evaluation/prod/pt-en/flores-dev.google.en.bleu b/evaluation/prod/pt-en/flores-dev.google.en.bleu
@@ -0,0 +1 @@
+50.4
diff --git a/...on/dev/pt-en/flores-dev.microsoft.en.bleu → ...n/prod/pt-en/flores-dev.microsoft.en.bleu b/...on/dev/pt-en/flores-dev.microsoft.en.bleu → ...n/prod/pt-en/flores-dev.microsoft.en.bleu
diff --git a/evaluation/prod/pt-en/flores-test.bergamot.en.bleu b/evaluation/prod/pt-en/flores-test.bergamot.en.bleu
@@ -0,0 +1 @@
+46.6
diff --git a/evaluation/prod/pt-en/flores-test.google.en.bleu b/evaluation/prod/pt-en/flores-test.google.en.bleu
@@ -0,0 +1 @@
+50.3
diff --git a/...n/dev/pt-en/flores-test.microsoft.en.bleu → .../prod/pt-en/flores-test.microsoft.en.bleu b/...n/dev/pt-en/flores-test.microsoft.en.bleu → .../prod/pt-en/flores-test.microsoft.en.bleu
diff --git a/evaluation/prod/pt-en/mtedx_test.bergamot.en.bleu b/evaluation/prod/pt-en/mtedx_test.bergamot.en.bleu
@@ -0,0 +1 @@
+40.2
diff --git a/evaluation/prod/pt-en/mtedx_test.google.en.bleu b/evaluation/prod/pt-en/mtedx_test.google.en.bleu
@@ -0,0 +1 @@
+39.1
diff --git a/evaluation/prod/pt-en/mtedx_test.microsoft.en.bleu b/evaluation/prod/pt-en/mtedx_test.microsoft.en.bleu
@@ -0,0 +1 @@
+41.0