Update Italian -> English (#40)

* Update iten * Update evaluation results [skip ci] * Add git pull to ci * Move iten to prod * Update evaluation results [skip ci] * Fix git pull * Fix readme * Restart ci * Update evaluation results [skip ci] * Update model registry [skip ci] Co-authored-by: CircleCI evaluation job <ci-models-evaluation@firefox-translations>
mozilla · Mar 25, 2022 · e0b29e5 · e0b29e5
1 parent 8fc518d
commit e0b29e5
Show file tree

Hide file tree

Showing 31 changed files with 63 additions and 60 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -87,6 +87,7 @@ jobs:
           command: |
             git config user.email "ci-models-evaluation@firefox-translations"
             git config user.name "CircleCI evaluation job"
+            git pull origin "$CIRCLE_BRANCH"
             git add registry.json
 
             if [[ $(git status --porcelain) ]]; then

diff --git a/README.md b/README.md
@@ -7,7 +7,7 @@ The model files are hosted using [Git LFS](https://docs.github.com/en/github/man
 
 [dev](models/dev) - test models under development (can be of low quality or speed). 
 
-When a dev model is tested by humans in the extension and has satisfactory quality, it is moved to prod.
+When a dev model has satisfactory quality, it is moved to prod.
 
 # Automatic quality evaluation
 
@@ -62,10 +62,12 @@ The models will be automatically uploaded to GCS bucket `gs://bergamot-models-sa
 - Czech <-> English
 - Bulgarian <-> English
 - Norwegian Bokmål -> English
+- Portuguese -> English
+- Italian -> English
 ## Dev
 - Russian <-> English
-- Portuguese <-> English
-- Italian <-> English
+- Portuguese <- English
+- Italian <- English
 - Persian (Farsi) <-> English
 - Icelandic -> English
 - Norwegian Nynorsk -> English

diff --git a/evaluation/dev/img/avg.png b/evaluation/dev/img/avg.png
diff --git a/evaluation/dev/img/it-en.png b/evaluation/dev/img/it-en.png
diff --git a/evaluation/dev/it-en/flores-dev.bergamot.en.bleu b/evaluation/dev/it-en/flores-dev.bergamot.en.bleu
diff --git a/evaluation/dev/it-en/flores-test.bergamot.en.bleu b/evaluation/dev/it-en/flores-test.bergamot.en.bleu
diff --git a/evaluation/dev/it-en/mtedx_test.bergamot.en.bleu b/evaluation/dev/it-en/mtedx_test.bergamot.en.bleu
diff --git a/evaluation/dev/it-en/wmt09.bergamot.en.bleu b/evaluation/dev/it-en/wmt09.bergamot.en.bleu
diff --git a/evaluation/dev/results.md b/evaluation/dev/results.md
@@ -57,11 +57,11 @@ Both absolute and relative differences in BLEU scores between Bergamot and other
 
 ## avg
 
-| Translator/Dataset | en-it | en-pt | ru-en | en-ru | en-fa | fa-en | is-en | it-en |
-| --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| bergamot | 29.53 | 46.55 | 33.37 | 30.47 | 17.30 | 28.70 | 23.50 | 32.30 |
-| google | 29.47 (-0.07, -0.23%) | 56.05 (+9.50, +20.41%) | 36.53 (+3.15, +9.45%) | 33.72 (+3.25, +10.67%) | 27.70 (+10.40, +60.12%) | 36.05 (+7.35, +25.61%) | 34.95 (+11.45, +48.72%) | 34.50 (+2.20, +6.81%) |
-| microsoft | 32.20 (+2.67, +9.03%) | 50.25 (+3.70, +7.95%) | 36.87 (+3.50, +10.49%) | 33.68 (+3.21, +10.53%) | 20.50 (+3.20, +18.50%) | 36.15 (+7.45, +25.96%) | 34.90 (+11.40, +48.51%) | 34.55 (+2.25, +6.97%) |
+| Translator/Dataset | en-it | en-pt | ru-en | en-ru | en-fa | fa-en | is-en |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| bergamot | 29.53 | 46.55 | 33.37 | 30.47 | 17.30 | 28.70 | 23.50 |
+| google | 29.47 (-0.07, -0.23%) | 56.05 (+9.50, +20.41%) | 36.53 (+3.15, +9.45%) | 33.72 (+3.25, +10.67%) | 27.70 (+10.40, +60.12%) | 36.05 (+7.35, +25.61%) | 34.95 (+11.45, +48.72%) |
+| microsoft | 32.20 (+2.67, +9.03%) | 50.25 (+3.70, +7.95%) | 36.87 (+3.50, +10.49%) | 33.68 (+3.21, +10.53%) | 20.50 (+3.20, +18.50%) | 36.15 (+7.45, +25.96%) | 34.90 (+11.40, +48.51%) |
 
 ![Results](img/avg.png)
 
@@ -133,14 +133,4 @@ Both absolute and relative differences in BLEU scores between Bergamot and other
 | google | 35.50 (+11.90, +50.42%) | 34.40 (+11.00, +47.01%) |
 | microsoft | 35.10 (+11.50, +48.73%) | 34.70 (+11.30, +48.29%) |
 
-![Results](img/is-en.png)
-
-## it-en
-
-| Translator/Dataset | mtedx_test | flores-dev | flores-test | wmt09 |
-| --- | --- | --- | --- | --- |
-| bergamot | 35.50 | 30.70 | 29.90 | 33.10 |
-| google | 35.90 (+0.40, +1.13%) | 33.40 (+2.70, +8.79%) | 33.30 (+3.40, +11.37%) | 35.40 (+2.30, +6.95%) |
-| microsoft | 36.40 (+0.90, +2.54%) | 33.30 (+2.60, +8.47%) | 32.70 (+2.80, +9.36%) | 35.80 (+2.70, +8.16%) |
-
-![Results](img/it-en.png)
+![Results](img/is-en.png)
diff --git a/evaluation/prod/img/avg.png b/evaluation/prod/img/avg.png
diff --git a/evaluation/prod/img/it-en.png b/evaluation/prod/img/it-en.png
diff --git a/evaluation/prod/it-en/flores-dev.bergamot.en.bleu b/evaluation/prod/it-en/flores-dev.bergamot.en.bleu
@@ -0,0 +1 @@
+31.1
diff --git a/...ation/dev/it-en/flores-dev.google.en.bleu → ...tion/prod/it-en/flores-dev.google.en.bleu b/...ation/dev/it-en/flores-dev.google.en.bleu → ...tion/prod/it-en/flores-dev.google.en.bleu
diff --git a/...on/dev/it-en/flores-dev.microsoft.en.bleu → ...n/prod/it-en/flores-dev.microsoft.en.bleu b/...on/dev/it-en/flores-dev.microsoft.en.bleu → ...n/prod/it-en/flores-dev.microsoft.en.bleu
diff --git a/evaluation/prod/it-en/flores-test.bergamot.en.bleu b/evaluation/prod/it-en/flores-test.bergamot.en.bleu
@@ -0,0 +1 @@
+30.4
diff --git a/...tion/dev/it-en/flores-test.google.en.bleu → ...ion/prod/it-en/flores-test.google.en.bleu b/...tion/dev/it-en/flores-test.google.en.bleu → ...ion/prod/it-en/flores-test.google.en.bleu
diff --git a/...n/dev/it-en/flores-test.microsoft.en.bleu → .../prod/it-en/flores-test.microsoft.en.bleu b/...n/dev/it-en/flores-test.microsoft.en.bleu → .../prod/it-en/flores-test.microsoft.en.bleu
diff --git a/evaluation/prod/it-en/mtedx_test.bergamot.en.bleu b/evaluation/prod/it-en/mtedx_test.bergamot.en.bleu
@@ -0,0 +1 @@
+35.7
diff --git a/...ation/dev/it-en/mtedx_test.google.en.bleu → ...tion/prod/it-en/mtedx_test.google.en.bleu b/...ation/dev/it-en/mtedx_test.google.en.bleu → ...tion/prod/it-en/mtedx_test.google.en.bleu
diff --git a/...on/dev/it-en/mtedx_test.microsoft.en.bleu → ...n/prod/it-en/mtedx_test.microsoft.en.bleu b/...on/dev/it-en/mtedx_test.microsoft.en.bleu → ...n/prod/it-en/mtedx_test.microsoft.en.bleu
diff --git a/evaluation/prod/it-en/wmt09.bergamot.en.bleu b/evaluation/prod/it-en/wmt09.bergamot.en.bleu
@@ -0,0 +1 @@
+33.5
diff --git a/evaluation/dev/it-en/wmt09.google.en.bleu → evaluation/prod/it-en/wmt09.google.en.bleu b/evaluation/dev/it-en/wmt09.google.en.bleu → evaluation/prod/it-en/wmt09.google.en.bleu
diff --git a/evaluation/dev/it-en/wmt09.microsoft.en.bleu → ...uation/prod/it-en/wmt09.microsoft.en.bleu b/evaluation/dev/it-en/wmt09.microsoft.en.bleu → ...uation/prod/it-en/wmt09.microsoft.en.bleu
diff --git a/evaluation/prod/results.md b/evaluation/prod/results.md
@@ -57,11 +57,11 @@ Both absolute and relative differences in BLEU scores between Bergamot and other
 
 ## avg
 
-| Translator/Dataset | cs-en | en-et | et-en | nb-en | bg-en | en-es | en-bg | en-cs | de-en | pt-en | es-en | en-de |
-| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| bergamot | 30.34 | 25.50 | 32.37 | 37.60 | 38.50 | 32.41 | 42.10 | 24.65 | 33.51 | 44.87 | 32.38 | 32.27 |
-| google | 32.40 (+2.06, +6.80%) | 28.60 (+3.10, +12.16%) | 35.80 (+3.43, +10.61%) | 42.05 (+4.45, +11.84%) | 41.30 (+2.80, +7.27%) | 34.74 (+2.32, +7.17%) | 44.60 (+2.50, +5.94%) | 26.73 (+2.09, +8.47%) | 35.98 (+2.48, +7.39%) | 46.60 (+1.73, +3.86%) | 33.64 (+1.27, +3.91%) | 33.05 (+0.77, +2.40%) |
-| microsoft | 33.53 (+3.19, +10.53%) | 28.47 (+2.97, +11.63%) | 36.17 (+3.80, +11.74%) | 42.90 (+5.30, +14.10%) | 41.20 (+2.70, +7.01%) | 33.76 (+1.35, +4.17%) | 38.55 (-3.55, -8.43%) | 27.75 (+3.11, +12.60%) | 38.21 (+4.70, +14.03%) | 46.47 (+1.60, +3.57%) | 32.93 (+0.56, +1.72%) | 33.54 (+1.27, +3.93%) |
+| Translator/Dataset | cs-en | en-et | et-en | nb-en | bg-en | en-es | en-bg | en-cs | de-en | it-en | pt-en | es-en | en-de |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| bergamot | 30.34 | 25.50 | 32.37 | 37.60 | 38.50 | 32.41 | 42.10 | 24.65 | 33.51 | 32.67 | 44.87 | 32.38 | 32.27 |
+| google | 32.40 (+2.06, +6.80%) | 28.60 (+3.10, +12.16%) | 35.80 (+3.43, +10.61%) | 42.05 (+4.45, +11.84%) | 41.30 (+2.80, +7.27%) | 34.74 (+2.32, +7.17%) | 44.60 (+2.50, +5.94%) | 26.73 (+2.09, +8.47%) | 35.98 (+2.48, +7.39%) | 34.50 (+1.83, +5.59%) | 46.60 (+1.73, +3.86%) | 33.64 (+1.27, +3.91%) | 33.05 (+0.77, +2.40%) |
+| microsoft | 33.53 (+3.19, +10.53%) | 28.47 (+2.97, +11.63%) | 36.17 (+3.80, +11.74%) | 42.90 (+5.30, +14.10%) | 41.20 (+2.70, +7.01%) | 33.76 (+1.35, +4.17%) | 38.55 (-3.55, -8.43%) | 27.75 (+3.11, +12.60%) | 38.21 (+4.70, +14.03%) | 34.55 (+1.88, +5.74%) | 46.47 (+1.60, +3.57%) | 32.93 (+0.56, +1.72%) | 33.54 (+1.27, +3.93%) |
 
 ![Results](img/avg.png)
 
@@ -155,6 +155,16 @@ Both absolute and relative differences in BLEU scores between Bergamot and other
 
 ![Results](img/de-en.png)
 
+## it-en
+
+| Translator/Dataset | mtedx_test | flores-dev | flores-test | wmt09 |
+| --- | --- | --- | --- | --- |
+| bergamot | 35.70 | 31.10 | 30.40 | 33.50 |
+| google | 35.90 (+0.20, +0.56%) | 33.40 (+2.30, +7.40%) | 33.30 (+2.90, +9.54%) | 35.40 (+1.90, +5.67%) |
+| microsoft | 36.40 (+0.70, +1.96%) | 33.30 (+2.20, +7.07%) | 32.70 (+2.30, +7.57%) | 35.80 (+2.30, +6.87%) |
+
+![Results](img/it-en.png)
+
 ## pt-en
 
 | Translator/Dataset | mtedx_test | flores-dev | flores-test |

diff --git a/models/dev/iten/lex.50.50.iten.s2t.bin.gz b/models/dev/iten/lex.50.50.iten.s2t.bin.gz
diff --git a/models/dev/iten/model.iten.intgemm.alphas.bin.gz b/models/dev/iten/model.iten.intgemm.alphas.bin.gz
diff --git a/models/dev/iten/vocab.iten.spm.gz b/models/dev/iten/vocab.iten.spm.gz
diff --git a/models/prod/iten/lex.50.50.iten.s2t.bin.gz b/models/prod/iten/lex.50.50.iten.s2t.bin.gz
diff --git a/models/prod/iten/model.iten.intgemm.alphas.bin.gz b/models/prod/iten/model.iten.intgemm.alphas.bin.gz
diff --git a/models/prod/iten/vocab.iten.spm.gz b/models/prod/iten/vocab.iten.spm.gz
diff --git a/registry.json b/registry.json
@@ -250,6 +250,29 @@
       "modelType": "prod"
     }
   },
+  "iten": {
+    "model": {
+      "name": "model.iten.intgemm.alphas.bin",
+      "size": 17140899,
+      "estimatedCompressedSize": 13220889,
+      "expectedSha256Hash": "7dfdf189146d9353fdea264b9e4c8ac36441c770dc4353a8380b64e589dc035b",
+      "modelType": "prod"
+    },
+    "lex": {
+      "name": "lex.50.50.iten.s2t.bin",
+      "size": 4977500,
+      "estimatedCompressedSize": 2743096,
+      "expectedSha256Hash": "e30ec549bd0da9ac42cccdcd3806d3be84d485f7fd329f90f6e40ee027e841d9",
+      "modelType": "prod"
+    },
+    "vocab": {
+      "name": "vocab.iten.spm",
+      "size": 812781,
+      "estimatedCompressedSize": 408834,
+      "expectedSha256Hash": "603f3349657c3deb9736a0c567452d102a5a03c377dfdf1d32c428608f2cff1b",
+      "modelType": "prod"
+    }
+  },
   "nben": {
     "model": {
       "name": "model.nben.intgemm.alphas.bin",
@@ -434,29 +457,6 @@
       "modelType": "dev"
     }
   },
-  "iten": {
-    "model": {
-      "name": "model.iten.intgemm.alphas.bin",
-      "size": 17140836,
-      "estimatedCompressedSize": 13423308,
-      "expectedSha256Hash": "1fae546faeb9046f80b1b7e940b37b660974ce72902778181d6cd1c30b717f35",
-      "modelType": "dev"
-    },
-    "lex": {
-      "name": "lex.50.50.iten.s2t.bin",
-      "size": 5238420,
-      "estimatedCompressedSize": 2860178,
-      "expectedSha256Hash": "357d362373022b029ee9965975a133e6f36fdb0fed749202ff578365cf0111f8",
-      "modelType": "dev"
-    },
-    "vocab": {
-      "name": "vocab.iten.spm",
-      "size": 814151,
-      "estimatedCompressedSize": 405416,
-      "expectedSha256Hash": "22d5ce6973be5360a921103acbe984a9bfca952a1f6c55c9cb5ef7de4fd58266",
-      "modelType": "dev"
-    }
-  },
   "nnen": {
     "model": {
       "name": "model.nnen.intgemm.alphas.bin",