Add more languages (#31)

eu9ene · CircleCI evaluation job · web-flow · commit d4a4d3010779 · 2022-02-10T16:50:43.000-08:00
* Add statmt importing script

* Add Farsi models

* Add Icelandic -&gt; English

* Add Norwegian -&gt; English models

* Update statmt script

* Update readme

* Fix readme

* Update evaluation results [skip ci]

* Move models with bad quality to dev

* Move evaluation results to dev

* Update evaluation results [skip ci]

Co-authored-by: CircleCI evaluation job &lt;ci-models-evaluation@firefox-translations&gt;
diff --git a/README.md b/README.md
@@ -61,10 +61,14 @@ The models will be automatically uploaded to GCS bucket `gs://bergamot-models-sa
 - English <-> German
 - Czech <-> English
 - Bulgarian <-> English
+- Norwegian Bokmål -> English
 ## Dev
--  Russian <-> English
--  Portuguese <-> English
--  Italian <-> English
+- Russian <-> English
+- Portuguese <-> English
+- Italian <-> English
+- Persian (Farsi) <-> English
+- Icelandic -> English
+- Norwegian Nynorsk -> English
 ## Upcoming
 -  French <-> English
 -  Polish <-> English
diff --git a/evaluation/dev/en-fa/flores-dev.bergamot.fa.bleu b/evaluation/dev/en-fa/flores-dev.bergamot.fa.bleu
@@ -0,0 +1 @@
+17.2
diff --git a/evaluation/dev/en-fa/flores-dev.google.fa.bleu b/evaluation/dev/en-fa/flores-dev.google.fa.bleu
@@ -0,0 +1 @@
+27.2
diff --git a/evaluation/dev/en-fa/flores-dev.microsoft.fa.bleu b/evaluation/dev/en-fa/flores-dev.microsoft.fa.bleu
@@ -0,0 +1 @@
+19.9
diff --git a/evaluation/dev/en-fa/flores-test.bergamot.fa.bleu b/evaluation/dev/en-fa/flores-test.bergamot.fa.bleu
@@ -0,0 +1 @@
+17.4
diff --git a/evaluation/dev/en-fa/flores-test.google.fa.bleu b/evaluation/dev/en-fa/flores-test.google.fa.bleu
@@ -0,0 +1 @@
+28.2
diff --git a/evaluation/dev/en-fa/flores-test.microsoft.fa.bleu b/evaluation/dev/en-fa/flores-test.microsoft.fa.bleu
@@ -0,0 +1 @@
+21.1
diff --git a/evaluation/dev/fa-en/flores-dev.bergamot.en.bleu b/evaluation/dev/fa-en/flores-dev.bergamot.en.bleu
@@ -0,0 +1 @@
+29.1
diff --git a/evaluation/dev/fa-en/flores-dev.google.en.bleu b/evaluation/dev/fa-en/flores-dev.google.en.bleu
@@ -0,0 +1 @@
+36.7
diff --git a/evaluation/dev/fa-en/flores-dev.microsoft.en.bleu b/evaluation/dev/fa-en/flores-dev.microsoft.en.bleu
@@ -0,0 +1 @@
+36.5
diff --git a/evaluation/dev/fa-en/flores-test.bergamot.en.bleu b/evaluation/dev/fa-en/flores-test.bergamot.en.bleu
@@ -0,0 +1 @@
+28.3
diff --git a/evaluation/dev/fa-en/flores-test.google.en.bleu b/evaluation/dev/fa-en/flores-test.google.en.bleu
@@ -0,0 +1 @@
+35.4
diff --git a/evaluation/dev/fa-en/flores-test.microsoft.en.bleu b/evaluation/dev/fa-en/flores-test.microsoft.en.bleu
@@ -0,0 +1 @@
+35.8
diff --git a/evaluation/dev/img/avg.png b/evaluation/dev/img/avg.png
diff --git a/evaluation/dev/img/en-fa.png b/evaluation/dev/img/en-fa.png
diff --git a/evaluation/dev/img/fa-en.png b/evaluation/dev/img/fa-en.png
diff --git a/evaluation/dev/img/is-en.png b/evaluation/dev/img/is-en.png
diff --git a/evaluation/dev/is-en/flores-dev.bergamot.en.bleu b/evaluation/dev/is-en/flores-dev.bergamot.en.bleu
@@ -0,0 +1 @@
+23.6
diff --git a/evaluation/dev/is-en/flores-dev.google.en.bleu b/evaluation/dev/is-en/flores-dev.google.en.bleu
@@ -0,0 +1 @@
+35.5
diff --git a/evaluation/dev/is-en/flores-dev.microsoft.en.bleu b/evaluation/dev/is-en/flores-dev.microsoft.en.bleu
@@ -0,0 +1 @@
+35.1
diff --git a/evaluation/dev/is-en/flores-test.bergamot.en.bleu b/evaluation/dev/is-en/flores-test.bergamot.en.bleu
@@ -0,0 +1 @@
+23.4
diff --git a/evaluation/dev/is-en/flores-test.google.en.bleu b/evaluation/dev/is-en/flores-test.google.en.bleu
@@ -0,0 +1 @@
+34.4
diff --git a/evaluation/dev/is-en/flores-test.microsoft.en.bleu b/evaluation/dev/is-en/flores-test.microsoft.en.bleu
@@ -0,0 +1 @@
+34.7
diff --git a/evaluation/dev/results.md b/evaluation/dev/results.md
@@ -57,14 +57,24 @@ Both absolute and relative differences in BLEU scores between Bergamot and other
 
 ## avg
 
-| Translator/Dataset | pt-en | ru-en | it-en | en-ru | en-it | en-pt |
-| --- | --- | --- | --- | --- | --- | --- |
-| bergamot | 43.43 | 33.37 | 32.30 | 30.47 | 29.53 | 46.55 |
-| google | 45.77 (+2.33, +5.37%) | 36.53 (+3.15, +9.45%) | 34.50 (+2.20, +6.81%) | 33.72 (+3.25, +10.67%) | 29.47 (-0.07, -0.23%) | 56.05 (+9.50, +20.41%) |
-| microsoft | 46.43 (+3.00, +6.91%) | 36.87 (+3.50, +10.49%) | 34.55 (+2.25, +6.97%) | 33.68 (+3.21, +10.53%) | 32.20 (+2.67, +9.03%) | 50.25 (+3.70, +7.95%) |
+| Translator/Dataset | fa-en | pt-en | ru-en | it-en | en-ru | en-it | en-fa | en-pt | is-en |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| bergamot | 28.70 | 43.43 | 33.37 | 32.30 | 30.47 | 29.53 | 17.30 | 46.55 | 23.50 |
+| google | 36.05 (+7.35, +25.61%) | 45.77 (+2.33, +5.37%) | 36.53 (+3.15, +9.45%) | 34.50 (+2.20, +6.81%) | 33.72 (+3.25, +10.67%) | 29.47 (-0.07, -0.23%) | 27.70 (+10.40, +60.12%) | 56.05 (+9.50, +20.41%) | 34.95 (+11.45, +48.72%) |
+| microsoft | 36.15 (+7.45, +25.96%) | 46.43 (+3.00, +6.91%) | 36.87 (+3.50, +10.49%) | 34.55 (+2.25, +6.97%) | 33.68 (+3.21, +10.53%) | 32.20 (+2.67, +9.03%) | 20.50 (+3.20, +18.50%) | 50.25 (+3.70, +7.95%) | 34.90 (+11.40, +48.51%) |
 
 ![Results](img/avg.png)
 
+## fa-en
+
+| Translator/Dataset | flores-dev | flores-test |
+| --- | --- | --- |
+| bergamot | 29.10 | 28.30 |
+| google | 36.70 (+7.60, +26.12%) | 35.40 (+7.10, +25.09%) |
+| microsoft | 36.50 (+7.40, +25.43%) | 35.80 (+7.50, +26.50%) |
+
+![Results](img/fa-en.png)
+
 ## pt-en
 
 | Translator/Dataset | mtedx_test | flores-dev | flores-test |
@@ -115,6 +125,16 @@ Both absolute and relative differences in BLEU scores between Bergamot and other
 
 ![Results](img/en-it.png)
 
+## en-fa
+
+| Translator/Dataset | flores-dev | flores-test |
+| --- | --- | --- |
+| bergamot | 17.20 | 17.40 |
+| google | 27.20 (+10.00, +58.14%) | 28.20 (+10.80, +62.07%) |
+| microsoft | 19.90 (+2.70, +15.70%) | 21.10 (+3.70, +21.26%) |
+
+![Results](img/en-fa.png)
+
 ## en-pt
 
 | Translator/Dataset | flores-test | flores-dev |
@@ -123,4 +143,14 @@ Both absolute and relative differences in BLEU scores between Bergamot and other
 | google | 55.70 (+9.50, +20.56%) | 56.40 (+9.50, +20.26%) |
 | microsoft | 50.70 (+4.50, +9.74%) | 49.80 (+2.90, +6.18%) |
 
-![Results](img/en-pt.png)
+![Results](img/en-pt.png)
+
+## is-en
+
+| Translator/Dataset | flores-dev | flores-test |
+| --- | --- | --- |
+| bergamot | 23.60 | 23.40 |
+| google | 35.50 (+11.90, +50.42%) | 34.40 (+11.00, +47.01%) |
+| microsoft | 35.10 (+11.50, +48.73%) | 34.70 (+11.30, +48.29%) |
+
+![Results](img/is-en.png)
diff --git a/evaluation/prod/img/avg.png b/evaluation/prod/img/avg.png
diff --git a/evaluation/prod/img/nb-en.png b/evaluation/prod/img/nb-en.png
diff --git a/evaluation/prod/nb-en/flores-dev.bergamot.en.bleu b/evaluation/prod/nb-en/flores-dev.bergamot.en.bleu
@@ -0,0 +1 @@
+37.4
diff --git a/evaluation/prod/nb-en/flores-dev.google.en.bleu b/evaluation/prod/nb-en/flores-dev.google.en.bleu
@@ -0,0 +1 @@
+41.8
diff --git a/evaluation/prod/nb-en/flores-dev.microsoft.en.bleu b/evaluation/prod/nb-en/flores-dev.microsoft.en.bleu
@@ -0,0 +1 @@
+42.7
diff --git a/evaluation/prod/nb-en/flores-test.bergamot.en.bleu b/evaluation/prod/nb-en/flores-test.bergamot.en.bleu
@@ -0,0 +1 @@
+37.8
diff --git a/evaluation/prod/nb-en/flores-test.google.en.bleu b/evaluation/prod/nb-en/flores-test.google.en.bleu
@@ -0,0 +1 @@
+42.3
diff --git a/evaluation/prod/nb-en/flores-test.microsoft.en.bleu b/evaluation/prod/nb-en/flores-test.microsoft.en.bleu
@@ -0,0 +1 @@
+43.1
diff --git a/evaluation/prod/results.md b/evaluation/prod/results.md
@@ -57,11 +57,11 @@ Both absolute and relative differences in BLEU scores between Bergamot and other
 
 ## avg
 
-| Translator/Dataset | es-en | bg-en | et-en | en-cs | cs-en | de-en | en-es | en-et | en-bg | en-de |
-| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| bergamot | 32.38 | 38.50 | 32.37 | 24.65 | 30.34 | 33.51 | 32.41 | 25.50 | 42.10 | 32.27 |
-| google | 33.64 (+1.27, +3.91%) | 41.30 (+2.80, +7.27%) | 35.80 (+3.43, +10.61%) | 26.73 (+2.09, +8.47%) | 32.40 (+2.06, +6.80%) | 35.98 (+2.48, +7.39%) | 34.74 (+2.32, +7.17%) | 28.60 (+3.10, +12.16%) | 44.60 (+2.50, +5.94%) | 33.05 (+0.77, +2.40%) |
-| microsoft | 32.93 (+0.56, +1.72%) | 41.20 (+2.70, +7.01%) | 36.17 (+3.80, +11.74%) | 27.75 (+3.11, +12.60%) | 33.53 (+3.19, +10.53%) | 38.21 (+4.70, +14.03%) | 33.76 (+1.35, +4.17%) | 28.47 (+2.97, +11.63%) | 38.55 (-3.55, -8.43%) | 33.54 (+1.27, +3.93%) |
+| Translator/Dataset | es-en | nb-en | bg-en | et-en | en-cs | cs-en | de-en | en-es | en-et | en-bg | en-de |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| bergamot | 32.38 | 37.60 | 38.50 | 32.37 | 24.65 | 30.34 | 33.51 | 32.41 | 25.50 | 42.10 | 32.27 |
+| google | 33.64 (+1.27, +3.91%) | 42.05 (+4.45, +11.84%) | 41.30 (+2.80, +7.27%) | 35.80 (+3.43, +10.61%) | 26.73 (+2.09, +8.47%) | 32.40 (+2.06, +6.80%) | 35.98 (+2.48, +7.39%) | 34.74 (+2.32, +7.17%) | 28.60 (+3.10, +12.16%) | 44.60 (+2.50, +5.94%) | 33.05 (+0.77, +2.40%) |
+| microsoft | 32.93 (+0.56, +1.72%) | 42.90 (+5.30, +14.10%) | 41.20 (+2.70, +7.01%) | 36.17 (+3.80, +11.74%) | 27.75 (+3.11, +12.60%) | 33.53 (+3.19, +10.53%) | 38.21 (+4.70, +14.03%) | 33.76 (+1.35, +4.17%) | 28.47 (+2.97, +11.63%) | 38.55 (-3.55, -8.43%) | 33.54 (+1.27, +3.93%) |
 
 ![Results](img/avg.png)
 
@@ -75,6 +75,16 @@ Both absolute and relative differences in BLEU scores between Bergamot and other
 
 ![Results](img/es-en.png)
 
+## nb-en
+
+| Translator/Dataset | flores-dev | flores-test |
+| --- | --- | --- |
+| bergamot | 37.40 | 37.80 |
+| google | 41.80 (+4.40, +11.76%) | 42.30 (+4.50, +11.90%) |
+| microsoft | 42.70 (+5.30, +14.17%) | 43.10 (+5.30, +14.02%) |
+
+![Results](img/nb-en.png)
+
 ## bg-en
 
 | Translator/Dataset | flores-dev | flores-test |
diff --git a/models/dev/enfa/lex.50.50.enfa.s2t.bin.gz b/models/dev/enfa/lex.50.50.enfa.s2t.bin.gz
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e9e0885217bc5e4d200ab217d30fd15df1bcd1e938f3f86021ee782feed6f324
+size 2623030
diff --git a/models/dev/enfa/model.enfa.intgemm.alphas.bin.gz b/models/dev/enfa/model.enfa.intgemm.alphas.bin.gz
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9eef5c5601c38e020b176639f7b16d4f59b61d3b23368c48841b514087c39beb
+size 13376428
diff --git a/models/dev/enfa/vocab.enfa.spm.gz b/models/dev/enfa/vocab.enfa.spm.gz
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:82d0c2e6f52bdebcb9aef4f180b79ff4d74e0b35bfe3b44393111430b880bdf1
+size 372636
diff --git a/models/dev/faen/lex.50.50.faen.s2t.bin.gz b/models/dev/faen/lex.50.50.faen.s2t.bin.gz
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d25f9ddae341873b79d834db9fe5c77578d370981edac5189a9216ccd9b655a
+size 3230408
diff --git a/models/dev/faen/model.faen.intgemm.alphas.bin.gz b/models/dev/faen/model.faen.intgemm.alphas.bin.gz
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:641b4447be71c24def35d2fcae99cdfcab634a70b52e2a4f07a242630d16b20c
+size 13077214
diff --git a/models/dev/faen/vocab.faen.spm.gz b/models/dev/faen/vocab.faen.spm.gz
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8ecd7649ef77244b6becacdea158c182676ca328a858e6c5dd829ab8419a15bd
+size 421102
diff --git a/models/dev/isen/lex.50.50.isen.s2t.bin.gz b/models/dev/isen/lex.50.50.isen.s2t.bin.gz
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0870eca2fee1d6c4a382eb8f23c36c944d91dc93b1b1900dc30f4c4d0abd7712
+size 1502107
diff --git a/models/dev/isen/model.isen.intgemm.alphas.bin.gz b/models/dev/isen/model.isen.intgemm.alphas.bin.gz
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3132a1628f32e2ae7a0fa6adf907eae2d5dbd56206425252971242cfc4f6166c
+size 13989880
diff --git a/models/dev/isen/vocab.isen.spm.gz b/models/dev/isen/vocab.isen.spm.gz
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dab6ca0f0d862e33f787a8bf37b91008f793171c13ea4174e69894ff2989e3eb
+size 416849
diff --git a/models/dev/nnen/lex.50.50.nnen.s2t.bin.gz b/models/dev/nnen/lex.50.50.nnen.s2t.bin.gz
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:da94ec4e1b149c79cfa02f52b3041461a4d5334fbe69ab6adcaa6b56c04f8679
+size 731899
diff --git a/models/dev/nnen/model.nnen.intgemm.alphas.bin.gz b/models/dev/nnen/model.nnen.intgemm.alphas.bin.gz
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:54ede4b47a2505768667938d2f41f480a30bd99ef5bf7fbcf5298bb8172065a2
+size 10465158
diff --git a/models/dev/nnen/vocab.nnen.spm.gz b/models/dev/nnen/vocab.nnen.spm.gz
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d7f93e11e24510aae532480e46b7f88839f401207dc6b22e3814c5d719997c47
+size 269397
diff --git a/models/prod/nben/lex.50.50.nben.s2t.bin.gz b/models/prod/nben/lex.50.50.nben.s2t.bin.gz
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a5082e96461e13d22e51535df14cd668f4697bdefc609206a18c66ac9862165e
+size 721631
diff --git a/models/prod/nben/model.nben.intgemm.alphas.bin.gz b/models/prod/nben/model.nben.intgemm.alphas.bin.gz
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ccc1633225bc7e8c26f1998b8a436eec55f5262cb1fbf778637d7741390c04dd
+size 10284054
diff --git a/models/prod/nben/vocab.nben.spm.gz b/models/prod/nben/vocab.nben.spm.gz
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6e6c78acd1369cc3a9d49abffa33f8ef8b62f24188df00a13f8a32954a449e8c
+size 269244
diff --git a/scripts/import-statmt.sh b/scripts/import-statmt.sh
@@ -0,0 +1,17 @@
+#!/bin/bash
+
+pair=$1
+
+mkdir -P models/prod/$pair
+
+# directory structure is not consistent on statmt
+wget -nc -P models/prod/$pair/ http://data.statmt.org/bergamot/models/$pair/$pair.student.tiny11/lex.s2t.bin
+wget -nc -P models/prod/$pair/ http://data.statmt.org/bergamot/models/$pair/$pair.student.tiny11/model.intgemm.alphas.bin
+#wget -nc -P models/prod/$pair/ http://data.statmt.org/bergamot/models/$pair/$pair.student.tiny11/vocab.$pair.spm
+wget -nc -P models/prod/$pair/ http://data.statmt.org/bergamot/models/$pair/vocab.nben.spm
+
+# follow our naming convention
+mv models/prod/$pair/lex.s2t.bin models/prod/$pair/lex.50.50.$pair.s2t.bin
+mv models/prod/$pair/model.intgemm.alphas.bin models/prod/$pair/model.$pair.intgemm.alphas.bin
+
+gzip models/prod/$pair/*

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+version https://git-lfs.github.com/spec/v1`
	`2`	`+oid sha256:e9e0885217bc5e4d200ab217d30fd15df1bcd1e938f3f86021ee782feed6f324`
	`3`	`+size 2623030`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+version https://git-lfs.github.com/spec/v1`
	`2`	`+oid sha256:9eef5c5601c38e020b176639f7b16d4f59b61d3b23368c48841b514087c39beb`
	`3`	`+size 13376428`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+version https://git-lfs.github.com/spec/v1`
	`2`	`+oid sha256:82d0c2e6f52bdebcb9aef4f180b79ff4d74e0b35bfe3b44393111430b880bdf1`
	`3`	`+size 372636`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+version https://git-lfs.github.com/spec/v1`
	`2`	`+oid sha256:0d25f9ddae341873b79d834db9fe5c77578d370981edac5189a9216ccd9b655a`
	`3`	`+size 3230408`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+version https://git-lfs.github.com/spec/v1`
	`2`	`+oid sha256:641b4447be71c24def35d2fcae99cdfcab634a70b52e2a4f07a242630d16b20c`
	`3`	`+size 13077214`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+version https://git-lfs.github.com/spec/v1`
	`2`	`+oid sha256:8ecd7649ef77244b6becacdea158c182676ca328a858e6c5dd829ab8419a15bd`
	`3`	`+size 421102`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+version https://git-lfs.github.com/spec/v1`
	`2`	`+oid sha256:0870eca2fee1d6c4a382eb8f23c36c944d91dc93b1b1900dc30f4c4d0abd7712`
	`3`	`+size 1502107`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+version https://git-lfs.github.com/spec/v1`
	`2`	`+oid sha256:3132a1628f32e2ae7a0fa6adf907eae2d5dbd56206425252971242cfc4f6166c`
	`3`	`+size 13989880`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+version https://git-lfs.github.com/spec/v1`
	`2`	`+oid sha256:dab6ca0f0d862e33f787a8bf37b91008f793171c13ea4174e69894ff2989e3eb`
	`3`	`+size 416849`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+version https://git-lfs.github.com/spec/v1`
	`2`	`+oid sha256:da94ec4e1b149c79cfa02f52b3041461a4d5334fbe69ab6adcaa6b56c04f8679`
	`3`	`+size 731899`