Add de-en model (#25)

* Add de-en model * Update evaluation results [skip ci] Co-authored-by: CircleCI evaluation job <ci-models-evaluation@firefox-translations>
mozilla · Sep 2, 2021 · 139fbf4 · 139fbf4
1 parent e08e94a
commit 139fbf4
Show file tree

Hide file tree

Showing 54 changed files with 72 additions and 5 deletions.
diff --git a/evaluation/prod/de-en/flores-dev.bergamot.en.bleu b/evaluation/prod/de-en/flores-dev.bergamot.en.bleu
@@ -0,0 +1 @@
+39.4
diff --git a/evaluation/prod/de-en/flores-dev.google.en.bleu b/evaluation/prod/de-en/flores-dev.google.en.bleu
@@ -0,0 +1 @@
+43.1
diff --git a/evaluation/prod/de-en/flores-dev.microsoft.en.bleu b/evaluation/prod/de-en/flores-dev.microsoft.en.bleu
@@ -0,0 +1 @@
+44.9
diff --git a/evaluation/prod/de-en/flores-test.bergamot.en.bleu b/evaluation/prod/de-en/flores-test.bergamot.en.bleu
@@ -0,0 +1 @@
+39.1
diff --git a/evaluation/prod/de-en/flores-test.google.en.bleu b/evaluation/prod/de-en/flores-test.google.en.bleu
@@ -0,0 +1 @@
+42.8
diff --git a/evaluation/prod/de-en/flores-test.microsoft.en.bleu b/evaluation/prod/de-en/flores-test.microsoft.en.bleu
@@ -0,0 +1 @@
+45.8
diff --git a/evaluation/prod/de-en/iwslt17.bergamot.en.bleu b/evaluation/prod/de-en/iwslt17.bergamot.en.bleu
@@ -0,0 +1 @@
+28.9
diff --git a/evaluation/prod/de-en/iwslt17.google.en.bleu b/evaluation/prod/de-en/iwslt17.google.en.bleu
@@ -0,0 +1 @@
+30.1
diff --git a/evaluation/prod/de-en/iwslt17.microsoft.en.bleu b/evaluation/prod/de-en/iwslt17.microsoft.en.bleu
@@ -0,0 +1 @@
+32.5
diff --git a/evaluation/prod/de-en/wmt08.bergamot.en.bleu b/evaluation/prod/de-en/wmt08.bergamot.en.bleu
@@ -0,0 +1 @@
+26.3
diff --git a/evaluation/prod/de-en/wmt08.google.en.bleu b/evaluation/prod/de-en/wmt08.google.en.bleu
@@ -0,0 +1 @@
+27.6
diff --git a/evaluation/prod/de-en/wmt08.microsoft.en.bleu b/evaluation/prod/de-en/wmt08.microsoft.en.bleu
@@ -0,0 +1 @@
+29.4
diff --git a/evaluation/prod/de-en/wmt09.bergamot.en.bleu b/evaluation/prod/de-en/wmt09.bergamot.en.bleu
@@ -0,0 +1 @@
+26.4
diff --git a/evaluation/prod/de-en/wmt09.google.en.bleu b/evaluation/prod/de-en/wmt09.google.en.bleu
@@ -0,0 +1 @@
+27.2
diff --git a/evaluation/prod/de-en/wmt09.microsoft.en.bleu b/evaluation/prod/de-en/wmt09.microsoft.en.bleu
@@ -0,0 +1 @@
+29.1
diff --git a/evaluation/prod/de-en/wmt10.bergamot.en.bleu b/evaluation/prod/de-en/wmt10.bergamot.en.bleu
@@ -0,0 +1 @@
+29.3
diff --git a/evaluation/prod/de-en/wmt10.google.en.bleu b/evaluation/prod/de-en/wmt10.google.en.bleu
@@ -0,0 +1 @@
+32.1
diff --git a/evaluation/prod/de-en/wmt10.microsoft.en.bleu b/evaluation/prod/de-en/wmt10.microsoft.en.bleu
@@ -0,0 +1 @@
+33.4
diff --git a/evaluation/prod/de-en/wmt11.bergamot.en.bleu b/evaluation/prod/de-en/wmt11.bergamot.en.bleu
@@ -0,0 +1 @@
+26.3
diff --git a/evaluation/prod/de-en/wmt11.google.en.bleu b/evaluation/prod/de-en/wmt11.google.en.bleu
@@ -0,0 +1 @@
+27.3
diff --git a/evaluation/prod/de-en/wmt11.microsoft.en.bleu b/evaluation/prod/de-en/wmt11.microsoft.en.bleu
@@ -0,0 +1 @@
+29.2
diff --git a/evaluation/prod/de-en/wmt12.bergamot.en.bleu b/evaluation/prod/de-en/wmt12.bergamot.en.bleu
@@ -0,0 +1 @@
+27.6
diff --git a/evaluation/prod/de-en/wmt12.google.en.bleu b/evaluation/prod/de-en/wmt12.google.en.bleu
@@ -0,0 +1 @@
+29.5
diff --git a/evaluation/prod/de-en/wmt12.microsoft.en.bleu b/evaluation/prod/de-en/wmt12.microsoft.en.bleu
@@ -0,0 +1 @@
+31.3
diff --git a/evaluation/prod/de-en/wmt13.bergamot.en.bleu b/evaluation/prod/de-en/wmt13.bergamot.en.bleu
@@ -0,0 +1 @@
+30.9
diff --git a/evaluation/prod/de-en/wmt13.google.en.bleu b/evaluation/prod/de-en/wmt13.google.en.bleu
@@ -0,0 +1 @@
+32.4
diff --git a/evaluation/prod/de-en/wmt13.microsoft.en.bleu b/evaluation/prod/de-en/wmt13.microsoft.en.bleu
@@ -0,0 +1 @@
+34.3
diff --git a/evaluation/prod/de-en/wmt14.bergamot.en.bleu b/evaluation/prod/de-en/wmt14.bergamot.en.bleu
@@ -0,0 +1 @@
+33.6
diff --git a/evaluation/prod/de-en/wmt14.google.en.bleu b/evaluation/prod/de-en/wmt14.google.en.bleu
@@ -0,0 +1 @@
+37.4
diff --git a/evaluation/prod/de-en/wmt14.microsoft.en.bleu b/evaluation/prod/de-en/wmt14.microsoft.en.bleu
@@ -0,0 +1 @@
+39.2
diff --git a/evaluation/prod/de-en/wmt15.bergamot.en.bleu b/evaluation/prod/de-en/wmt15.bergamot.en.bleu
@@ -0,0 +1 @@
+33.5
diff --git a/evaluation/prod/de-en/wmt15.google.en.bleu b/evaluation/prod/de-en/wmt15.google.en.bleu
@@ -0,0 +1 @@
+36.1
diff --git a/evaluation/prod/de-en/wmt15.microsoft.en.bleu b/evaluation/prod/de-en/wmt15.microsoft.en.bleu
@@ -0,0 +1 @@
+38.1
diff --git a/evaluation/prod/de-en/wmt16.bergamot.en.bleu b/evaluation/prod/de-en/wmt16.bergamot.en.bleu
@@ -0,0 +1 @@
+39.6
diff --git a/evaluation/prod/de-en/wmt16.google.en.bleu b/evaluation/prod/de-en/wmt16.google.en.bleu
@@ -0,0 +1 @@
+42.3
diff --git a/evaluation/prod/de-en/wmt16.microsoft.en.bleu b/evaluation/prod/de-en/wmt16.microsoft.en.bleu
@@ -0,0 +1 @@
+46.3
diff --git a/evaluation/prod/de-en/wmt17.bergamot.en.bleu b/evaluation/prod/de-en/wmt17.bergamot.en.bleu
@@ -0,0 +1 @@
+34.9
diff --git a/evaluation/prod/de-en/wmt17.google.en.bleu b/evaluation/prod/de-en/wmt17.google.en.bleu
@@ -0,0 +1 @@
+38.7
diff --git a/evaluation/prod/de-en/wmt17.microsoft.en.bleu b/evaluation/prod/de-en/wmt17.microsoft.en.bleu
@@ -0,0 +1 @@
+40.8
diff --git a/evaluation/prod/de-en/wmt18.bergamot.en.bleu b/evaluation/prod/de-en/wmt18.bergamot.en.bleu
@@ -0,0 +1 @@
+43.5
diff --git a/evaluation/prod/de-en/wmt18.google.en.bleu b/evaluation/prod/de-en/wmt18.google.en.bleu
@@ -0,0 +1 @@
+46.2
diff --git a/evaluation/prod/de-en/wmt18.microsoft.en.bleu b/evaluation/prod/de-en/wmt18.microsoft.en.bleu
@@ -0,0 +1 @@
+49.6
diff --git a/evaluation/prod/de-en/wmt19.bergamot.en.bleu b/evaluation/prod/de-en/wmt19.bergamot.en.bleu
@@ -0,0 +1 @@
+39.2
diff --git a/evaluation/prod/de-en/wmt19.google.en.bleu b/evaluation/prod/de-en/wmt19.google.en.bleu
@@ -0,0 +1 @@
+41.1
diff --git a/evaluation/prod/de-en/wmt19.microsoft.en.bleu b/evaluation/prod/de-en/wmt19.microsoft.en.bleu
@@ -0,0 +1 @@
+43.8
diff --git a/evaluation/prod/de-en/wmt20.bergamot.en.bleu b/evaluation/prod/de-en/wmt20.bergamot.en.bleu
@@ -0,0 +1 @@
+39.1
diff --git a/evaluation/prod/de-en/wmt20.google.en.bleu b/evaluation/prod/de-en/wmt20.google.en.bleu
@@ -0,0 +1 @@
+41.8
diff --git a/evaluation/prod/de-en/wmt20.microsoft.en.bleu b/evaluation/prod/de-en/wmt20.microsoft.en.bleu
@@ -0,0 +1 @@
+43.6
diff --git a/evaluation/prod/img/avg.png b/evaluation/prod/img/avg.png
diff --git a/evaluation/prod/img/de-en.png b/evaluation/prod/img/de-en.png
diff --git a/evaluation/prod/results.md b/evaluation/prod/results.md
@@ -57,11 +57,11 @@ Both absolute and relative differences in BLEU scores between Bergamot and other
 
 ## avg
 
-| Translator/Dataset | cs-en | en-et | et-en | en-es | en-cs | es-en | en-de |
-| --- | --- | --- | --- | --- | --- | --- | --- |
-| bergamot | 30.34 | 25.50 | 32.37 | 32.41 | 24.65 | 32.38 | 31.20 |
-| google | 32.40 (+2.06, +6.80%) | 28.60 (+3.10, +12.16%) | 35.80 (+3.43, +10.61%) | 34.74 (+2.32, +7.17%) | 26.73 (+2.09, +8.47%) | 33.64 (+1.27, +3.91%) | 33.05 (+1.85, +5.93%) |
-| microsoft | 33.53 (+3.19, +10.53%) | 28.47 (+2.97, +11.63%) | 36.17 (+3.80, +11.74%) | 33.76 (+1.35, +4.17%) | 27.75 (+3.11, +12.60%) | 32.93 (+0.56, +1.72%) | 33.54 (+2.34, +7.51%) |
+| Translator/Dataset | cs-en | en-et | et-en | en-es | en-cs | de-en | es-en | en-de |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| bergamot | 30.34 | 25.50 | 32.37 | 32.41 | 24.65 | 33.60 | 32.38 | 31.20 |
+| google | 32.40 (+2.06, +6.80%) | 28.60 (+3.10, +12.16%) | 35.80 (+3.43, +10.61%) | 34.74 (+2.32, +7.17%) | 26.73 (+2.09, +8.47%) | 35.98 (+2.38, +7.09%) | 33.64 (+1.27, +3.91%) | 33.05 (+1.85, +5.93%) |
+| microsoft | 33.53 (+3.19, +10.53%) | 28.47 (+2.97, +11.63%) | 36.17 (+3.80, +11.74%) | 33.76 (+1.35, +4.17%) | 27.75 (+3.11, +12.60%) | 38.21 (+4.61, +13.71%) | 32.93 (+0.56, +1.72%) | 33.54 (+2.34, +7.51%) |
 
 ![Results](img/avg.png)
 
@@ -115,6 +115,16 @@ Both absolute and relative differences in BLEU scores between Bergamot and other
 
 ![Results](img/en-cs.png)
 
+## de-en
+
+| Translator/Dataset | wmt08 | wmt19 | wmt17 | iwslt17 | wmt10 | flores-dev | flores-test | wmt12 | wmt11 | wmt14 | wmt15 | wmt16 | wmt13 | wmt18 | wmt09 | wmt20 |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| bergamot | 26.30 | 39.20 | 34.90 | 28.90 | 29.30 | 39.40 | 39.10 | 27.60 | 26.30 | 33.60 | 33.50 | 39.60 | 30.90 | 43.50 | 26.40 | 39.10 |
+| google | 27.60 (+1.30, +4.94%) | 41.10 (+1.90, +4.85%) | 38.70 (+3.80, +10.89%) | 30.10 (+1.20, +4.15%) | 32.10 (+2.80, +9.56%) | 43.10 (+3.70, +9.39%) | 42.80 (+3.70, +9.46%) | 29.50 (+1.90, +6.88%) | 27.30 (+1.00, +3.80%) | 37.40 (+3.80, +11.31%) | 36.10 (+2.60, +7.76%) | 42.30 (+2.70, +6.82%) | 32.40 (+1.50, +4.85%) | 46.20 (+2.70, +6.21%) | 27.20 (+0.80, +3.03%) | 41.80 (+2.70, +6.91%) |
+| microsoft | 29.40 (+3.10, +11.79%) | 43.80 (+4.60, +11.73%) | 40.80 (+5.90, +16.91%) | 32.50 (+3.60, +12.46%) | 33.40 (+4.10, +13.99%) | 44.90 (+5.50, +13.96%) | 45.80 (+6.70, +17.14%) | 31.30 (+3.70, +13.41%) | 29.20 (+2.90, +11.03%) | 39.20 (+5.60, +16.67%) | 38.10 (+4.60, +13.73%) | 46.30 (+6.70, +16.92%) | 34.30 (+3.40, +11.00%) | 49.60 (+6.10, +14.02%) | 29.10 (+2.70, +10.23%) | 43.60 (+4.50, +11.51%) |
+
+![Results](img/de-en.png)
+
 ## es-en
 
 | Translator/Dataset | wmt08 | mtedx_test | wmt10 | flores-dev | flores-test | wmt12 | wmt11 | wmt13 | wmt09 |

diff --git a/models/prod/deen/lex.50.50.deen.s2t.bin.gz b/models/prod/deen/lex.50.50.deen.s2t.bin.gz
diff --git a/models/prod/deen/model.deen.intgemm.alphas.bin.gz b/models/prod/deen/model.deen.intgemm.alphas.bin.gz
diff --git a/models/prod/deen/vocab.deen.spm.gz b/models/prod/deen/vocab.deen.spm.gz