Expands per/unit mappings and updates the cache (#227)

* Expands per/unit mappings and updates cache Signed-off-by: Simon Zuberek <[email protected]> * Adds tests documenting edge cases Signed-off-by: Simon Zuberek <[email protected]> --------- Signed-off-by: Simon Zuberek <[email protected]> Co-authored-by: Simon Zuberek <[email protected]>
NVIDIA · Sep 11, 2024 · 4289052 · 4289052
1 parent 92bdf93
commit 4289052
Show file tree

Hide file tree

Showing 3 changed files with 63 additions and 2 deletions.
diff --git a/Jenkinsfile b/Jenkinsfile
@@ -13,7 +13,7 @@ pipeline {
 
     AR_TN_CACHE='/home/jenkinsci/TestData/text_norm/ci/grammars/04-24-24-0'
     DE_TN_CACHE='/home/jenkinsci/TestData/text_norm/ci/grammars/06-03-24-0'
-    EN_TN_CACHE='/home/jenkinsci/TestData/text_norm/ci/grammars/08-22-24-0'
+    EN_TN_CACHE='/home/jenkinsci/TestData/text_norm/ci/grammars/09-04-24-0'
     ES_TN_CACHE='/home/jenkinsci/TestData/text_norm/ci/grammars/08-30-24-0'
     ES_EN_TN_CACHE='/home/jenkinsci/TestData/text_norm/ci/grammars/08-30-24-0'
     FR_TN_CACHE='/home/jenkinsci/TestData/text_norm/ci/grammars/06-04-24-0'

diff --git a/nemo_text_processing/text_normalization/en/data/money/per_unit.tsv b/nemo_text_processing/text_normalization/en/data/money/per_unit.tsv
@@ -8,4 +8,63 @@
 /wk	per week
 /mo	per month
 /yr	per year
-/svc	per service
+/svc	per service
+/L	per liter
+/l	per liter
+/gal	per gallon
+/gallon	per gallon
+/kg	per kilogram
+/g	per gram
+/lb	per pound
+/pound	per pound
+/oz	per ounce
+/ounce	per ounce
+/unit	per unit
+/ft	per foot
+/foot	per foot
+/'	per foot
+/in	per inch
+/inch	per inch
+/"	per inch
+/m	per meter
+/meter	per meter
+/yard	per yard
+/yd	per yard
+/mile	per mile
+/mi	per mile
+/square meter	per square meter
+/sq meter	per square meter
+/sq m	per square meter
+/m²	per square meter
+/square foot	per square foot
+/sq foot	per square foot
+/sq ft	per square foot
+/ft²	per square foot
+/cubic foot	per cubic foot
+/cu foot	per cubic foot
+/cu ft	per cubic foot
+/ft³	per cubic foot
+/cubic meter	per cubic meter
+/cu meter	per cubic meter
+/cu m	per cubic meter
+/m³	per cubic meter
+/visit	per visit
+/page	per page
+/pg	per page
+/usr	per user
+/user	per user
+/night	per night
+/passenger	per passenger
+/word	per word
+/wd	per word
+/GB	per gigabyte
+/MB	per megabyte
+/TB	per terrabyte
+/cycle	per cycle
+/load	per load
+/person	per person
+/task	per task
+/shift	per shift
+/project	per project
+/class	per class
+/session	per session
diff --git a/tests/nemo_text_processing/en/data_text_normalization/test_cases_address.txt b/tests/nemo_text_processing/en/data_text_normalization/test_cases_address.txt
@@ -7,3 +7,5 @@
 12 S 1st st~twelve South first Street
 1990 for the Ata ST~nineteen ninety for the Ata ST
 Main St.~Main St.
+Nancy lived at 1428 Elm St. It was a strange place.~Nancy lived at fourteen twenty eight Elm Street. It was a strange place.
+(incorrect normalization) Nancy lived at 1428 Elm St. 4 doors down lived her best friend.~(incorrect normalization) Nancy lived at fourteen twenty eight Elm Street four doors down lived her best friend.