Zh tn #50

BuyuanCui · 2023-02-27T19:00:55Z

What does this PR do?

Changes are made to the ZH_TN from outside contributor. Updates to the existing grammars and alignments applied to keep the consistency with the ZH_ITN Grammar.

Cardinal grammar is separated into two grammars, cardinal and decimal. Decimal being an independent class.
Increased cardinal grammar coverage up to hundred billion.
Added ordinal grammar that works based on the cardinal grammar by processing a morpheme that indicates the order "第".
Update on the date grammar, not processing inputs with only two of the components of year, month, and date. For example, 2002/02, 02/11 are not accepted. The reason is these input formats are not idea according to the national guideline (http://www.zgzlyx.com/uploadfile/news_images/zlyx/2022-05-26/%E4%B8%AD%E5%8D%8E%E4%BA%BA%E6%B0%91%E5%85%B1%E5%92%8C%E5%9B%BD%E5%9B%BD%E5%AE%B6%E6%A0%87%E5%87%86%E2%80%94%E2%80%94%E5%87%BA%E7%89%88%E7%89%A9%E4%B8%8A%E6%95%B0%E5%AD%97%E7%94%A8%E6%B3%95%EF%BC%88GB%EF%BC%8FT%2015835-2011%EF%BC%89.pdf).
Update to time grammar include expressions that are limited to the format of 'hour: minute: second'. The grammar can also process Inputs like "5点6分". Another update processes time expressed to mean a range of time. Some of the sample expressions are "五个小时", "5秒钟" and "五个钟头".
Update to fraction grammar include expressions that are in percentages, for example, "50%" or "百分之五十".
Update to money grammar to process expressions involving units like "块", "毛", and "分"。Large money expressed in decimal format, for example, "1.5万美元". Also including expressions where the currency is not in symbol format for are in Mandarins, for example, ”￥15“ vs. "15人民币".
Did not include measure, math and preprocessor. After discussing with the team, the plan is to align the classes to the ITN grammar, so only cardinal, ordinal, fraction, decimal, time, date, and money are included.

Add a one line overview of what this PR aims to accomplish.
Update the cardinal, ordinal, decimal, fraction, time, date, and money grammar, and remove math, measure, and preprocessors.

Before your PR is "Ready for review"

Pre checks:

Have you signed your commits? Use git commit -s to sign.
[*] Do all unittests finish successfully before sending PR?
1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here.
[*] Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files?
[*] Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ?
[*] Have you added the correct license header Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files?
[*] If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. See an example here.
[*] Remove import guards (try import: ... except: ...) if not already done.
If you added a new language or a new feature please update the NeMo documentation (lives in different repo). (Couldn't access)
[*] Have you added your language support to tools/text_processing_deployment/pynini_export.py.

PR Type:

[*] New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

This is the updates for zh_CN TN grammars.

rebasing on main.

Signed-off-by: BuyuanCui <[email protected]>

* save Signed-off-by: Yang Zhang <[email protected]> * save Signed-off-by: Yang Zhang <[email protected]> * extend alignment for itn Signed-off-by: Yang Zhang <[email protected]> --------- Signed-off-by: Yang Zhang <[email protected]> Signed-off-by: BuyuanCui <[email protected]>

Signed-off-by: BuyuanCui <[email protected]>

yzhang123 · 2023-02-27T22:16:00Z

@fayejf could you pls review this PR?

fayejf · 2023-02-27T22:30:12Z

@yzhang123 Sure!

github-actions · 2023-03-14T01:52:05Z

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

github-actions · 2023-03-21T01:53:50Z

This PR was closed because it has been inactive for 7 days since being marked as stale.

…m to MAndarin Signed-off-by: BuyuanCui <[email protected]>

… to last reviews and changing weights Signed-off-by: BuyuanCui <[email protected]>

Signed-off-by: BuyuanCui <[email protected]>

…H_TN

jimregan · 2023-06-28T15:19:58Z

tests/nemo_text_processing/zh/__init__ copy.py

@@ -1,4 +1,4 @@
-# Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.
+# Copyright (c) 2021, NVIDIA CORPORATION.  All rights reserved.


Having a file called __init__ copy.py seems like it might cause problems later on. Also, it's unneeded

jimregan · 2023-06-28T15:21:45Z

nemo_text_processing/text_normalization/zh/utils.py

@@ -11,14 +11,11 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
+# gaph_utils is kept due to the fatc that importing from en folders will cause import errors that the data file names have to be the same with what are in the en folder


This is only true for get_abs_path

Yes, when importing the data, I used get_abs_path everytime.

jimregan · 2023-06-28T15:22:28Z

nemo_text_processing/text_normalization/zh/graph_utils.py

+    """
+    return fst @ pynini.cdrewrite(pynini.cross(NEMO_SPACE, NEMO_NON_BREAKING_SPACE), "", "", NEMO_SIGMA)
+
+
 class GraphFst:


You don't need this, just import from en

jimregan · 2023-06-28T15:23:01Z

nemo_text_processing/text_normalization/zh/graph_utils.py

@@ -61,6 +90,44 @@ def generator_main(file_name: str, graphs: Dict[str, 'pynini.FstLike']):
    logging.info(f'Created {file_name}')


+def get_plurals(fst):


This is very specific to English

Would remove it.

nemo_text_processing/text_normalization/zh/Changes.md

nemo_text_processing/text_normalization/zh/taggers/ordinal.py

+
+import pynini
+from nemo_text_processing.text_normalization.zh.graph_utils import GraphFst
+from nemo_text_processing.text_normalization.zh.taggers.cardinal import CardinalFst


nemo_text_processing/text_normalization/zh/taggers/date.py


-    def __init__(self, deterministic: bool = True, lm: bool = False):
-        super().__init__(name="date", kind="classify", deterministic=deterministic)
+        cardinal = cardinal.just_cardinals


nemo_text_processing/text_normalization/zh/verbalizers/money.py

tests/nemo_text_processing/zh/test_math.py

+# class TestMath:
+#    normalizer_zh = Normalizer(lang='zh', cache_dir=CACHE_DIR, overwrite_cache=False, input_case='cased')
+#
+#    @parameterized.expand(parse_test_case_file('zh/data_text_normalization/test_cases_math.txt'))
+#    @pytest.mark.run_only_on('CPU')
+#    @pytest.mark.unit
+#    def test_norm_math(self, test_input, expected):
+#        preds = self.normalizer_zh.normalize(test_input)
+#        assert expected == preds


nemo_text_processing/text_normalization/zh/graph_utils.py

nemo_text_processing/text_normalization/zh/verbalizers/money.py

Signed-off-by: Buyuan(Alex) Cui <[email protected]>

for more information, see https://pre-commit.ci

BuyuanCui · 2023-07-17T19:18:05Z

Re-ran PR. at: #89
Due to rebase and conflict issues. I ran the PR again.

ekmb · 2023-07-20T21:13:38Z

@BuyuanCui could this be closed?

github-actions · 2023-08-04T01:51:49Z

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

github-actions · 2023-08-12T01:45:41Z

This PR was closed because it has been inactive for 7 days since being marked as stale.

BuyuanCui and others added 13 commits February 23, 2023 09:44

zh_TN updates, grammar

afc1901

test datas for zh_tn

4a67449

Merge branch 'main' of github.com:NVIDIA/NeMo-text-processing into main

b477373

This is the updates for zh_CN TN grammars.

style fix and formats

7d2507a

style fix and format

f6f2240

udate with style issues

a28dc6c

removed some of the files to align with itn first release

9b3739a

a note for rough explanations on the changes

af778e3

Merge branch 'main' of github.com:NVIDIA/NeMo-text-processing into ZH_TN

832617d

rebasing on main.

removed some of the files to align with itn first release

1120559

Signed-off-by: BuyuanCui <[email protected]>

a note for rough explanations on the changes

dbc83d6

Signed-off-by: BuyuanCui <[email protected]>

update on the file date

5a8785f

Signed-off-by: BuyuanCui <[email protected]>

BuyuanCui assigned yzhang123 Feb 27, 2023

yzhang123 removed their assignment Feb 27, 2023

github-actions bot added the Stale label Mar 14, 2023

github-actions bot closed this Mar 21, 2023

BuyuanCui added 7 commits June 6, 2023 09:09

removing some data files to not process am and pm, not converting the…

d7858be

…m to MAndarin Signed-off-by: BuyuanCui <[email protected]>

update for cardinal, time, and tokenize_andclassufy grammar according…

098b5ab

… to last reviews and changing weights Signed-off-by: BuyuanCui <[email protected]>

updates adapting to the changes in tagger

f21fbd3

Signed-off-by: BuyuanCui <[email protected]>

updating money test data, updating time test data

171f9ac

Signed-off-by: BuyuanCui <[email protected]>

commenting out this file but will keep it for future release

db1179e

Signed-off-by: BuyuanCui <[email protected]>

commenting out this file but will keep it for now for future release

f2a8560

Signed-off-by: BuyuanCui <[email protected]>

Merge branch 'ZH_TN' of github.com:NVIDIA/NeMo-text-processing into Z…

99acfcd

…H_TN

BuyuanCui reopened this Jun 23, 2023

BuyuanCui marked this pull request as ready for review June 23, 2023 23:29

github-actions bot removed the Stale label Jun 24, 2023

fayejf self-requested a review June 27, 2023 22:31

jimregan reviewed Jun 28, 2023

View reviewed changes

nemo_text_processing/text_normalization/zh/Changes.md Show resolved Hide resolved

jimregan reviewed Jun 28, 2023

View reviewed changes

nemo_text_processing/text_normalization/zh/Changes.md Show resolved Hide resolved

github-advanced-security bot found potential problems Jun 30, 2023

View reviewed changes

nemo_text_processing/text_normalization/zh/verbalizers/money.py Fixed Show fixed Hide fixed

BuyuanCui force-pushed the ZH_TN branch 3 times, most recently from 8cea91c to 99acfcd Compare July 6, 2023 20:01

BuyuanCui and others added 2 commits July 7, 2023 12:59

Merge branch 'main' into ZH_TN

6224a89

Signed-off-by: Buyuan(Alex) Cui <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

d022cab

for more information, see https://pre-commit.ci

github-actions bot added the Stale label Aug 4, 2023

github-actions bot closed this Aug 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Zh tn #50

Zh tn #50

BuyuanCui commented Feb 27, 2023 •

edited

Loading

yzhang123 commented Feb 27, 2023

fayejf commented Feb 27, 2023

github-actions bot commented Mar 14, 2023

github-actions bot commented Mar 21, 2023

jimregan Jun 28, 2023 •

edited

Loading

jimregan Jun 28, 2023

BuyuanCui Jun 30, 2023

jimregan Jun 28, 2023

jimregan Jun 28, 2023

BuyuanCui Jun 30, 2023

BuyuanCui commented Jul 17, 2023

ekmb commented Jul 20, 2023

github-actions bot commented Aug 4, 2023

github-actions bot commented Aug 12, 2023

		@@ -1,4 +1,4 @@
		# Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
		# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.

		@@ -61,6 +90,44 @@ def generator_main(file_name: str, graphs: Dict[str, 'pynini.FstLike']):
		logging.info(f'Created {file_name}')


		def get_plurals(fst):

Zh tn #50

Zh tn #50

Conversation

BuyuanCui commented Feb 27, 2023 • edited Loading

What does this PR do?

Before your PR is "Ready for review"

yzhang123 commented Feb 27, 2023

fayejf commented Feb 27, 2023

github-actions bot commented Mar 14, 2023

github-actions bot commented Mar 21, 2023

jimregan Jun 28, 2023 • edited Loading

Choose a reason for hiding this comment

jimregan Jun 28, 2023

Choose a reason for hiding this comment

BuyuanCui Jun 30, 2023

Choose a reason for hiding this comment

jimregan Jun 28, 2023

Choose a reason for hiding this comment

jimregan Jun 28, 2023

Choose a reason for hiding this comment

BuyuanCui Jun 30, 2023

Choose a reason for hiding this comment

BuyuanCui commented Jul 17, 2023

ekmb commented Jul 20, 2023

github-actions bot commented Aug 4, 2023

github-actions bot commented Aug 12, 2023

BuyuanCui commented Feb 27, 2023 •

edited

Loading

jimregan Jun 28, 2023 •

edited

Loading