Jp tn 20241017

[BuyuanCui]

What does this PR do ?

Japanese TN grammar and test file

Before your PR is "Ready for review"

Pre checks:

• Have you signed your commits? Use git commit -s to sign.
• Do all unittests finish successfully before sending PR?
  1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
  2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
• If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here.
• Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files?
• Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ?
• Have you added the correct license header Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files?
• If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. See an example here.
• Remove import guards (try import: ... except: ...) if not already done.
• If you added a new language or a new feature please update the NeMo documentation (lives in different repo).
• Have you added your language support to tools/text_processing_deployment/pynini_export.py.

PR Type:

• New Feature
• Bugfix
• Documentation
• Test

If you haven't finished some of the above items you can still open "Draft" PR.

[zoobereq]

Here I may handle capitalization as part of the grammar. Again, doing so helps to reduce the size of the graph.

[zoobereq]

You can use this to capitalize chars directly in the verbalizer/tagger grammars.

[zoobereq]

Looks fine to me.

Just a few general comments:

• After having both Spanish and French grammars consistently blow up my memory, I try to make the graphs as small as possible. A number of my suggestions points to places where the graph can be made smaller.
• If you have time to make the above changes -- excellent. Otherwise, if the checks are passing, we can leave them as is.
• Are there certain classes that this TN system won't be supporting? If so, it would be helpful to mention that in the PR description and/or add it to the description of the tagger and verbalizer.
• Any line that's commented out that itself is not a comment, is generally understood as and optional component that can be included if needed. If that's not the case, it's safe to remove it.

[zoobereq]

Looks good here. Let's wait until CI checks pass.

[zoobereq]

Looks ready to merge.