Add Cjk width token filter #7917
Conversation
Signed-off-by: AntonEliatra <[email protected]>
Signed-off-by: AntonEliatra <[email protected]>
Signed-off-by: AntonEliatra <[email protected]>
…into cjk-width-token-filter
Signed-off-by: AntonEliatra <[email protected]>
AntonEliatra
requested review from
hdhalter,
kolchfa-aws,
Naarcha-AWS,
vagimeli,
AMoo-Miki,
natebower,
dlvenable and
epugh
as code owners
|
Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged. Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer. When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review. |
|
@udabhas Could you review this PR for technical accuracy? |
|
@varun-lodaya The documentation is awaiting tech review and approval, which is delaying progress. Could you please suggest alternative reviewers who can assist with this task in a timely manner? We're eager to move this forward. Thank you. |
vagimeli
added
4 - Doc review
Thank you for the review. Moving forward. @kolchfa-aws This PR is ready for doc review at your availability. |
Signed-off-by: AntonEliatra <[email protected]>
|
|
||
| The CJK Width token filter in OpenSearch normalizes Chinese, Japanese, and Korean (CJK) tokens by converting full-width ASCII Character to their standard (half-width) ASCII equivalents and half-width Katakana characters to their full-width. | ||
|
|
||
| - **Converting full-width ASCII Character**: In CJK texts, ASCII characters (such as letters and numbers) can appear in full-width form, which occupies the space of two half-width characters. Full-width ASCII characters are typically used in East Asian typography to align with the width of CJK characters. However, for the purpose of indexing and searching, these full-width characters need to be normalized to their standard (half-width) ASCII equivalents. |
There was a problem hiding this comment.
Could you make these headings level 3?
|
|
||
| See following example: | ||
|
|
||
| Full-Width: ＡＢＣＤＥ １２３４５ |
There was a problem hiding this comment.
I think these are better formatted as a code block and aligned so the letter A is approximately at the same column in both lines (lines 16 and 17)
Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: AntonEliatra <[email protected]>
Signed-off-by: AntonEliatra <[email protected]>
|
@kolchfa-aws thank you for the suggestions, its been updated now |
Signed-off-by: AntonEliatra <[email protected]>
Signed-off-by: kolchfa-aws <[email protected]>
There was a problem hiding this comment.
@AntonEliatra @kolchfa-aws Please see my changes and let me know if you have any questions. Thanks!
Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: AntonEliatra <[email protected]>
* adding token filter page for cjk width #7875 Signed-off-by: AntonEliatra <[email protected]> * adding details to the page Signed-off-by: AntonEliatra <[email protected]> * adding details to the page Signed-off-by: AntonEliatra <[email protected]> * Updating details as per comments Signed-off-by: AntonEliatra <[email protected]> * Update cjk-width.md Signed-off-by: AntonEliatra <[email protected]> * Apply suggestions from code review Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: AntonEliatra <[email protected]> * Update cjk-width.md Signed-off-by: AntonEliatra <[email protected]> * Update cjk-width.md Signed-off-by: AntonEliatra <[email protected]> * Apply suggestions from code review Signed-off-by: kolchfa-aws <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: AntonEliatra <[email protected]> --------- Signed-off-by: AntonEliatra <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> (cherry picked from commit 41b1b06) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* adding token filter page for cjk width opensearch-project#7875 Signed-off-by: AntonEliatra <[email protected]> * adding details to the page Signed-off-by: AntonEliatra <[email protected]> * adding details to the page Signed-off-by: AntonEliatra <[email protected]> * Updating details as per comments Signed-off-by: AntonEliatra <[email protected]> * Update cjk-width.md Signed-off-by: AntonEliatra <[email protected]> * Apply suggestions from code review Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: AntonEliatra <[email protected]> * Update cjk-width.md Signed-off-by: AntonEliatra <[email protected]> * Update cjk-width.md Signed-off-by: AntonEliatra <[email protected]> * Apply suggestions from code review Signed-off-by: kolchfa-aws <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: AntonEliatra <[email protected]> --------- Signed-off-by: AntonEliatra <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Noah Staveley <[email protected]>
Description
Add Cjk width token filter
Issues Resolved
Closes #7875
Version
all
Checklist
For more information on following Developer Certificate of Origin and signing off your commits, please check here.