Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add condition token filter docs #7923 #7950

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
+135 −1
Open

Add condition token filter docs #7923 #7950

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f0897ae
adding condition token filter docs #7923
AntonEliatra Aug 9, 2024
22dbf46
Merge branch 'main' into adding-conditional-token-filer
vagimeli Aug 26, 2024
b4deec0
Update condition.md
AntonEliatra Sep 12, 2024
af7d59f
Merge branch 'main' into adding-conditional-token-filer
vagimeli Oct 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
134 changes: 134 additions & 0 deletions _analyzers/token-filters/condition.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
---
layout: default
title: condition
parent: Token filters
nav_order: 70
---

# Condition token filter

The `condition` token filter in OpenSearch is a special type of filter that allows you to apply other token filters conditionally based on certain criteria. This provides more control over when certain token filters should be applied during text analysis.
Multiple filters can be configured and only applied based on the conditions you define.
This token filter can be very useful for language-specific processing and handling of special characters.


## Parameters

There are two *required* parameters that have to be configured to use `condition` token filter:

`filter`: specifies which token filters should be applied to the tokens when the specified condition (defined by the script parameter) is met.

`script`: defines the condition that needs to be met for the filters specified in the filter parameter to be applied. This condition is expressed in the form of an inline script.


## Example

The following example request creates a new index named `my_conditional_index` and configures an analyzer with `condition` filter. This filter applies `lowercase` filter to any tokens which contain characters "UM":

```json
PUT /my_conditional_index
{
"settings": {
"analysis": {
"filter": {
"my_conditional_filter": {
"type": "condition",
"filter": ["lowercase"],
"script": {
"source": "token.getTerm().toString().contains('UM')"

Check warning on line 38 in _analyzers/token-filters/condition.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.AcronymParentheses] 'UM': Spell out acronyms the first time that you use them on a page and follow them with the acronym in parentheses. Subsequently, use the acronym alone.

Raw Output:
{"message": "[OpenSearch.AcronymParentheses] 'UM': Spell out acronyms the first time that you use them on a page and follow them with the acronym in parentheses. Subsequently, use the acronym alone.", "location": {"path": "_analyzers/token-filters/condition.md", "range": {"start": {"line": 38, "column": 61}}}, "severity": "WARNING"}
}
}
},
"analyzer": {
"my_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"my_conditional_filter"
]
}
}
}
}
}
```
{% include copy-curl.html %}

## Generated tokens

Use the following request to examine the tokens generated using the analyzer:

```json
GET /my_conditional_index/_analyze
{
"analyzer": "my_analyzer",
"text": "THE BLACK CAT JUMPS OVER A LAZY DOG"
}
```
{% include copy-curl.html %}

The response contains the generated tokens:

```json
{
"tokens": [
{
"token": "THE",
"start_offset": 0,
"end_offset": 3,
"type": "<ALPHANUM>",
"position": 0
},
{
"token": "BLACK",
"start_offset": 4,
"end_offset": 9,
"type": "<ALPHANUM>",
"position": 1
},
{
"token": "CAT",
"start_offset": 10,
"end_offset": 13,
"type": "<ALPHANUM>",
"position": 2
},
{
"token": "jumps",
"start_offset": 14,
"end_offset": 19,
"type": "<ALPHANUM>",
"position": 3
},
{
"token": "OVER",
"start_offset": 20,
"end_offset": 24,
"type": "<ALPHANUM>",
"position": 4
},
{
"token": "A",
"start_offset": 25,
"end_offset": 26,
"type": "<ALPHANUM>",
"position": 5
},
{
"token": "LAZY",
"start_offset": 27,
"end_offset": 31,
"type": "<ALPHANUM>",
"position": 6
},
{
"token": "DOG",
"start_offset": 32,
"end_offset": 35,
"type": "<ALPHANUM>",
"position": 7
}
]
}
```

2 changes: 1 addition & 1 deletion _analyzers/token-filters/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Token filter | Underlying Lucene token filter| Description
[`cjk_width`]({{site.url}}{{site.baseurl}}/analyzers/token-filters/cjk-width/) | [CJKWidthFilter](https://lucene.apache.org/core/9_10_0/analysis/common/org/apache/lucene/analysis/cjk/CJKWidthFilter.html) | Normalizes Chinese, Japanese, and Korean (CJK) tokens according to the following rules: <br> - Folds full-width ASCII character variants into their equivalent basic Latin characters. <br> - Folds half-width katakana character variants into their equivalent kana characters.
`classic` | [ClassicFilter](https://lucene.apache.org/core/9_10_0/analysis/common/org/apache/lucene/analysis/classic/ClassicFilter.html) | Performs optional post-processing on the tokens generated by the classic tokenizer. Removes possessives (`'s`) and removes `.` from acronyms.
`common_grams` | [CommonGramsFilter](https://lucene.apache.org/core/9_10_0/analysis/common/org/apache/lucene/analysis/commongrams/CommonGramsFilter.html) | Generates bigrams for a list of frequently occurring terms. The output contains both single terms and bigrams.
`conditional` | [ConditionalTokenFilter](https://lucene.apache.org/core/9_10_0/analysis/common/org/apache/lucene/analysis/miscellaneous/ConditionalTokenFilter.html) | Applies an ordered list of token filters to tokens that match the conditions provided in a script.
[`conditional`]({{site.url}}{{site.baseurl}}/analyzers/token-filters/condition/) | [ConditionalTokenFilter](https://lucene.apache.org/core/9_10_0/analysis/common/org/apache/lucene/analysis/miscellaneous/ConditionalTokenFilter.html) | Applies an ordered list of token filters to tokens that match the conditions provided in a script.
`decimal_digit` | [DecimalDigitFilter](https://lucene.apache.org/core/9_10_0/analysis/common/org/apache/lucene/analysis/core/DecimalDigitFilter.html) | Converts all digits in the Unicode decimal number general category to basic Latin digits (0--9).
`delimited_payload` | [DelimitedPayloadTokenFilter](https://lucene.apache.org/core/9_10_0/analysis/common/org/apache/lucene/analysis/payloads/DelimitedPayloadTokenFilter.html) | Separates a token stream into tokens with corresponding payloads, based on a provided delimiter. A token consists of all characters before the delimiter, and a payload consists of all characters after the delimiter. For example, if the delimiter is `|`, then for the string `foo|bar`, `foo` is the token and `bar` is the payload.
[`delimited_term_freq`]({{site.url}}{{site.baseurl}}/analyzers/token-filters/delimited-term-frequency/) | [DelimitedTermFrequencyTokenFilter](https://lucene.apache.org/core/9_7_0/analysis/common/org/apache/lucene/analysis/miscellaneous/DelimitedTermFrequencyTokenFilter.html) | Separates a token stream into tokens with corresponding term frequencies, based on a provided delimiter. A token consists of all characters before the delimiter, and a term frequency is the integer after the delimiter. For example, if the delimiter is `|`, then for the string `foo|5`, `foo` is the token and `5` is the term frequency.
Expand Down
Loading