From 2ea5e8070bdb5551e40ad929776c04241bdc3f98 Mon Sep 17 00:00:00 2001 From: Marc Handalian Date: Wed, 11 Sep 2024 10:32:15 -0700 Subject: [PATCH 1/7] update aggregation support in limitations and add sample req/response. Signed-off-by: Marc Handalian --- _field-types/supported-field-types/derived.md | 72 ++++++++++++++++++- 1 file changed, 71 insertions(+), 1 deletion(-) diff --git a/_field-types/supported-field-types/derived.md b/_field-types/supported-field-types/derived.md index d989c3e4a4..17271d5394 100644 --- a/_field-types/supported-field-types/derived.md +++ b/_field-types/supported-field-types/derived.md @@ -28,7 +28,8 @@ Despite the potential performance impact of query-time computations, the flexibi Currently, derived fields have the following limitations: -- **Aggregation, scoring, and sorting**: Not yet supported. +- **Scoring and sorting**: Not yet supported. +- **Aggregations are supported with limitations**: Unsupported agg types: any Geo based Aggregation, Significant Terms/Text and Scripted Metric - **Dashboard support**: These fields are not displayed in the list of available fields in OpenSearch Dashboards. However, you can still use them for filtering if you know the derived field name. - **Chained derived fields**: One derived field cannot be used to define another derived field. - **Join field type**: Derived fields are not supported for the [join field type]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/join/). @@ -541,6 +542,75 @@ The response specifies highlighting in the `url` field: ``` +## Aggregations + +Most aggregation types are supported with derived fields since 2.17. Geo Aggregations, Significant Terms/Text and Scripted Metric are not supported. +For example, the following request creates a simple terms aggregation on the `status` derived field: + +```json +POST /logs/_search +{ + "size": 0, + "aggs": { + "methods": { + "terms": { + "field": "method" + } + } + } +} +``` +{% include copy-curl.html %} +The response contains the following buckets: + +
+ + Response + + {: .text-delta} + +```json +{ + "took" : 12, + "timed_out" : false, + "_shards" : { + "total" : 1, + "successful" : 1, + "skipped" : 0, + "failed" : 0 + }, + "hits" : { + "total" : { + "value" : 5, + "relation" : "eq" + }, + "max_score" : null, + "hits" : [ ] + }, + "aggregations" : { + "methods" : { + "doc_count_error_upper_bound" : 0, + "sum_other_doc_count" : 0, + "buckets" : [ + { + "key" : "GET", + "doc_count" : 2 + }, + { + "key" : "POST", + "doc_count" : 2 + }, + { + "key" : "DELETE", + "doc_count" : 1 + } + ] + } + } +} +``` +
+ ## Performance Derived fields are not indexed but are computed dynamically by retrieving values from the `_source` field or doc values. Thus, they run more slowly. To improve performance, try the following: From 21ab396a567551d400540dba87ce2d8f8fbed28c Mon Sep 17 00:00:00 2001 From: Marc Handalian Date: Wed, 11 Sep 2024 10:32:41 -0700 Subject: [PATCH 2/7] remove concurrent search limitation Signed-off-by: Marc Handalian --- _field-types/supported-field-types/derived.md | 1 - 1 file changed, 1 deletion(-) diff --git a/_field-types/supported-field-types/derived.md b/_field-types/supported-field-types/derived.md index 17271d5394..54bb7bc3e5 100644 --- a/_field-types/supported-field-types/derived.md +++ b/_field-types/supported-field-types/derived.md @@ -33,7 +33,6 @@ Currently, derived fields have the following limitations: - **Dashboard support**: These fields are not displayed in the list of available fields in OpenSearch Dashboards. However, you can still use them for filtering if you know the derived field name. - **Chained derived fields**: One derived field cannot be used to define another derived field. - **Join field type**: Derived fields are not supported for the [join field type]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/join/). -- **Concurrent segment search**: Derived fields are not supported for [concurrent segment search]({{site.url}}{{site.baseurl}}/search-plugins/concurrent-segment-search/). We are planning to address these limitations in future versions. From 57d4bf1316bc55c84618e41889ecfbb7a9571a0b Mon Sep 17 00:00:00 2001 From: Marc Handalian Date: Wed, 11 Sep 2024 10:49:19 -0700 Subject: [PATCH 3/7] minor updates Signed-off-by: Marc Handalian --- _field-types/supported-field-types/derived.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_field-types/supported-field-types/derived.md b/_field-types/supported-field-types/derived.md index 54bb7bc3e5..d5eacd9c58 100644 --- a/_field-types/supported-field-types/derived.md +++ b/_field-types/supported-field-types/derived.md @@ -29,7 +29,7 @@ Despite the potential performance impact of query-time computations, the flexibi Currently, derived fields have the following limitations: - **Scoring and sorting**: Not yet supported. -- **Aggregations are supported with limitations**: Unsupported agg types: any Geo based Aggregation, Significant Terms/Text and Scripted Metric +- **With 2.17 Aggregations are supported with limitations**: Unsupported agg types: Geo, Significant Terms/Text and Scripted Metric - **Dashboard support**: These fields are not displayed in the list of available fields in OpenSearch Dashboards. However, you can still use them for filtering if you know the derived field name. - **Chained derived fields**: One derived field cannot be used to define another derived field. - **Join field type**: Derived fields are not supported for the [join field type]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/join/). @@ -544,7 +544,7 @@ The response specifies highlighting in the `url` field: ## Aggregations Most aggregation types are supported with derived fields since 2.17. Geo Aggregations, Significant Terms/Text and Scripted Metric are not supported. -For example, the following request creates a simple terms aggregation on the `status` derived field: +For example, the following request creates a simple terms aggregation on the `method` derived field: ```json POST /logs/_search From a993bb93c1ad52b30a1b52640c6670dd1e128dfc Mon Sep 17 00:00:00 2001 From: Marc Handalian Date: Wed, 11 Sep 2024 11:43:49 -0700 Subject: [PATCH 4/7] fix style job error Signed-off-by: Marc Handalian --- _field-types/supported-field-types/derived.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/supported-field-types/derived.md b/_field-types/supported-field-types/derived.md index d5eacd9c58..78326ce670 100644 --- a/_field-types/supported-field-types/derived.md +++ b/_field-types/supported-field-types/derived.md @@ -29,7 +29,7 @@ Despite the potential performance impact of query-time computations, the flexibi Currently, derived fields have the following limitations: - **Scoring and sorting**: Not yet supported. -- **With 2.17 Aggregations are supported with limitations**: Unsupported agg types: Geo, Significant Terms/Text and Scripted Metric +- **With 2.17 Aggregations are supported with limitations**: Unsupported aggregation types: Geo, Significant Terms/Text and Scripted Metric - **Dashboard support**: These fields are not displayed in the list of available fields in OpenSearch Dashboards. However, you can still use them for filtering if you know the derived field name. - **Chained derived fields**: One derived field cannot be used to define another derived field. - **Join field type**: Derived fields are not supported for the [join field type]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/join/). From 95173fbbf0c1a4b3c8b37dedc3c771eab185d606 Mon Sep 17 00:00:00 2001 From: Fanit Kolchina Date: Thu, 12 Sep 2024 14:25:51 -0400 Subject: [PATCH 5/7] Doc review Signed-off-by: Fanit Kolchina --- _field-types/supported-field-types/derived.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/_field-types/supported-field-types/derived.md b/_field-types/supported-field-types/derived.md index 78326ce670..53f8d15216 100644 --- a/_field-types/supported-field-types/derived.md +++ b/_field-types/supported-field-types/derived.md @@ -29,7 +29,7 @@ Despite the potential performance impact of query-time computations, the flexibi Currently, derived fields have the following limitations: - **Scoring and sorting**: Not yet supported. -- **With 2.17 Aggregations are supported with limitations**: Unsupported aggregation types: Geo, Significant Terms/Text and Scripted Metric +- **Aggregations**: Starting with OpenSearch version 2.17, derived fields support most aggregation types. The following aggregations are not supported: geographic, significant terms, significant text, and scripted metric. - **Dashboard support**: These fields are not displayed in the list of available fields in OpenSearch Dashboards. However, you can still use them for filtering if you know the derived field name. - **Chained derived fields**: One derived field cannot be used to define another derived field. - **Join field type**: Derived fields are not supported for the [join field type]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/join/). @@ -543,8 +543,12 @@ The response specifies highlighting in the `url` field: ## Aggregations -Most aggregation types are supported with derived fields since 2.17. Geo Aggregations, Significant Terms/Text and Scripted Metric are not supported. -For example, the following request creates a simple terms aggregation on the `method` derived field: +Starting with OpenSearch version 2.17, derived fields support most aggregation types. + +Geographic, significant terms, significant text, and scripted metric aggregations are not supported. +{: .note} + +For example, the following request creates a simple `terms` aggregation on the `method` derived field: ```json POST /logs/_search @@ -560,6 +564,7 @@ POST /logs/_search } ``` {% include copy-curl.html %} + The response contains the following buckets:
From 2c6eab06354b7b204f88bd4a42e794b44505b769 Mon Sep 17 00:00:00 2001 From: Fanit Kolchina Date: Thu, 12 Sep 2024 14:36:14 -0400 Subject: [PATCH 6/7] Clarify the list of unsupported geo aggregations Signed-off-by: Fanit Kolchina --- _field-types/supported-field-types/derived.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/supported-field-types/derived.md b/_field-types/supported-field-types/derived.md index 53f8d15216..30de510099 100644 --- a/_field-types/supported-field-types/derived.md +++ b/_field-types/supported-field-types/derived.md @@ -29,7 +29,7 @@ Despite the potential performance impact of query-time computations, the flexibi Currently, derived fields have the following limitations: - **Scoring and sorting**: Not yet supported. -- **Aggregations**: Starting with OpenSearch version 2.17, derived fields support most aggregation types. The following aggregations are not supported: geographic, significant terms, significant text, and scripted metric. +- **Aggregations**: Starting with OpenSearch version 2.17, derived fields support most aggregation types. The following aggregations are not supported: geographic (geodistance, geohash grid, geohex grid, geotile grid, geobounds, geocentroid), significant terms, significant text, and scripted metric. - **Dashboard support**: These fields are not displayed in the list of available fields in OpenSearch Dashboards. However, you can still use them for filtering if you know the derived field name. - **Chained derived fields**: One derived field cannot be used to define another derived field. - **Join field type**: Derived fields are not supported for the [join field type]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/join/). From ec592f29731dc55c2311666ee0847c191344f9b6 Mon Sep 17 00:00:00 2001 From: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Date: Fri, 13 Sep 2024 09:00:18 -0400 Subject: [PATCH 7/7] Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --- _field-types/supported-field-types/derived.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_field-types/supported-field-types/derived.md b/_field-types/supported-field-types/derived.md index 30de510099..cb358f47f9 100644 --- a/_field-types/supported-field-types/derived.md +++ b/_field-types/supported-field-types/derived.md @@ -29,7 +29,7 @@ Despite the potential performance impact of query-time computations, the flexibi Currently, derived fields have the following limitations: - **Scoring and sorting**: Not yet supported. -- **Aggregations**: Starting with OpenSearch version 2.17, derived fields support most aggregation types. The following aggregations are not supported: geographic (geodistance, geohash grid, geohex grid, geotile grid, geobounds, geocentroid), significant terms, significant text, and scripted metric. +- **Aggregations**: Starting with OpenSearch 2.17, derived fields support most aggregation types. The following aggregations are not supported: geographic (geodistance, geohash grid, geohex grid, geotile grid, geobounds, geocentroid), significant terms, significant text, and scripted metric. - **Dashboard support**: These fields are not displayed in the list of available fields in OpenSearch Dashboards. However, you can still use them for filtering if you know the derived field name. - **Chained derived fields**: One derived field cannot be used to define another derived field. - **Join field type**: Derived fields are not supported for the [join field type]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/join/). @@ -543,7 +543,7 @@ The response specifies highlighting in the `url` field: ## Aggregations -Starting with OpenSearch version 2.17, derived fields support most aggregation types. +Starting with OpenSearch 2.17, derived fields support most aggregation types. Geographic, significant terms, significant text, and scripted metric aggregations are not supported. {: .note}