Merge branch 'main' into children-aggregation

.github/CODEOWNERS

@@ -1 +1 @@
-*  @hdhalter @kolchfa-aws @Naarcha-AWS @vagimeli @AMoo-Miki @natebower @dlvenable @scrawfor99 @epugh
+*  @kolchfa-aws @Naarcha-AWS @vagimeli @AMoo-Miki @natebower @dlvenable @stephen-crawford @epugh

.github/PULL_REQUEST_TEMPLATE.md

@@ -2,7 +2,7 @@
 _Describe what this change achieves._
 
 ### Issues Resolved
-_List any issues this PR will resolve, e.g. Closes [...]._
+Closes #[_insert issue number_]
 
 ### Version
 _List the OpenSearch version to which this PR applies, e.g. 2.14, 2.12--2.14, or all._

.github/vale/styles/Vocab/OpenSearch/Products/accept.txt

@@ -1,14 +1,18 @@
 Active Directory
 Adoptium
+AI21 Labs Jurassic
 Amazon
 Amazon OpenSearch Serverless
 Amazon OpenSearch Service
 Amazon Bedrock
 Amazon SageMaker
 Ansible
+Anthropic Claude
 Auditbeat
 AWS Cloud
 Cohere Command
+Cohere Embed English
+Cohere Embed Multilingual
 Cognito
 Dashboards Query Language
 Data Prepper
@@ -76,7 +80,6 @@ Painless
 Peer Forwarder
 Performance Analyzer
 Piped Processing Language
-Point in Time
 Powershell
 Python
 PyTorch
@@ -87,6 +90,9 @@ RPM Package Manager
 Ruby
 Simple Schema for Observability
 Tableau
+Textract
+Titan Multimodal Embeddings
+Titan Text Embeddings
 TorchScript
 Tribuo
 VisBuilder

.github/vale/styles/Vocab/OpenSearch/Words/accept.txt

@@ -77,9 +77,11 @@ Levenshtein
 [Mm]ultivalued
 [Mm]ultiword
 [Nn]amespace
-[Oo]versamples?
+[Oo]ffline
 [Oo]nboarding
+[Oo]versamples?
 pebibyte
+p\d{2}
 [Pp]erformant
 [Pp]laintext
 [Pp]luggable
@@ -93,15 +95,18 @@ pebibyte
 [Pp]reprocess
 [Pp]retrain
 [Pp]seudocode
-[Quantiz](e|ation|ing|er)
+[Qq]uantiles?
+[Qq]uantiz(e|ation|ing|er)
 [Rr]ebalance
 [Rr]ebalancing
 [Rr]edownload
 [Rr]eenable
 [Rr]eindex
 [Rr]eingest
+[Rr]eprovision(ed|ing)?
 [Rr]erank(er|ed|ing)?
 [Rr]epo
+[Rr]escor(e|ed|ing)?
 [Rr]ewriter
 [Rr]ollout
 [Rr]ollup
@@ -125,6 +130,7 @@ stdout
 [Ss]ubvector
 [Ss]ubwords?
 [Ss]uperset
+[Ss]uperadmins?
 [Ss]yslog
 tebibyte
 [Tt]emplated

.github/workflows/pr_checklist.yml

@@ -0,0 +1,43 @@
+name: PR Checklist
+
+on:
+  pull_request_target:
+    types: [opened]
+
+permissions:
+  pull-requests: write
+
+jobs:
+  add-checklist:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Comment PR with checklist
+        uses: peter-evans/create-or-update-comment@v3
+        with:
+          token: ${{ secrets.GITHUB_TOKEN }}
+          issue-number: ${{ github.event.pull_request.number }}
+          body: |
+            Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.
+            
+            Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a [maintainer](https://github.com/opensearch-project/documentation-website/blob/main/MAINTAINERS.md).
+
+            **When you're ready for doc review, tag the assignee of this PR**. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.
+
+      - name: Auto assign PR to repo owner
+        uses: actions/github-script@v6
+        with:
+          script: |
+            let assignee = context.payload.pull_request.user.login;
+            const prOwners = ['Naarcha-AWS', 'kolchfa-aws', 'vagimeli', 'natebower'];
+            
+            if (!prOwners.includes(assignee)) {
+              assignee = 'kolchfa-aws'
+            }
+
+            github.rest.issues.addAssignees({
+                issue_number: context.issue.number,
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                assignees: [assignee]
+              });

.gitignore

@@ -4,4 +4,7 @@ _site
 .DS_Store
 Gemfile.lock
 .idea
+*.iml
 .jekyll-cache
+.project
+vendor/bundle

API_STYLE_GUIDE.md

@@ -115,7 +115,7 @@ Include a table with these columns:
 Field | Data type | Description
 :--- | :--- | :--- 
 
-#### Example request
+## Example request
 
 Provide a sentence that describes what is shown in the example, followed by a cut-and-paste-ready API request in JSON format. Make sure that you test the request yourself in the Dashboards Dev Tools console to make sure it works. See the following examples.
 
@@ -139,7 +139,7 @@ POST _reindex
 }
 ```
 
-#### Example response
+## Example response
 
 Include a JSON example response to show what the API returns. See the following examples.
 

CONTRIBUTING.md

@@ -78,6 +78,8 @@ Follow these steps to set up your local copy of the repository:
 
 1. Navigate to your cloned repository.
 
+##### Building using locally installed packages 
+
 1. Install [Ruby](https://www.ruby-lang.org/en/) if you don't already have it. We recommend [RVM](https://rvm.io/), but you can use any method you prefer:
 
    ```
@@ -98,12 +100,20 @@ Follow these steps to set up your local copy of the repository:
    bundle install
    ```
 
+##### Building using containerization
+
+Assuming you have `docker-compose` installed, run the following command:
+
+   ```
+   docker compose -f docker-compose.dev.yml up
+   ```
+
 #### Troubleshooting
 
-If you encounter an error while trying to build the documentation website, find the error in the following troubleshooting list: 
+Try the following troubleshooting steps if you encounter an error when trying to build the documentation website:  
 
-- When running `rvm install 3.2` if you receive a `Error running '__rvm_make -j10'`, resolve this by running `rvm install 3.2.0 -C --with-openssl-dir=/opt/homebrew/opt/[email protected]` instead of `rvm install 3.2`.
-- If receive a `bundle install`: `An error occurred while installing posix-spawn (0.3.15), and Bundler cannot continue.` error when trying to run `bundle install`, resolve this by running `gem install posix-spawn -v 0.3.15 -- --with-cflags=\"-Wno-incompatible-function-pointer-types\"`. Then, run `bundle install`.
+- If you see the `Error running '__rvm_make -j10'` error when running `rvm install 3.2`, you can resolve it by running `rvm install 3.2.0 -C --with-openssl-dir=/opt/homebrew/opt/[email protected]` instead of `rvm install 3.2`.
+- If you see the `bundle install`: `An error occurred while installing posix-spawn (0.3.15), and Bundler cannot continue.` error when trying to run `bundle install`, you can resolve it by running `gem install posix-spawn -v 0.3.15 -- --with-cflags=\"-Wno-incompatible-function-pointer-types\"` and then `bundle install`.
 
 
 

MAINTAINERS.md

@@ -6,12 +6,17 @@ This document lists the maintainers in this repo. See [opensearch-project/.githu
 
 | Maintainer       | GitHub ID                                       | Affiliation |
 | ---------------- | ----------------------------------------------- | ----------- |
-| Heather Halter   | [hdhalter](https://github.com/hdhalter)         | Amazon      |
 | Fanit Kolchina   | [kolchfa-aws](https://github.com/kolchfa-aws)   | Amazon      |
 | Nate Archer      | [Naarcha-AWS](https://github.com/Naarcha-AWS)   | Amazon      |
 | Nathan Bower     | [natebower](https://github.com/natebower)       | Amazon      |
 | Melissa Vagi     | [vagimeli](https://github.com/vagimeli)         | Amazon      |
 | Miki Barahmand   | [AMoo-Miki](https://github.com/AMoo-Miki)       | Amazon      |
 | David Venable    | [dlvenable](https://github.com/dlvenable)       | Amazon      | 
-| Stephen Crawford | [scraw99](https://github.com/scrawfor99)        | Amazon      |
+| Stephen Crawford | [stephen-crawford](https://github.com/stephen-crawford) | Amazon      |
 | Eric Pugh        | [epugh](https://github.com/epugh)               | OpenSource Connections  | 
+
+## Emeritus
+
+| Maintainer       | GitHub ID                                       | Affiliation |
+| ---------------- | ----------------------------------------------- | ----------- |
+| Heather Halter   | [hdhalter](https://github.com/hdhalter)         | Amazon      |

README.md

@@ -21,7 +21,6 @@ The following resources provide important guidance regarding contributions to th
 
 If you encounter problems or have questions when contributing to the documentation, these people can help:
 
-- [hdhalter](https://github.com/hdhalter)
 - [kolchfa-aws](https://github.com/kolchfa-aws)
 - [Naarcha-AWS](https://github.com/Naarcha-AWS)
 - [vagimeli](https://github.com/vagimeli)

_about/index.md

@@ -22,16 +22,21 @@ This section contains documentation for OpenSearch and OpenSearch Dashboards.
 
 ## Getting started
 
-- [Intro to OpenSearch]({{site.url}}{{site.baseurl}}/intro/)
-- [Quickstart]({{site.url}}{{site.baseurl}}/quickstart/)
+To get started, explore the following documentation:
+
+- [Getting started guide]({{site.url}}{{site.baseurl}}/getting-started/): 
+  - [Intro to OpenSearch]({{site.url}}{{site.baseurl}}/getting-started/intro/)
+  - [Installation quickstart]({{site.url}}{{site.baseurl}}/getting-started/quickstart/)
+  - [Communicate with OpenSearch]({{site.url}}{{site.baseurl}}/getting-started/communicate/)
+  - [Ingest data]({{site.url}}{{site.baseurl}}/getting-started/ingest-data/)
+  - [Search data]({{site.url}}{{site.baseurl}}/getting-started/search-data/)
+  - [Getting started with OpenSearch security]({{site.url}}{{site.baseurl}}/getting-started/security/)
 - [Install OpenSearch]({{site.url}}{{site.baseurl}}/install-and-configure/install-opensearch/index/)
 - [Install OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/install-and-configure/install-dashboards/index/)
-- [See the FAQ](https://opensearch.org/faq)
+- [FAQ](https://opensearch.org/faq)
 
 ## Why use OpenSearch?
 
-With OpenSearch, you can perform the following use cases:
-
 <table style="table-layout: auto ; width: 100%;">
 <tbody>
 <tr style="text-align: center; vertical-align:center;">
@@ -41,35 +46,38 @@ With OpenSearch, you can perform the following use cases:
 <td><img src="{{site.url}}{{site.baseurl}}/images/4_tracking.png" class="no-border" alt="Operational health tracking" height="100"/></td>
 </tr>
 <tr style="text-align: left; vertical-align:top; font-weight: bold; color: rgb(0,59,92)">
-<td>Fast, Scalable Full-text Search</td>
-<td>Application and Infrastructure Monitoring</td>
-<td>Security and Event Information Management</td>
-<td>Operational Health Tracking</td>
+<td>Fast, scalable full-text search</td>
+<td>Application and infrastructure monitoring</td>
+<td>Security and event information management</td>
+<td>Operational health tracking</td>
 </tr>
 <tr style="text-align: left; vertical-align:top;">
 <td>Help users find the right information within your application, website, or data lake catalog. </td>
-<td>Easily store and analyze log data, and set automated alerts for underperformance.</td>
+<td>Easily store and analyze log data, and set automated alerts for performance issues.</td>
 <td>Centralize logs to enable real-time security monitoring and forensic analysis.</td>
-<td>Use observability logs, metrics, and traces to monitor your applications and business in real time.</td>
+<td>Use observability logs, metrics, and traces to monitor your applications in real time.</td>
 </tr>
 </tbody>
 </table>
 
-**Additional features and plugins:**
+## Key features
+
+OpenSearch provides several features to help index, secure, monitor, and analyze your data:
 
-OpenSearch has several features and plugins to help index, secure, monitor, and analyze your data. Most OpenSearch plugins have corresponding OpenSearch Dashboards plugins that provide a convenient, unified user interface.
-- [Anomaly detection]({{site.url}}{{site.baseurl}}/monitoring-plugins/ad/) - Identify atypical data and receive automatic notifications
-- [KNN]({{site.url}}{{site.baseurl}}/search-plugins/knn/) - Find “nearest neighbors” in your vector data
-- [Performance Analyzer]({{site.url}}{{site.baseurl}}/monitoring-plugins/pa/) - Monitor and optimize your cluster
-- [SQL]({{site.url}}{{site.baseurl}}/search-plugins/sql/index/) - Use SQL or a piped processing language to query your data
-- [Index State Management]({{site.url}}{{site.baseurl}}/im-plugin/) - Automate index operations
-- [ML Commons plugin]({{site.url}}{{site.baseurl}}/ml-commons-plugin/index/) - Train and execute machine-learning models
-- [Asynchronous search]({{site.url}}{{site.baseurl}}/search-plugins/async/) - Run search requests in the background
-- [Cross-cluster replication]({{site.url}}{{site.baseurl}}/replication-plugin/index/) - Replicate your data across multiple OpenSearch clusters
+- [Anomaly detection]({{site.url}}{{site.baseurl}}/monitoring-plugins/ad/) -- Identify atypical data and receive automatic notifications.
+- [SQL]({{site.url}}{{site.baseurl}}/search-plugins/sql/index/) -- Use SQL or a Piped Processing Language (PPL) to query your data.
+- [Index State Management]({{site.url}}{{site.baseurl}}/im-plugin/) -- Automate index operations.
+- [Search methods]({{site.url}}{{site.baseurl}}/search-plugins/knn/) -- From traditional lexical search to advanced vector and hybrid search, discover the optimal search method for your use case.
+- [Machine learning]({{site.url}}{{site.baseurl}}/ml-commons-plugin/index/) -- Integrate machine learning models into your workloads.
+- [Workflow automation]({{site.url}}{{site.baseurl}}/automating-configurations/index/) -- Automate complex OpenSearch setup and preprocessing tasks.
+- [Performance evaluation]({{site.url}}{{site.baseurl}}/monitoring-plugins/pa/) -- Monitor and optimize your cluster.
+- [Asynchronous search]({{site.url}}{{site.baseurl}}/search-plugins/async/) -- Run search requests in the background.
+- [Cross-cluster replication]({{site.url}}{{site.baseurl}}/replication-plugin/index/) -- Replicate your data across multiple OpenSearch clusters.
 
 
 ## The secure path forward
-OpenSearch includes a demo configuration so that you can get up and running quickly, but before using OpenSearch in a production environment, you must [configure the Security plugin manually]({{site.url}}{{site.baseurl}}/security/configuration/index/) with your own certificates, authentication method, users, and passwords.
+
+OpenSearch includes a demo configuration so that you can get up and running quickly, but before using OpenSearch in a production environment, you must [configure the Security plugin manually]({{site.url}}{{site.baseurl}}/security/configuration/index/) with your own certificates, authentication method, users, and passwords. To get started, see [Getting started with OpenSearch security]({{site.url}}{{site.baseurl}}/getting-started/security/).
 
 ## Looking for the Javadoc?
 

_about/version-history.md

@@ -9,6 +9,9 @@ permalink: /version-history/
 
 OpenSearch version | Release highlights | Release date  
 :--- | :--- | :--- 
+[2.17.1](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.17.1.md) | Includes bug fixes for ML Commons, anomaly detection, k-NN, and security analytics. Adds various infrastructure and maintenance updates. For a full list of release highlights, see the Release Notes. | 1 October 2024
+[2.17.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.17.0.md) | Includes disk-optimized vector search, binary quantization, and byte vector encoding in k-NN. Adds asynchronous batch ingestion for ML tasks. Provides search and query performance enhancements and a new custom trace source in trace analytics. Includes application-based configuration templates. For a full list of release highlights, see the Release Notes. | 17 September 2024
+[2.16.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.16.0.md) | Includes built-in byte vector quantization and binary vector support in k-NN. Adds new sort, split, and ML inference search processors for search pipelines. Provides application-based configuration templates and additional plugins to integrate multiple data sources in OpenSearch Dashboards. Includes an experimental Batch Predict ML Commons API. For a full list of release highlights, see the Release Notes. | 06 August 2024
 [2.15.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.15.0.md) | Includes parallel ingestion processing, SIMD support for exact search, and the ability to disable doc values for the k-NN field. Adds wildcard and derived field types. Improves performance for single-cardinality aggregations, rolling upgrades to remote-backed clusters, and more metrics for top N queries. For a full list of release highlights, see the Release Notes. | 25 June 2024
 [2.14.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.14.0.md) | Includes performance improvements to hybrid search and date histogram queries with multi-range traversal, ML model integration within the Ingest API, semantic cache for LangChain applications, low-level vector query interface for neural sparse queries, and improved k-NN search filtering. Provides an experimental tiered cache feature. For a full list of release highlights, see the Release Notes. | 14 May 2024
 [2.13.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.13.0.md) | Makes agents and tools and the OpenSearch Assistant Toolkit generally available. Introduces vector quantization within OpenSearch. Adds LLM guardrails and hybrid search with aggregations. Adds the Bloom filter skipping index for Apache Spark data sources, I/O-based admission control, and the ability to add an alerting cluster that manages all alerting tasks. For a full list of release highlights, see the Release Notes. | 2 April 2024
@@ -30,6 +33,8 @@ OpenSearch version | Release highlights | Release date
 [2.0.1](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.0.1.md) | Includes bug fixes and maintenance updates for Alerting and Anomaly Detection. | 16 June 2022
 [2.0.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.0.0.md) | Includes document-level monitors for alerting, OpenSearch Notifications plugins, and Geo Map Tiles in OpenSearch Dashboards. Also adds support for Lucene 9 and bug fixes for all OpenSearch plugins. For a full list of release highlights, see the Release Notes. | 26 May 2022
 [2.0.0-rc1](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.0.0-rc1.md) | The Release Candidate for 2.0.0. This version allows you to preview the upcoming 2.0.0 release before the GA release. The preview release adds document-level alerting, support for Lucene 9, and the ability to use term lookup queries in document level security. | 03 May 2022
+[1.3.19](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-1.3.19.md) | Includes bug fixes and maintenance updates for OpenSearch security, OpenSearch security Dashboards, and anomaly detection. | 27 August 2024
+[1.3.18](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-1.3.18.md) | Includes maintenance updates for OpenSearch security. | 16 July 2024
 [1.3.17](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-1.3.17.md) | Includes maintenance updates for OpenSearch security and OpenSearch Dashboards security. | 06 June 2024
 [1.3.16](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-1.3.16.md) | Includes bug fixes and maintenance updates for OpenSearch security, index management, performance analyzer, and reporting. | 23 April 2024
 [1.3.15](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-1.3.15.md) | Includes bug fixes and maintenance updates for cross-cluster replication, SQL, OpenSearch Dashboards reporting, and alerting.  | 05 March 2024

_aggregations/bucket/adjacency-matrix.md

@@ -2,7 +2,6 @@
 layout: default
 title: Adjacency matrix
 parent: Bucket aggregations
-grand_parent: Aggregations
 nav_order: 10
 redirect_from:
   - /query-dsl/aggregations/bucket/adjacency-matrix/

_aggregations/bucket/date-histogram.md

@@ -2,7 +2,6 @@
 layout: default
 title: Date histogram
 parent: Bucket aggregations
-grand_parent: Aggregations
 nav_order: 20
 redirect_from:
   - /query-dsl/aggregations/bucket/date-histogram/

_aggregations/bucket/date-range.md

@@ -2,7 +2,6 @@
 layout: default
 title: Date range
 parent: Bucket aggregations
-grand_parent: Aggregations
 nav_order: 30
 redirect_from:
   - /query-dsl/aggregations/bucket/date-range/