From 49f6c517887e99dd9cd800bb0099df852de09f4e Mon Sep 17 00:00:00 2001 From: mdbirnstiehl Date: Wed, 8 Jan 2025 13:19:44 -0600 Subject: [PATCH 1/5] add-e5-model --- docs/en/observability/observability-ai-assistant.asciidoc | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/en/observability/observability-ai-assistant.asciidoc b/docs/en/observability/observability-ai-assistant.asciidoc index 60295e06a0..9c9d537a0d 100644 --- a/docs/en/observability/observability-ai-assistant.asciidoc +++ b/docs/en/observability/observability-ai-assistant.asciidoc @@ -152,7 +152,7 @@ Search connectors are only needed when importing external data into the Knowledg {ref}/es-connectors.html[Connectors] allow you to index content from external sources thereby making it available for the AI Assistant. This can greatly improve the relevance of the AI Assistant’s responses. Data can be integrated from sources such as GitHub, Confluence, Google Drive, Jira, AWS S3, Microsoft Teams, Slack, and more. -UI affordances for creating and managing search connectors are available in the Search Solution in {kib}. +UI affordances for creating and managing search connectors are available in the Search Solution in {kib}. You can also use the {es} {ref}/connector-apis.html[Connector APIs] to create and manage search connectors. The infrastructure for deploying connectors can be managed by Elastic or self-managed. Managed connectors require an {enterprise-search-ref}/server.html[Enterprise Search] server connected to the Elastic Stack. Self-managed connectors are run on your own infrastructure and don't require the Enterprise Search service. @@ -173,21 +173,21 @@ For example, if you create a {ref}/es-connectors-github.html[GitHub connector] y + Learn more about configuring and {ref}/es-connectors-usage.html[using connectors] in the Elasticsearch documentation. + -. Create a pipeline and process the data with ELSER. +. Create a pipeline and process the data with a machine learning model. + To create the embeddings needed by the AI Assistant (weights and tokens into a sparse vector field), you have to create an *ML Inference Pipeline*: + .. Open the previously created connector and select the *Pipelines* tab. .. Select *Copy and customize* button at the `Unlock your custom pipelines` box. .. Select *Add Inference Pipeline* button at the `Machine Learning Inference Pipelines` box. -.. Select *ELSER (Elastic Learned Sparse EncodeR)* ML model to add the necessary embeddings to the data. +.. Select your preferred machine learning model (ELSER, E5). .. Select the fields that need to be evaluated as part of the inference pipeline. .. Test and save the inference pipeline and the overall pipeline. . Sync the data. + Once the pipeline is set up, perform a *Full Content Sync* of the connector. The inference pipeline will process the data as follows: + -* As data comes in, ELSER is applied to the data, and embeddings (weights and tokens into a sparse vector field) are added to capture semantic meaning and context of the data. +* As data comes in, the machine learning model (ELSER, E5) is applied to the data, and embeddings (weights and tokens into a sparse vector field) are added to capture semantic meaning and context of the data. * When you look at the documents that are ingested, you can see how the weights and token are added to the `predicted_value` field in the documents. . Check if AI Assistant can use the index (optional). + From e8bd422a9bb5ebe50dcc09aeac8248633ca36db4 Mon Sep 17 00:00:00 2001 From: mdbirnstiehl Date: Thu, 9 Jan 2025 10:40:52 -0600 Subject: [PATCH 2/5] add new models --- docs/en/observability/observability-ai-assistant.asciidoc | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/en/observability/observability-ai-assistant.asciidoc b/docs/en/observability/observability-ai-assistant.asciidoc index 9c9d537a0d..576977148f 100644 --- a/docs/en/observability/observability-ai-assistant.asciidoc +++ b/docs/en/observability/observability-ai-assistant.asciidoc @@ -173,26 +173,28 @@ For example, if you create a {ref}/es-connectors-github.html[GitHub connector] y + Learn more about configuring and {ref}/es-connectors-usage.html[using connectors] in the Elasticsearch documentation. + -. Create a pipeline and process the data with a machine learning model. +. Create a pipeline and process the data with a machine learning model. When querying search connectors using `semantic_text`, you can use any model such as E5. For search connectors not using `semantic_text`, use ELSER. + To create the embeddings needed by the AI Assistant (weights and tokens into a sparse vector field), you have to create an *ML Inference Pipeline*: + .. Open the previously created connector and select the *Pipelines* tab. .. Select *Copy and customize* button at the `Unlock your custom pipelines` box. .. Select *Add Inference Pipeline* button at the `Machine Learning Inference Pipelines` box. -.. Select your preferred machine learning model (ELSER, E5). +.. Select your preferred machine learning model such as ELSER or E5. .. Select the fields that need to be evaluated as part of the inference pipeline. .. Test and save the inference pipeline and the overall pipeline. . Sync the data. + Once the pipeline is set up, perform a *Full Content Sync* of the connector. The inference pipeline will process the data as follows: + -* As data comes in, the machine learning model (ELSER, E5) is applied to the data, and embeddings (weights and tokens into a sparse vector field) are added to capture semantic meaning and context of the data. +* As data comes in, the machine learning model is applied to the data, and embeddings (weights and tokens into a sparse vector field) are added to capture semantic meaning and context of the data. * When you look at the documents that are ingested, you can see how the weights and token are added to the `predicted_value` field in the documents. . Check if AI Assistant can use the index (optional). + Ask something to the AI Assistant related with the indexed data. +Add + [discrete] [[obs-ai-interact]] == Interact with the AI Assistant From 8b66169c4513ef27cdc492de3ba9ac0123207286 Mon Sep 17 00:00:00 2001 From: mdbirnstiehl Date: Wed, 15 Jan 2025 15:26:37 -0600 Subject: [PATCH 3/5] add semantic field instructions --- .../observability-ai-assistant.asciidoc | 44 +++++++++++++------ 1 file changed, 31 insertions(+), 13 deletions(-) diff --git a/docs/en/observability/observability-ai-assistant.asciidoc b/docs/en/observability/observability-ai-assistant.asciidoc index 576977148f..94a661ecb0 100644 --- a/docs/en/observability/observability-ai-assistant.asciidoc +++ b/docs/en/observability/observability-ai-assistant.asciidoc @@ -172,28 +172,46 @@ If your {kib} Space doesn't include the Search solution you will have to create For example, if you create a {ref}/es-connectors-github.html[GitHub connector] you have to set a `name`, attach it to a new or existing `index`, add your `personal access token` and include the `list of repositories` to synchronize. + Learn more about configuring and {ref}/es-connectors-usage.html[using connectors] in the Elasticsearch documentation. -+ -. Create a pipeline and process the data with a machine learning model. When querying search connectors using `semantic_text`, you can use any model such as E5. For search connectors not using `semantic_text`, use ELSER. -+ -To create the embeddings needed by the AI Assistant (weights and tokens into a sparse vector field), you have to create an *ML Inference Pipeline*: -+ -.. Open the previously created connector and select the *Pipelines* tab. -.. Select *Copy and customize* button at the `Unlock your custom pipelines` box. -.. Select *Add Inference Pipeline* button at the `Machine Learning Inference Pipelines` box. -.. Select your preferred machine learning model such as ELSER or E5. -.. Select the fields that need to be evaluated as part of the inference pipeline. -.. Test and save the inference pipeline and the overall pipeline. + +After creating your connector, create the embeddings needed by the AI Assistant. You can do this using either <>, which requires the ELSER model, or <>, which can use any available model (ELSER, E5, or a custom model). + +[discrete] +[[obs-ai-search-connectors-ml-embeddings]] +==== Use machine learning pipelines to create AI Assistant embeddings + +To create the embeddings needed by the AI Assistant (weights and tokens into a sparse vector field) using an *ML Inference Pipeline*: + +. Open the previously created connector, and select the *Pipelines* tab. +. Select *Copy and customize* button at the `Unlock your custom pipelines` box. +. Select *Add Inference Pipeline* button at the `Machine Learning Inference Pipelines` box. +. Select *ELSER (Elastic Learned Sparse EncodeR)* ML model to add the necessary embeddings to the data. +. Select the fields that need to be evaluated as part of the inference pipeline. +. Test and save the inference pipeline and the overall pipeline. + +After creating the pipeline, complete the following steps: + . Sync the data. + Once the pipeline is set up, perform a *Full Content Sync* of the connector. The inference pipeline will process the data as follows: + -* As data comes in, the machine learning model is applied to the data, and embeddings (weights and tokens into a sparse vector field) are added to capture semantic meaning and context of the data. +* As data comes in, ELSER is applied to the data, and embeddings (weights and tokens into a sparse vector field) are added to capture semantic meaning and context of the data. * When you look at the documents that are ingested, you can see how the weights and token are added to the `predicted_value` field in the documents. . Check if AI Assistant can use the index (optional). + Ask something to the AI Assistant related with the indexed data. -Add +[discrete] +[[obs-ai-search-connectors-semantic-text]] +==== Use a `semantic_text` field type to create AI Assistant embeddings + +To create the embeddings needed by the AI Assistant using a {ref}/semantic-text.html[`semantic_text`] field type: + +. Open the previously created connector, and select the *Mappings* tab. +. Select *Add field*. +. Under *Field type*, select *Semantic text*. +. Under *Reference field*, select the field you want to use for model inference. +. Under *Select an inference endpoint*, select the model you want to use to add the embeddings to the data. +. Add the field to your mapping by selecting *Add field*. [discrete] [[obs-ai-interact]] From 4a2552e13291c2fd654fafb145dd6e70c0afb5d2 Mon Sep 17 00:00:00 2001 From: mdbirnstiehl Date: Wed, 15 Jan 2025 15:41:23 -0600 Subject: [PATCH 4/5] add steps --- docs/en/observability/observability-ai-assistant.asciidoc | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/en/observability/observability-ai-assistant.asciidoc b/docs/en/observability/observability-ai-assistant.asciidoc index 94a661ecb0..52bfc7df37 100644 --- a/docs/en/observability/observability-ai-assistant.asciidoc +++ b/docs/en/observability/observability-ai-assistant.asciidoc @@ -212,6 +212,9 @@ To create the embeddings needed by the AI Assistant using a {ref}/semantic-text. . Under *Reference field*, select the field you want to use for model inference. . Under *Select an inference endpoint*, select the model you want to use to add the embeddings to the data. . Add the field to your mapping by selecting *Add field*. +. Sync the data by selecting *Full Content* from the *Sync* menu. + +The AI Assistant will now query the connector you've set up using the model you've selected. Check if the AI Assistant is using the index by asking it something related to the indexed data. [discrete] [[obs-ai-interact]] From f7da52dab4f64fbdce5a8fcaff7b025d786959cb Mon Sep 17 00:00:00 2001 From: mdbirnstiehl Date: Fri, 24 Jan 2025 10:18:01 -0600 Subject: [PATCH 5/5] review updates --- .../observability-ai-assistant.asciidoc | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/docs/en/observability/observability-ai-assistant.asciidoc b/docs/en/observability/observability-ai-assistant.asciidoc index 52bfc7df37..732aa9ef59 100644 --- a/docs/en/observability/observability-ai-assistant.asciidoc +++ b/docs/en/observability/observability-ai-assistant.asciidoc @@ -173,7 +173,10 @@ For example, if you create a {ref}/es-connectors-github.html[GitHub connector] y + Learn more about configuring and {ref}/es-connectors-usage.html[using connectors] in the Elasticsearch documentation. -After creating your connector, create the embeddings needed by the AI Assistant. You can do this using either <>, which requires the ELSER model, or <>, which can use any available model (ELSER, E5, or a custom model). +After creating your connector, create the embeddings needed by the AI Assistant. You can do this using either: + +* <>: requires the ELSER ML model. +* <>: can use any available ML model (ELSER, E5, or a custom model). [discrete] [[obs-ai-search-connectors-ml-embeddings]] @@ -182,9 +185,9 @@ After creating your connector, create the embeddings needed by the AI Assistant. To create the embeddings needed by the AI Assistant (weights and tokens into a sparse vector field) using an *ML Inference Pipeline*: . Open the previously created connector, and select the *Pipelines* tab. -. Select *Copy and customize* button at the `Unlock your custom pipelines` box. -. Select *Add Inference Pipeline* button at the `Machine Learning Inference Pipelines` box. -. Select *ELSER (Elastic Learned Sparse EncodeR)* ML model to add the necessary embeddings to the data. +. Select *Copy and customize* under `Unlock your custom pipelines`. +. Select *Add Inference Pipeline* under `Machine Learning Inference Pipelines`. +. Select the *ELSER (Elastic Learned Sparse EncodeR)* ML model to add the necessary embeddings to the data. . Select the fields that need to be evaluated as part of the inference pipeline. . Test and save the inference pipeline and the overall pipeline. @@ -194,8 +197,8 @@ After creating the pipeline, complete the following steps: + Once the pipeline is set up, perform a *Full Content Sync* of the connector. The inference pipeline will process the data as follows: + -* As data comes in, ELSER is applied to the data, and embeddings (weights and tokens into a sparse vector field) are added to capture semantic meaning and context of the data. -* When you look at the documents that are ingested, you can see how the weights and token are added to the `predicted_value` field in the documents. +* As data comes in, ELSER is applied to the data, and embeddings (weights and tokens into a {ref}/query-dsl-sparse-vector-query.html[sparse vector field]) are added to capture semantic meaning and context of the data. +* When you look at the ingested documents, you can see the embeddings are added to the `predicted_value` field in the documents. . Check if AI Assistant can use the index (optional). + Ask something to the AI Assistant related with the indexed data. @@ -214,7 +217,8 @@ To create the embeddings needed by the AI Assistant using a {ref}/semantic-text. . Add the field to your mapping by selecting *Add field*. . Sync the data by selecting *Full Content* from the *Sync* menu. -The AI Assistant will now query the connector you've set up using the model you've selected. Check if the AI Assistant is using the index by asking it something related to the indexed data. +The AI Assistant will now query the connector you've set up using the model you've selected. +Check that the AI Assistant is using the index by asking it something related to the indexed data. [discrete] [[obs-ai-interact]]