diff --git a/docs/en/stack/ml/anomaly-detection/anomaly-how-tos.asciidoc b/docs/en/stack/ml/anomaly-detection/anomaly-how-tos.asciidoc index 1a6ef27d4..a8d6ab6cd 100644 --- a/docs/en/stack/ml/anomaly-detection/anomaly-how-tos.asciidoc +++ b/docs/en/stack/ml/anomaly-detection/anomaly-how-tos.asciidoc @@ -13,6 +13,7 @@ The guides in this section describe some best practices for generating useful * <> * <> * <> +* <> * <> * <> * <> diff --git a/docs/en/stack/ml/anomaly-detection/geographic-anomalies.asciidoc b/docs/en/stack/ml/anomaly-detection/geographic-anomalies.asciidoc index 02bcae275..99f52ff3f 100644 --- a/docs/en/stack/ml/anomaly-detection/geographic-anomalies.asciidoc +++ b/docs/en/stack/ml/anomaly-detection/geographic-anomalies.asciidoc @@ -211,6 +211,11 @@ behavior: image::images/ecommerce-anomaly-explorer-geopoint.jpg[A screenshot of an anomalous event in the eCommerce data in Anomaly Explorer] // NOTE: This is an autogenerated screenshot. Do not edit it directly. +A "typical" value indicates a centroid of a cluster of previously observed +locations that is closest to the "actual" location at that time. For example, +there may be one centroid near the user's home and another near the user's +work place since there are many records associated with these distinct locations. + Likewise, there are time periods in the web logs sample data where there are both unusually high sums of data transferred and unusual geographical coordinates: diff --git a/docs/en/stack/ml/anomaly-detection/images/categorization-wizard.png b/docs/en/stack/ml/anomaly-detection/images/categorization-wizard.png new file mode 100644 index 000000000..a3f9a0de2 Binary files /dev/null and b/docs/en/stack/ml/anomaly-detection/images/categorization-wizard.png differ diff --git a/docs/en/stack/ml/anomaly-detection/images/ml-category-analyzer.png b/docs/en/stack/ml/anomaly-detection/images/ml-category-analyzer.png new file mode 100644 index 000000000..34c0d2032 Binary files /dev/null and b/docs/en/stack/ml/anomaly-detection/images/ml-category-analyzer.png differ diff --git a/docs/en/stack/ml/anomaly-detection/images/ml-category-anomalies.png b/docs/en/stack/ml/anomaly-detection/images/ml-category-anomalies.png new file mode 100644 index 000000000..a50f96ff2 Binary files /dev/null and b/docs/en/stack/ml/anomaly-detection/images/ml-category-anomalies.png differ diff --git a/docs/en/stack/ml/anomaly-detection/index.asciidoc b/docs/en/stack/ml/anomaly-detection/index.asciidoc index 0a7f3338b..d3d2af798 100644 --- a/docs/en/stack/ml/anomaly-detection/index.asciidoc +++ b/docs/en/stack/ml/anomaly-detection/index.asciidoc @@ -36,6 +36,8 @@ include::{es-repo-dir}/ml/anomaly-detection/ml-configuring-transform.asciidoc[le include::{es-repo-dir}/ml/anomaly-detection/ml-configuring-detector-custom-rules.asciidoc[leveloffset=+2] +include::ml-detect-categories.asciidoc[leveloffset=+2] + include::ml-revert-model-snapshot.asciidoc[leveloffset=+2] include::geographic-anomalies.asciidoc[leveloffset=+2] diff --git a/docs/en/stack/ml/anomaly-detection/ml-detect-categories.asciidoc b/docs/en/stack/ml/anomaly-detection/ml-detect-categories.asciidoc new file mode 100644 index 000000000..b929c9358 --- /dev/null +++ b/docs/en/stack/ml/anomaly-detection/ml-detect-categories.asciidoc @@ -0,0 +1,253 @@ +[[ml-configuring-categories]] += Detecting anomalous categories of data + +Categorization is a {ml} process that tokenizes a text field, clusters similar data together, and classifies it into categories. +It works best on machine-written messages and application output that typically consist of repeated elements. +<> enable you to find anomalous behavior in your categorized data. +Categorization is not natural language processing (NLP). +When you create a categorization {anomaly-job}, the {ml} model learns what volume and pattern is normal for each category over time. +You can then detect anomalies and surface rare events or unusual types of messages by using <> or <> functions. +Categorization works well on finite set of possible messages, for example: + +[source,js] +---------------------------------- +{"@timestamp":1549596476000, +"message":"org.jdbi.v2.exceptions.UnableToExecuteStatementException: com.mysql.jdbc.exceptions.MySQLTimeoutException: Statement cancelled due to timeout or client request [statement:\"SELECT id, customer_id, name, force_disabled, enabled FROM customers\"]", +"type":"logs"} +---------------------------------- +//NOTCONSOLE + + +[discrete] +[[categ-recommendations]] +== Recommendations + +* Categorization is tuned to work best on data like log messages by taking token order into account, including stop words, and not considering synonyms in its analysis. +Use machine-written messages for categorization analysis. +* Complete sentences in human communication or literary text (for example email, wiki pages, prose, or other human-generated content) can be extremely diverse in structure. +Since categorization is tuned for machine data, it gives poor results for human-generated data. +It would create so many categories that they couldn’t be handled effectively. +Avoid using human-generated data for categorization analysis. + +[discrete] +[[creating-categorization-jobs]] +== Creating categorization jobs + +. In {kib}, navigate to **{ml-app} > Anomaly Detection > Jobs**. +. Click **Create {anomaly-jobs}**, select the {data-view} you want to analyze. +. Select the **Categorization** wizard from the list. +. Choose a categorization detector - it's the `count` function in this example - and the field you want to categorize - the `message` field in this example. ++ +-- +[role="screenshot"] +image::images/categorization-wizard.png[Creating a categorization job in Kibana] +-- +. Click **Next**. +. Provide a job ID and click **Next**. +. If the validation is successful, click **Next** to review the summary of the job creation. +. Click **Create job**. + +This example job generates categories from the contents of the `message` field and uses the `count` function to determine when certain categories are occurring at anomalous rates. + +[%collapsible] +.API example +==== +[source,console] +---------------------------------- +PUT _ml/anomaly_detectors/it_ops_app_logs +{ + "description" : "IT ops application logs", + "analysis_config" : { + "categorization_field_name": "message",<1> + "bucket_span":"30m", + "detectors" :[{ + "function":"count", + "by_field_name": "mlcategory"<2> + }] + }, + "data_description" : { + "time_field":"@timestamp" + } +} +---------------------------------- +// TEST[skip:needs-licence] +<1> This field is used to derive categories. +<2> The categories are used in a detector by setting `by_field_name`, `over_field_name`, or `partition_field_name` to the keyword `mlcategory`. +If you do not specify this keyword in one of those properties, the API request fails. +==== + + +[discrete] +[[categorization-job-results]] +=== Viewing the job results + +Use the **Anomaly Explorer** in {kib} to view the analysis results: + +[role="screenshot"] +image::images/ml-category-anomalies.png["Categorization results in the Anomaly Explorer"] + +For this type of job, the results contain extra information for each anomaly: the name of the category (for example, `mlcategory 2`) and examples of the messages in that category. +You can use these details to investigate occurrences of unusually high message counts. + + +[discrete] +[[advanced-categorization-options]] +=== Advanced configuration options + +If you use the advanced {anomaly-job} wizard in {kib} or the {ref}/ml-put-job.html[create {anomaly-jobs} API], there are additional configuration options. +For example, the optional `categorization_examples_limit` property specifies the maximum number of examples that are stored in memory and in the results data store for each category. +The default value is `4`. +Note that this setting does not affect the categorization; it just affects the list of visible examples. +If you increase this value, more examples are available, but you must have more storage available. +If you set this value to `0`, no examples are stored. + +Another advanced option is the `categorization_filters` property, which can contain an array of regular expressions. +If a categorization field value matches the regular expression, the portion of the field that is matched is not taken into consideration when defining categories. +The categorization filters are applied in the order they are listed in the job configuration, which enables you to disregard multiple sections of the categorization field value. +In this example, you might create a filter like `[ "\\[statement:.*\\]"]` to remove the SQL statement from the categorization algorithm. + + +[discrete] +[[ml-per-partition-categorization]] +== Per-partition categorization + +If you enable per-partition categorization, categories are determined independently for each partition. +For example, if your data includes messages from multiple types of logs from different applications, you can use a field like the ECS {ecs-ref}/ecs-event.html[`event.dataset` field] as the `partition_field_name` and categorize the messages for each type of log separately. + +If your job has multiple detectors, every detector that uses the `mlcategory` keyword must also define a `partition_field_name`. +You must use the same `partition_field_name` value in all of these detectors. +Otherwise, when you create or update a job and enable per-partition categorization, it fails. + +When per-partition categorization is enabled, you can also take advantage of a `stop_on_warn` configuration option. +If the categorization status for a partition changes to `warn`, it doesn't categorize well and can cause unnecessary resource usage. +When you set `stop_on_warn` to `true`, the job stops analyzing these problematic partitions. +You can thus avoid an ongoing performance cost for partitions that are unsuitable for categorization. + + +[discrete] +[[ml-configuring-analyzer]] +== Customizing the categorization analyzer + +Categorization uses English dictionary words to identify log message categories. +By default, it also uses English tokenization rules. +For this reason, if you use the default categorization analyzer, only English language log messages are supported, as described in the <>. + +If you use the categorization wizard in {kib}, you can see which categorization analyzer it uses and highlighted examples of the tokens that it identifies. +You can also change the tokenization rules by customizing the way the categorization field values are interpreted: + +[role="screenshot"] +image::images/ml-category-analyzer.png["Editing the categorization analyzer in Kibana"] + +The categorization analyzer can refer to a built-in {es} analyzer or a combination of zero or more character filters, a tokenizer, and zero or more token filters. +In this example, adding a {ref}/analysis-pattern-replace-charfilter.html[`pattern_replace` character filter] achieves the same behavior as the `categorization_filters` job configuration option described earlier. +For more details about these properties, refer to the {ref}/ml-put-job.html#ml-put-job-request-body[`categorization_analyzer` API object]. + +If you use the default categorization analyzer in {kib} or omit the `categorization_analyzer` property from the API, the following default values are used: + +[source,console] +-------------------------------------------------- +POST _ml/anomaly_detectors/_validate +{ + "analysis_config" : { + "categorization_analyzer" : { + "char_filter" : [ + "first_line_with_letters" + ], + "tokenizer" : "ml_standard", + "filter" : [ + { "type" : "stop", "stopwords": [ + "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday", + "Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun", + "January", "February", "March", "April", "May", "June", "July", "August", "September", "October", "November", "December", + "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec", + "GMT", "UTC" + ] } + ] + }, + "categorization_field_name": "message", + "detectors" :[{ + "function":"count", + "by_field_name": "mlcategory" + }] + }, + "data_description" : { + } +} +-------------------------------------------------- + +If you specify any part of the `categorization_analyzer`, however, any omitted sub-properties are _not_ set to default values. + +The `ml_standard` tokenizer and the day and month stopword filter are almost equivalent to the following analyzer, which is defined using only built-in {es} {ref}/analysis-tokenizers.html[tokenizers] and {ref}/analysis-tokenfilters.html[token filters]: + +[source,console] +---------------------------------- +PUT _ml/anomaly_detectors/it_ops_new_logs +{ + "description" : "IT Ops Application Logs", + "analysis_config" : { + "categorization_field_name": "message", + "bucket_span":"30m", + "detectors" :[{ + "function":"count", + "by_field_name": "mlcategory", + "detector_description": "Unusual message counts" + }], + "categorization_analyzer":{ + "char_filter" : [ + "first_line_with_letters" <1> + ], + "tokenizer": { + "type" : "simple_pattern_split", + "pattern" : "[^-0-9A-Za-z_./]+" <2> + }, + "filter": [ + { "type" : "pattern_replace", "pattern": "^[0-9].*" }, <3> + { "type" : "pattern_replace", "pattern": "^[-0-9A-Fa-f.]+$" }, <4> + { "type" : "pattern_replace", "pattern": "^[^0-9A-Za-z]+" }, <5> + { "type" : "pattern_replace", "pattern": "[^0-9A-Za-z]+$" }, <6> + { "type" : "stop", "stopwords": [ + "", + "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday", + "Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun", + "January", "February", "March", "April", "May", "June", "July", "August", "September", "October", "November", "December", + "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec", + "GMT", "UTC" + ] } + ] + } + }, + "analysis_limits":{ + "categorization_examples_limit": 5 + }, + "data_description" : { + "time_field":"time", + "time_format": "epoch_ms" + } +} +---------------------------------- +// TEST[skip:needs-licence] + +<1> Only consider the first line of the message with letters for categorization purposes. +<2> Tokens consist of hyphens, digits, letters, underscores, dots and slashes. +<3> By default, categorization ignores tokens that begin with a digit. +<4> By default, categorization ignores tokens that are hexadecimal numbers. +<5> Underscores, hyphens, and dots are removed from the beginning of tokens. +<6> Underscores, hyphens, and dots are also removed from the end of tokens. + +The key difference between the default `categorization_analyzer` and this example analyzer is that using the `ml_standard` tokenizer is several times faster. +The `ml_standard` tokenizer also tries to preserve URLs, Windows paths and email addresses as single tokens. +Another difference in behavior is that the custom analyzer does not include accented letters in tokens whereas the `ml_standard` tokenizer does. +This could be fixed by using more complex regular expressions. + +If you are categorizing non-English messages in a language where words are separated by spaces, you might get better results if you change the day or month words in the stop token filter to the appropriate words in your language. +If you are categorizing messages in a language where words are not separated by spaces, you must use a different tokenizer as well in order to get sensible categorization results. + +It is important to be aware that analyzing for categorization of machine generated log messages is a little different from tokenizing for search. +Features that work well for search, such as stemming, synonym substitution, and lowercasing are likely to make the results of categorization worse. +However, to drill down from {ml} results to work correctly, the tokens the categorization analyzer produces must be similar to those produced by the search analyzer. +If they are sufficiently similar, when you search for the tokens that the categorization analyzer produces then you find the original document that the categorization field value came from. + + + + + diff --git a/docs/en/stack/ml/anomaly-detection/ootb-ml-jobs-metrics-ui.asciidoc b/docs/en/stack/ml/anomaly-detection/ootb-ml-jobs-metrics-ui.asciidoc index fda928cb3..3136f1603 100644 --- a/docs/en/stack/ml/anomaly-detection/ootb-ml-jobs-metrics-ui.asciidoc +++ b/docs/en/stack/ml/anomaly-detection/ootb-ml-jobs-metrics-ui.asciidoc @@ -1,10 +1,8 @@ ["appendix",role="exclude",id="ootb-ml-jobs-metrics-ui"] = Metrics {anomaly-detect} configurations -These {anomaly-jobs} can be created in the -{observability-guide}/analyze-metrics.html[{metrics-app}] in {kib}. For more -information about their usage, refer to -{observability-guide}/inspect-metric-anomalies.html[Inspect metric anomalies]. +These {anomaly-jobs} can be created in the {observability-guide}/analyze-metrics.html[{infrastructure-app}] in {kib}. +For more information about their usage, refer to {observability-guide}/inspect-metric-anomalies.html[Inspect metric anomalies]. // tag::metrics-jobs[] [discrete] diff --git a/docs/en/stack/ml/nlp/ml-nlp-deploy-models.asciidoc b/docs/en/stack/ml/nlp/ml-nlp-deploy-models.asciidoc index df673edc3..57561df7a 100644 --- a/docs/en/stack/ml/nlp/ml-nlp-deploy-models.asciidoc +++ b/docs/en/stack/ml/nlp/ml-nlp-deploy-models.asciidoc @@ -94,7 +94,8 @@ eland_import_hub_model \ <> to learn more. <3> Specify the identifier for the model in the Hugging Face model hub. <4> Specify the type of NLP task. Supported values are `fill_mask`, `ner`, -`text_classification`, `text_embedding`, and `zero_shot_classification`. +`question_answering`, `text_classification`, `text_embedding`, `text_expansion`, +`text_similarity`, and `zero_shot_classification`. For more details, refer to https://www.elastic.co/guide/en/elasticsearch/client/eland/current/machine-learning.html#ml-nlp-pytorch. @@ -112,7 +113,7 @@ $ docker run -it --rm --network host docker.elastic.co/eland/eland The `eland_import_hub_model` script can be run directly in the docker command: ```bash -docker run -it --rm elastic/eland \ +docker run -it --rm docker.elastic.co/eland/eland \ eland_import_hub_model \ --url $ELASTICSEARCH_URL \ --hub-model-id elastic/distilbert-base-uncased-finetuned-conll03-english \ diff --git a/docs/en/stack/ml/nlp/ml-nlp-e5.asciidoc b/docs/en/stack/ml/nlp/ml-nlp-e5.asciidoc index 20e3d0962..5b8595cbc 100644 --- a/docs/en/stack/ml/nlp/ml-nlp-e5.asciidoc +++ b/docs/en/stack/ml/nlp/ml-nlp-e5.asciidoc @@ -21,7 +21,11 @@ contextual meaning and user intent, rather than exact keyword matches. E5 has two versions: one cross-platform version which runs on any hardware and one version which is optimized for Intel® silicon. The **Model Management** > **Trained Models** page shows you which version of E5 is -recommended to deploy based on your cluster's hardware. +recommended to deploy based on your cluster's hardware. However, the +recommended way to use E5 is through the +{ref}/infer-service-elasticsearch.html[{infer} API] as a service which makes it +easier to download and deploy the model and you don't need to select from +different versions. Refer to the model cards of the https://huggingface.co/elastic/multilingual-e5-small[multilingual-e5-small] and @@ -42,14 +46,48 @@ for semantic search or the trial period activated. [[download-deploy-e5]] == Download and deploy E5 -You can download and deploy the E5 model either from -**{ml-app}** > **Trained Models**, from **Search** > **Indices**, or by using -the Dev Console. +The easiest and recommended way to download and deploy E5 is to use the {ref}/inference-apis.html[{infer} API]. + +1. In {kib}, navigate to the **Dev Console**. +2. Create an {infer} endpoint with the `elasticsearch` service by running the following API request: ++ +-- +[source,console] +---------------------------------- +PUT _inference/text_embedding/my-e5-model +{ + "service": "elasticsearch", + "service_settings": { + "num_allocations": 1, + "num_threads": 1, + "model_id": ".multilingual-e5-small" + } +} +---------------------------------- +-- +The API request automatically initiates the model download and then deploy the model. + +Refer to the {ref}/infer-service-elasticsearch.html[`elasticsearch` {infer} service documentation] to learn more about the available settings. + +After you created the E5 {infer} endpoint, it's ready to be used for semantic search. +The easiest way to perform semantic search in the {stack} is to {ref}/semantic-search-semantic-text.html[follow the `semantic_text` workflow]. +[discrete] +[[alternative-download-deploy-e5]] +=== Alternative methods to download and deploy E5 + +You can also download and deploy the E5 model either from **{ml-app}** > **Trained Models**, from **Search** > **Indices**, or by using the trained models API in Dev Console. + +NOTE: For most cases, the preferred version is the **Intel and Linux optimized** model, it is recommended to download and deploy that version. + + +.Using the Trained Models page +[%collapsible%closed] +===== [discrete] [[trained-model-e5]] -=== Using the Trained Models page +==== Using the Trained Models page 1. In {kib}, navigate to **{ml-app}** > **Trained Models**. E5 can be found in the list of trained models. There are two versions available: one portable @@ -77,14 +115,18 @@ allocations and threads per allocation values. + -- [role="screenshot"] -image::images/ml-nlp-deployment-id-e5.png[alt="Deploying ELSER",align="center"] +image::images/ml-nlp-deployment-id-e5.png[alt="Deploying E5",align="center"] -- 5. Click Start. +===== +.Using the search indices UI +[%collapsible%closed] +===== [discrete] [[elasticsearch-e5]] -=== Using the search indices UI +==== Using the search indices UI Alternatively, you can download and deploy the E5 model to an {infer} pipeline using the search indices UI. @@ -113,11 +155,15 @@ image::images/ml-nlp-start-e5-es.png[alt="Start E5 in Elasticsearch",align="cent When your E5 model is deployed and started, it is ready to be used in a pipeline. +===== +.Using the trained models API in Dev Console +[%collapsible%closed] +===== [discrete] [[dev-console-e5]] -=== Using the Dev Console +==== Using the trained models API in Dev Console 1. In {kib}, navigate to the **Dev Console**. 2. Create the E5 model configuration by running the following API call: @@ -146,6 +192,7 @@ with a delpoyment ID: POST _ml/trained_models/.multilingual-e5-small/deployment/_start?deployment_id=for_search ---------------------------------- -- +===== [discrete] @@ -194,7 +241,7 @@ to the `config/elasticsearch.yml` file: + -- ``` -xpack.ml.model_repository: file://${path.home}/config/models/` +xpack.ml.model_repository: file://${path.home}/config/models/ ``` -- . Repeat step 2 and step 3 on all master-eligible nodes. @@ -275,4 +322,7 @@ Customers may add third party trained models for management in Elastic. These models are not owned by Elastic. While Elastic will support the integration with these models in the performance according to the documentation, you understand and agree that Elastic has no control over, or liability for, the third party -models or the underlying training data they may utilize. \ No newline at end of file +models or the underlying training data they may utilize. + +This e5 model, as defined, hosted, integrated and used in conjunction with our +other Elastic Software is covered by our standard warranty. diff --git a/docs/en/stack/ml/nlp/ml-nlp-elser.asciidoc b/docs/en/stack/ml/nlp/ml-nlp-elser.asciidoc index 3596e58e4..36f797e9e 100644 --- a/docs/en/stack/ml/nlp/ml-nlp-elser.asciidoc +++ b/docs/en/stack/ml/nlp/ml-nlp-elser.asciidoc @@ -80,7 +80,11 @@ computing the similarity between a query and a document. ELSER v2 has two versions: one cross-platform version which runs on any hardware and one version which is optimized for Intel® silicon. The **Model Management** > **Trained Models** page shows you which version of ELSER -v2 is recommended to deploy based on your cluster's hardware. +v2 is recommended to deploy based on your cluster's hardware. However, the +recommended way to use ELSER is through the +{ref}/infer-service-elser.html[{infer} API] as a service which makes it easier +to download and deploy the model and you don't need to select from different +versions. If you want to learn more about the ELSER V2 improvements, refer to https://www.elastic.co/search-labs/introducing-elser-v2-part-1[this blog post]. @@ -105,13 +109,56 @@ that walks through upgrading an index to ELSER V2. [[download-deploy-elser]] == Download and deploy ELSER -You can download and deploy ELSER either from **{ml-app}** > **Trained Models**, -from **Search** > **Indices**, or by using the Dev Console. +The easiest and recommended way to download and deploy ELSER is to use the {ref}/inference-apis.html[{infer} API]. +1. In {kib}, navigate to the **Dev Console**. +2. Create an {infer} endpoint with the ELSER service by running the following API request: ++ +-- +[source,console] +---------------------------------- +PUT _inference/sparse_embedding/my-elser-model +{ + "service": "elser", + "service_settings": { + "num_allocations": 1, + "num_threads": 1 + } +} +---------------------------------- +-- +The API request automatically initiates the model download and then deploy the model. + +Refer to the {ref}/infer-service-elser.html[ELSER {infer} service documentation] to learn more about the available settings. + +After you created the ELSER {infer} endpoint, it's ready to be used for semantic search. +The easiest way to perform semantic search in the {stack} is to {ref}/semantic-search-semantic-text.html[follow the `semantic_text` workflow]. + +[discrete] +[[alternative-download-deploy]] +=== Alternative methods to download and deploy ELSER + +You can also download and deploy ELSER either from **{ml-app}** > **Trained Models**, from **Search** > **Indices**, or by using the trained models API in Dev Console. + +[NOTE] +==== +* For most cases, the preferred version is the **Intel and Linux optimized** +model, it is recommended to download and deploy that version. +* You can deploy the model multiple times by assigning a unique deployment ID +when starting the deployment. It enables you to have dedicated deployments for +different purposes, such as search and ingest. By doing so, you ensure that the +search speed remains unaffected by ingest workloads, and vice versa. Having +separate deployments for search and ingest mitigates performance issues +resulting from interactions between the two, which can be hard to diagnose. +==== + +.Using the Trained Models page +[%collapsible%closed] +===== [discrete] [[trained-model]] -=== Using the Trained Models page +==== Using the Trained Models page 1. In {kib}, navigate to **{ml-app}** > **Trained Models**. ELSER can be found in the list of trained models. There are two versions available: one portable @@ -142,11 +189,14 @@ allocations and threads per allocation values. image::images/ml-nlp-deployment-id-elser-v2.png[alt="Deploying ELSER",align="center"] -- 5. Click **Start**. +===== - +.Using the search indices UI +[%collapsible%closed] +===== [discrete] [[elasticsearch]] -=== Using the search indices UI +==== Using the search indices UI Alternatively, you can download and deploy ELSER to an {infer} pipeline using the search indices UI. @@ -172,43 +222,14 @@ model deployment. [role="screenshot"] image::images/ml-nlp-start-elser-v2-es.png[alt="Start ELSER in Elasticsearch",align="center"] -- +===== -When your ELSER model is deployed and started, it is ready to be used in a -pipeline. - - -[discrete] -[[elasticsearch-ingest-pipeline]] -==== Adding ELSER to an ingest pipeline - -To add ELSER to an ingest pipeline, you need to copy the default ingest -pipeline and then customize it according to your needs. - -1. Click **Copy and customize** under the **Unlock your custom pipelines** block -at the top of the page. This enables the **Add inference pipeline** button. -+ --- -[role="screenshot"] -image::images/ml-nlp-pipeline-copy-customize.png[alt="Start ELSER in Elasticsearch",align="center"] --- -2. Under **{ml-app} {infer-cap} Pipelines**, click **Add inference pipeline**. -3. Give a name to the pipeline, select ELSER from the list of trained ML models, -and click **Continue**. -4. Select the source text field, define the target field, and click **Add** then -**Continue**. -5. Review the index mappings updates. Click **Back** if you want to change the -mappings. Click **Continue** if you are satisfied with the updated index -mappings. -6. You can optionally test your pipeline. Click **Continue**. -7. **Create pipeline**. - -Once your pipeline is created, you are ready to ingest documents and utilize -ELSER for text expansions in your search queries. - - +.Using the traned models API in Dev Console +[%collapsible%closed] +===== [discrete] [[dev-console]] -=== Using the Dev Console +==== Using the trained models API in Dev Console 1. In {kib}, navigate to the **Dev Console**. 2. Create the ELSER model configuration by running the following API call: @@ -239,9 +260,7 @@ POST _ml/trained_models/.elser_model_2/deployment/_start?deployment_id=for_searc You can deploy the model multiple times with different deployment IDs. -- - -After the deployment is complete, ELSER is ready to use either in an ingest -pipeline or in a `text_expansion` query to perform semantic search. +===== [discrete] @@ -362,7 +381,7 @@ following line to the `config/elasticsearch.yml` file: + -- ``` -xpack.ml.model_repository: file://${path.home}/config/models/` +xpack.ml.model_repository: file://${path.home}/config/models/ ``` -- . Repeat step 2 and step 3 on all master-eligible nodes. @@ -428,10 +447,12 @@ To learn more about ELSER performance, refer to the <>. * {ref}/semantic-search-elser.html[Perform semantic search with ELSER] * https://www.elastic.co/blog/may-2023-launch-information-retrieval-elasticsearch-ai-model[Improving information retrieval in the Elastic Stack: Introducing Elastic Learned Sparse Encoder, our new retrieval model] - +[discrete] [[elser-benchmarks]] == Benchmark information +IMPORTANT: The recommended way to use ELSER is through the {ref}/infer-service-elser.html[{infer} API] as a service. + The following sections provide information about how ELSER performs on different hardwares and compares the model performance to {es} BM25 and other strong baselines. @@ -447,6 +468,7 @@ any platform. [discrete] +[[version-overview-v2]] ==== ELSER V2 Besides the performance improvements, the biggest change in ELSER V2 is the diff --git a/docs/en/stack/ml/nlp/ml-nlp-ner-example.asciidoc b/docs/en/stack/ml/nlp/ml-nlp-ner-example.asciidoc index f021297eb..7d25a380c 100644 --- a/docs/en/stack/ml/nlp/ml-nlp-ner-example.asciidoc +++ b/docs/en/stack/ml/nlp/ml-nlp-ner-example.asciidoc @@ -35,7 +35,7 @@ image: [source,shell] -------------------------------------------------- -docker run -it --rm elastic/eland \ +docker run -it --rm docker.elastic.co/eland/eland \ eland_import_hub_model \ --cloud-id $CLOUD_ID \ -u -p \ diff --git a/docs/en/stack/ml/nlp/ml-nlp-text-emb-vector-search-example.asciidoc b/docs/en/stack/ml/nlp/ml-nlp-text-emb-vector-search-example.asciidoc index 01b73b2ca..6c8e63c48 100644 --- a/docs/en/stack/ml/nlp/ml-nlp-text-emb-vector-search-example.asciidoc +++ b/docs/en/stack/ml/nlp/ml-nlp-text-emb-vector-search-example.asciidoc @@ -48,7 +48,7 @@ image: [source,shell] -------------------------------------------------- -docker run -it --rm elastic/eland \ +docker run -it --rm docker.elastic.co/eland/eland \ eland_import_hub_model \ --cloud-id $CLOUD_ID \ -u -p \ diff --git a/docs/en/stack/ml/redirects.asciidoc b/docs/en/stack/ml/redirects.asciidoc index 977f0ba8e..ba4bc5bfc 100644 --- a/docs/en/stack/ml/redirects.asciidoc +++ b/docs/en/stack/ml/redirects.asciidoc @@ -23,11 +23,6 @@ This page has been removed. Refer to <>. This page has been removed. Refer to <>. -[role="exclude",id="ml-configuring-categories"] -=== Detecting anomalous categories of data - -This page has been removed. Refer to <>. - [role="exclude",id="ml-inference-models"] === Trained {ml} models as functions