Custom Ingest Pipeline for Dynamic Data Enrichment in the .alerts-security.alerts-<namespace>-index-template Index

[Harmlos]

Describe the feature:

Add the ability to invoke an ingest pipeline for the .alerts-security.alerts system index, allowing the invocation of a custom ingest pipeline that can include other user-defined pipelines for further data enrichment or other purposes within the cluster. This will enable data enrichment and add context to alerts during incidents.

Describe a specific use case for the feature:

In many installations, it is necessary to enrich data when an incident is created.

Typical fields for all configurations may include:

• Additional information from the HR system
• Additional information from CMDB about the host that triggered the alert
• Additional information from CMDB about the relationship to the resource-service model and business processes
• Additional information about the event source that triggered the alert and the responsible party

These fields are necessary for understanding the cause of the alert and speeding up its resolution. For example, when an alert occurs due to incorrect password input, the event could include the password change date and the last vacation date. Users returning from vacation or those who recently changed their password often generate such alerts, which can help reduce the threat level of the event. However, if an alert is triggered by a user currently on vacation, it should be elevated in severity, as the actions would be suspicious.

Currently, these enrichments are implemented using a combination of component templates and the ingest pipeline defined within them, which are added to the default index template .alerts-security.alerts--index-template. However, the problem is that this index template is constantly reset to the default values due to an internal Elastic mechanism, which regularly resets index template settings to default values, deleting any customizations and connected component templates. As a result, all changes for enrichment and modifications need to be reconfigured.

Also, any update to a new version completely breaks the enrichment scheme, as all system index templates are reset to default values.

Creating a default mechanism for invoking a custom component template and custom ingest pipeline will provide flexibility for configuring each cluster, improve security, and reduce the risks of system interference. Additionally, it will eliminate the need to reconfigure after each update.

This will allow alert configuration, data enrichment, and adding more context to alerts in each cluster with reduced risks of damaging the integrity of the system. Users will be able to customize the functionality to meet their needs and objectives.

Implementing a centralized ingest pipeline invocation will significantly simplify the configuration and support of enrichments, reducing the risks associated with configuration errors and resetting settings.

This change is an expanded update compared to the one described here: #177802

That case describes only custom component templates, whereas this one addresses the need for centralized ingest pipeline invocation.