Skip to content

Latest commit

 

History

History
79 lines (60 loc) · 3.84 KB

inspect-log-anomalies.asciidoc

File metadata and controls

79 lines (60 loc) · 3.84 KB

Inspect log anomalies

When the {anomaly-detect} features of {ml} are enabled, you can use the Anomalies page in the {logs-app} to detect and inspect log anomalies and the log partitions where the log anomalies occur. This means you can easily see anomalous behavior without significant human intervention — no more manually sampling log data, calculating rates, and determining if rates are expected.

Anomalies automatically highlight periods where the log rate is outside expected bounds and therefore may be anomalous. For example:

  • A significant drop in the log rate might suggest that a piece of infrastructure stopped responding, and thus we’re serving fewer requests.

  • A spike in the log rate could denote a DDoS attack. This may lead to an investigation of IP addresses from incoming requests.

You can also view log anomalies directly in the {kibana-ref}/xpack-ml-anomalies.html[{ml-app} app].

Note
 This feature makes use of {ml} {anomaly-jobs}. To set up jobs, you must have all {kib} feature privileges for {ml-app}. Users that have full or read-only access to {ml-features} within a {kib} space can view the results of all {anomaly-jobs} that are visible in that space, even if they do not have access to the source indices of those jobs. You must carefully consider who is given access to {ml-features}; {anomaly-job} results may propagate field values that contain sensitive information from the source indices to the results. For more details, refer to {ml-docs}/setup.html[Set up {ml-features}].

Enable log rate analysis and {anomaly-detect}

Create a {ml} job to detect anomalous log entry rates automatically.

  1. Select Anomalies, and you’ll be prompted to create a {ml} job which will carry out the log rate analysis.

  2. Choose a time range for the {ml} analysis.

  3. Add the indices that contain the logs you want to examine. By default, Machine Learning analyzes messages in all log indices that match the patterns set in the logs source advanced setting. Update this setting by going to Management → Advanced Settings and searching for logs source.

  4. Click Create {ml-init} job.

  5. You’re now ready to explore your log partitions.

Anomalies chart

The Anomalies chart shows an overall, color-coded visualization of the log entry rate, partitioned according to the value of the Elastic Common Schema (ECS) {ecs-ref}/ecs-event.html[event.dataset] field. This chart helps you quickly spot increases or decreases in each partition’s log rate.

If you have a lot of log partitions, use the following to filter your data:

  • Hover over a time range to see the log rate for each partition.

  • Click or hover on a partition name to show, hide, or highlight the partition values.

Anomalies chart

The chart shows the time range where anomalies were detected. The typical rate values are shown in gray, while the anomalous regions are color-coded and superimposed on top.

When a time range is flagged as anomalous, the {ml} algorithms have detected unusual log rate activity. This might be because:

  • The log rate is significantly higher than usual.

  • The log rate is significantly lower than usual.

  • Other anomalous behavior has been detected. For example, the log rate is within bounds, but not fluctuating when it is expected to.

The level of anomaly detected in a time period is color-coded, from red, orange, yellow, to blue. Red indicates a critical anomaly level, while blue is a warning level.

To help you further drill down into a potential anomaly, you can view an anomaly chart for each partition. Anomaly scores range from 0 (no anomalies) to 100 (critical).

To analyze the anomalies in more detail, click View anomaly in {ml}, which opens the {ml-docs}/ml-gs-results.html[Anomaly Explorer in {ml-app}].