diff --git a/.github/workflows/pr-code-checks.yml b/.github/workflows/pr-code-checks.yml index 6a8437bc5a..b00c02f681 100644 --- a/.github/workflows/pr-code-checks.yml +++ b/.github/workflows/pr-code-checks.yml @@ -17,9 +17,6 @@ on: - 'external-images.yaml' workflow_dispatch: -env: - GITLEAKS_VERSION: 8.28.0 - jobs: unit-tests: runs-on: ubuntu-latest diff --git a/README.md b/README.md index 5b4822822e..413d45d2c6 100644 --- a/README.md +++ b/README.md @@ -8,25 +8,25 @@ ## Overview -[Telemetry Manager](docs/user/01-manager.md) is a Kubernetes [operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) that fulfils the [Kyma module interface](https://kyma-project.io/#/06-modules/README.md). It provides APIs for a managed agent/gateway setup for log, trace, and metric ingestion and dispatching into 3rd-party backend systems, in order to reduce the pain of orchestrating such setup on your own. Read more on the [manager](./docs/user/01-manager.md) itself or the general [usage](docs/user/README.md) of the module. +[Telemetry Manager](docs/user/01-manager.md) is a Kubernetes [operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) that fulfils the [Kyma module interface](https://kyma-project.io/#/06-modules/README.md). It provides APIs for a managed agent/gateway setup for log, trace, and metric ingestion and dispatching into 3rd-party backend systems, in order to reduce the pain of orchestrating such setup on your own. Read more on the [Architecture](./docs/user/architecture/README.md) itself or the general [usage](docs/user/README.md) of the module. ### Logs -The logging controllers generate a Fluent Bit DaemonSet and configuration from one or more LogPipeline and LogParser custom resources. The controllers ensure that all Fluent Bit Pods run the current configuration by restarting Pods after the configuration has changed. See all [CRD attributes](apis/telemetry/v1alpha1/logpipeline_types.go) and some [examples](samples). +The logging controllers generate an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) DaemonSet and Deployment, and configuration from one or more `LogPipeline` custom resources (CRs). The controllers ensure that all OTel Collector Pods run the current configuration by restarting Pods after the configuration has changed. See all [CRD attributes](apis/telemetry/v1alpha1/logpipeline_types.go) and some [examples](./samples). -For more information, see [Logs](./docs/user/02-logs.md). +For more information, see [Logs](./docs/user/collecting-logs/README.md). ### Traces -The trace controller creates an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) deployment and related Kubernetes objects from a `TracePipeline` custom resource. The collector is configured to receive traces using the [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/specs/otel/protocol/), and forwards the received traces to a configurable OTLP backend. +The trace controller creates an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) Deployment and related Kubernetes objects from a `TracePipeline` CR. The collector is configured to receive traces using the [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/specs/otel/protocol/), and forwards the received traces to a configurable OTLP backend. -For more information, see [Traces](./docs/user/03-traces.md). +For more information, see [Traces](./docs/user/collecting-traces/README.md). ### Metrics -The metric controller creates an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) and related Kubernetes objects from a `MetricPipeline` custom resource. The collector is deployed as a [Gateway](https://opentelemetry.io/docs/collector/deployment/#gateway). The controller is configured to receive metrics in the OTLP protocol and forward them to a configurable OTLP backend. +The metric controller creates an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) DaemonSet and Deployment and related Kubernetes objects from a `MetricPipeline` CR. The collector is deployed as a [Gateway](https://opentelemetry.io/docs/collector/deployment/#gateway). The controller is configured to receive metrics in the OTLP protocol and forward them to a configurable OTLP backend. -For more information, see [Metrics](./docs/user/04-metrics.md). +For more information, see [Metrics](./docs/user/collecting-metrics/README.md). ## Installation diff --git a/docs/user/01-manager.md b/docs/user/01-manager.md index b9bb5ad81f..b6b0b4f756 100644 --- a/docs/user/01-manager.md +++ b/docs/user/01-manager.md @@ -1,25 +1,3 @@ # Telemetry Manager -As the core element of the Telemetry module, Telemetry Manager manages the lifecycle of other Telemetry module components by watching user-created resources. - -## Module Lifecycle - -The Telemetry module includes Telemetry Manager, a Kubernetes [operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) that's described by a custom resource of type Telemetry. Telemetry Manager has the following tasks: - -1. Watch the module configuration for changes and sync the module status to it. -2. Watch for the user-created Kubernetes resources LogPipeline, TracePipeline, and MetricPipeline. In these resources, you specify what data of a signal type to collect and where to ship it. -3. Manage the lifecycle of the self monitor and the user-configured agents and gateways. - For example, only if you defined a LogPipeline resource, the Fluent Bit DaemonSet is deployed as log agent. - -![Manager](assets/manager-resources.drawio.svg) - -### Self Monitor - -The Telemetry module contains a self monitor, based on [Prometheus](https://prometheus.io/), to collect and evaluate metrics from the managed gateways and agents. Telemetry Manager retrieves the current pipeline health from the self monitor and adjusts the status of the pipeline resources and the module status. -Additionally, you can monitor the health of your pipelines in an integrated backend like [SAP Cloud Logging](./integration/sap-cloud-logging/README.md#use-sap-cloud-logging-alerts): To set up alerts and reports in the backend, use the [pipeline health metrics](./04-metrics.md#5-monitor-pipeline-health) emitted by your MetricPipeline. - -![Self-Monitor](assets/manager-arch.drawio.svg) - -## Module Configuration and Status - -For configuration options and the overall status of the module, see the specification of the related [Telemetry resource](./resources/01-telemetry.md). +This content moved to [Architecture](./architecture/README.md). diff --git a/docs/user/02-logs.md b/docs/user/02-logs.md index 919b3ea653..a5f9d1107e 100644 --- a/docs/user/02-logs.md +++ b/docs/user/02-logs.md @@ -1,12 +1,15 @@ # Application Logs (Fluent Bit) +> [!NOTE] +> The following API uses the LogPipeline output `http` and `custom` based on a FluentBit agent. As a new user, start with the approach based on OpenTelemetry: [Collecting Logs](./collecting-logs/README.md). + With application logs, you can debug an application and derive the internal state of an application. When logs are emitted with the correct severity level and context, they're essential for observing an application. ## Overview -The Telemetry module provides the [Fluent Bit](https://fluentbit.io/) log agent for the collection and shipment of application logs of any container running in the Kyma runtime. +The Telemetry module provides the [Fluent Bit](https://fluentbit.io/) log agent for the collection and shipment of application logs of any container running in Kyma runtime. -You can configure the log agent with external systems using runtime configuration with a dedicated Kubernetes API ([CRD](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#customresourcedefinitions)) named `LogPipeline`. With the LogPipeline's HTTP output, you can natively integrate with vendors that support this output, or with any vendor using a [Fluentd integration](https://medium.com/hepsiburadatech/fluent-logging-architecture-fluent-bit-fluentd-elasticsearch-ca4a898e28aa). +You can configure the log agent with external systems using runtime configuration with a dedicated Kubernetes API ([CRD](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#customresourcedefinitions)) named `LogPipeline`. With the LogPipeline's HTTP output, you can natively integrate with vendors that support this output, or with any vendor using a Fluentd integration. The feature is optional, if you don't want to use the Logs feature, simply don't set up a LogPipeline. @@ -478,7 +481,7 @@ The enriched timestamp attributes have the following meaning: The Telemetry module ensures that the log agent instances are operational and healthy at any time, for example, with buffering and retries. However, there may be situations when the instances drop logs, or cannot handle the log load. -To detect and fix such situations, check the [pipeline status](./resources/02-logpipeline.md#logpipeline-status) and check out [Troubleshooting](#troubleshooting). If you have set up [pipeline health monitoring](./04-metrics.md#5-monitor-pipeline-health), check the alerts and reports in an integrated backend like [SAP Cloud Logging](./integration/sap-cloud-logging/README.md#use-sap-cloud-logging-alerts). +To detect and fix such situations, check the [pipeline status](./resources/02-logpipeline.md#logpipeline-status) and check out [Troubleshooting](#troubleshooting). If you have set up [pipeline health monitoring](./monitor-pipeline-health.md), check the alerts and reports in an integrated backend like [SAP Cloud Logging](./integration/sap-cloud-logging/README.md#use-sap-cloud-logging-alerts). > [!WARNING] > It's not recommended to access the metrics endpoint of the used FluentBit instances directly, because the exposed metrics are no official API of the Kyma Telemetry module. Breaking changes can happen if the underlying FluentBit version introduces such. diff --git a/docs/user/03-traces.md b/docs/user/03-traces.md index 8c6bf70636..53e9e53d12 100644 --- a/docs/user/03-traces.md +++ b/docs/user/03-traces.md @@ -1,524 +1,3 @@ # Traces -The Telemetry module supports you in collecting all relevant trace data in a Kyma cluster, enriches them and ships them to a backend for further analysis. Kyma modules like Istio or Serverless contribute traces transparently. You can choose among multiple vendors for [OTLP-based backends](https://opentelemetry.io/ecosystem/vendors/). - -## Overview - -Observability tools aim to show the big picture, no matter if you're monitoring just a few or many components. In a cloud-native microservice architecture, a user request often flows through dozens of different microservices. Logging and monitoring tools help to track the request's path. However, they treat each component or microservice in isolation. This individual treatment results in operational issues. - -[Distributed tracing](https://opentelemetry.io/docs/concepts/observability-primer/#understanding-distributed-tracing) charts out the transactions in cloud-native systems, helping you to understand the application behavior and relations between the frontend actions and backend implementation. - -The following diagram shows how distributed tracing helps to track the request path: - -![Distributed tracing](./assets/traces-intro.drawio.svg) - -The Telemetry module provides a trace gateway for the shipment of traces of any container running in the Kyma runtime. - -You can configure the trace gateway with external systems using runtime configuration with a dedicated Kubernetes API ([CRD](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#customresourcedefinitions)) named TracePipeline. -The Trace feature is optional. If you don't want to use it, simply don't set up a TracePipeline. - -## Prerequisites - -For the recording of a distributed trace, every involved component must propagate at least the trace context. For details, see [Trace Context](https://www.w3.org/TR/trace-context/#problem-statement). - -- In Kyma, all modules involved in users’ requests support the [W3C Trace Context](https://www.w3.org/TR/trace-context) protocol. The involved Kyma modules are, for example, Istio, Serverless, and Eventing. -- Your application also must propagate the W3C Trace Context for any user-related activity. This can be achieved easily using the [Open Telemetry SDKs](https://opentelemetry.io/docs/instrumentation/) available for all common programming languages. If your application follows that guidance and is part of the Istio Service Mesh, it’s already outlined with dedicated span data in the trace data collected by the Kyma telemetry setup. -- Furthermore, your application must enrich a trace with additional span data and send these data to the cluster-central telemetry services. You can achieve this with [Open Telemetry SDKs](https://opentelemetry.io/docs/instrumentation/). - -## Architecture - -In the Kyma cluster, the Telemetry module provides a central deployment of an [OTel Collector](https://opentelemetry.io/docs/collector/) acting as a gateway. The gateway exposes endpoints to which all Kyma modules and users’ applications should send the trace data. - -![Architecture](./assets/traces-arch.drawio.svg) - -1. An end-to-end request is triggered and populated across the distributed application. Every involved component propagates the trace context using the [W3C Trace Context](https://www.w3.org/TR/trace-context/) protocol. -2. After contributing a new span to the trace, the involved components send the related span data to the trace gateway using the `telemetry-otlp-traces` service. The communication happens based on the [OpenTelemetry Protocol (OTLP)](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md) either using GRPC or HTTP. -3. Istio sends the related span data to the trace gateway as well. -4. The trace gateway discovers metadata that's typical for sources running on Kubernetes, like Pod identifiers, and then enriches the span data with that metadata. -5. Telemetry Manager configures the gateway according to the `TracePipeline` resource, including the target backend for the trace gateway. Also, it observes the trace flow to the backend and reports problems in the `TracePipeline` status. -6. The trace gateway sends the data to the observability system that's specified in your `TracePipeline` resource - either within the Kyma cluster, or, if authentication is set up, to an external observability backend. -7. You can analyze the trace data with your preferred backend system. - -### Telemetry Manager - -The TracePipeline resource is watched by Telemetry Manager, which is responsible for generating the custom parts of the OTel Collector configuration. - -![Manager resources](./assets/traces-resources.drawio.svg) - -1. Telemetry Manager watches all TracePipeline resources and related Secrets. -2. Furthermore, Telemetry Manager takes care of the full lifecycle of the OTel Collector Deployment itself. Only if you defined a TracePipeline, the collector is deployed. -3. Whenever the configuration changes, it validates the configuration and generates a new configuration for OTel Collector, where a ConfigMap for the configuration is generated. -4. Referenced Secrets are copied into one Secret that is mounted to the OTel Collector as well. - -### Trace Gateway - -In a Kyma cluster, the trace gateway is the central component to which all components can send their individual spans. The gateway collects, enriches, and dispatches the data to the configured backend. For more information, see [Telemetry Gateways](./gateways.md). - -## Setting up a TracePipeline - -In the following steps, you can see how to construct and deploy a typical TracePipeline. Learn more about the available [parameters and attributes](resources/04-tracepipeline.md). - -### 1. Create a TracePipeline - -To ship traces to a new OTLP output, create a resource of the kind `TracePipeline` and save the file (named, for example, `tracepipeline.yaml`). - -This configures the underlying OTel Collector with a pipeline for traces and opens a push endpoint that is accessible with the `telemetry-otlp-traces` service. For details, see [Gateway Usage](./gateways.md#usage). The following push URLs are set up: - -- GRPC: 'http://telemetry-otlp-traces.kyma-system:4317' -- HTTP: 'http://telemetry-otlp-traces.kyma-system:4318' - -The default protocol for shipping the data to a backend is GRPC, but you can choose HTTP instead. Depending on the configured protocol, an `otlp` or an `otlphttp` exporter is used. Ensure that the correct port is configured as part of the endpoint. - -- For GRPC, use: - - ```yaml - apiVersion: telemetry.kyma-project.io/v1alpha1 - kind: TracePipeline - metadata: - name: backend - spec: - output: - otlp: - endpoint: - value: https://backend.example.com:4317 - ``` - -- For HTTP, use the `protocol` attribute: - - ```yaml - apiVersion: telemetry.kyma-project.io/v1alpha1 - kind: TracePipeline - metadata: - name: backend - spec: - output: - otlp: - protocol: http - endpoint: - value: https://backend.example.com:4318 - ``` - -### 2. Enable Istio Tracing - -By default, the tracing feature of the Istio module is disabled to avoid increased network utilization if there is no TracePipeline. - -To activate the Istio tracing feature with a sampling rate of 5% (for recommendations, see [Istio](#istio)), use a resource similar to the following example: - -```yaml -apiVersion: telemetry.istio.io/v1 -kind: Telemetry -metadata: - name: tracing-default - namespace: istio-system -spec: - tracing: - - providers: - - name: "kyma-traces" - randomSamplingPercentage: 5.00 -``` - -### 3a. Add Authentication Details From Plain Text - -To integrate with external systems, you must configure authentication details. You can use mutual TLS (mTLS), Basic Authentication, or custom headers: - - - -#### **mTLS** - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: TracePipeline -metadata: - name: backend -spec: - output: - otlp: - endpoint: - value: https://backend.example.com/otlp:4317 - tls: - cert: - value: | - -----BEGIN CERTIFICATE----- - ... - key: - value: | - -----BEGIN RSA PRIVATE KEY----- - ... -``` - -#### **Basic Authentication** - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: TracePipeline -metadata: - name: backend -spec: - output: - otlp: - endpoint: - value: https://backend.example.com/otlp:4317 - authentication: - basic: - user: - value: myUser - password: - value: myPwd -``` - -#### **Token-based authentication with custom headers** - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: TracePipeline -metadata: - name: backend -spec: - output: - otlp: - endpoint: - value: https://backend.example.com/otlp:4317 - headers: - - name: Authorization - prefix: Bearer - value: "myToken" -``` - - - -### 3b. Add Authentication Details From Secrets - -Integrations into external systems usually need authentication details dealing with sensitive data. To handle that data properly in Secrets, TracePipeline supports the reference of Secrets. - -Using the **valueFrom** attribute, you can map Secret keys for mutual TLS (mTLS), Basic Authentication, or with custom headers. - -You can store the value of the token in the referenced Secret without any prefix or scheme, and you can configure it in the `headers` section of the TracePipeline. In the following example, the token has the prefix "Bearer". - - - -#### **mTLS** - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: TracePipeline -metadata: - name: backend -spec: - output: - otlp: - endpoint: - value: https://backend.example.com/otlp:4317 - tls: - cert: - valueFrom: - secretKeyRef: - name: backend - namespace: default - key: cert - key: - valueFrom: - secretKeyRef: - name: backend - namespace: default - key: key -``` - -#### **Basic Authentication** - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: TracePipeline -metadata: - name: backend -spec: - output: - otlp: - endpoint: - valueFrom: - secretKeyRef: - name: backend - namespace: default - key: endpoint - authentication: - basic: - user: - valueFrom: - secretKeyRef: - name: backend - namespace: default - key: user - password: - valueFrom: - secretKeyRef: - name: backend - namespace: default - key: password -``` - -#### **Token-based authentication with custom headers** - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: TracePipeline -metadata: - name: backend -spec: - output: - otlp: - endpoint: - value: https://backend.example.com:4317 - headers: - - name: Authorization - prefix: Bearer - valueFrom: - secretKeyRef: - name: backend - namespace: default - key: token -``` - - - -The related Secret must have the referenced name, be located in the referenced namespace, and contain the mapped key. See the following example: - -```yaml -kind: Secret -apiVersion: v1 -metadata: - name: backend - namespace: default -stringData: - endpoint: https://backend.example.com:4317 - user: myUser - password: XXX - token: YYY -``` - -### 4. Rotate the Secret - -Telemetry Manager continuously watches the Secret referenced with the **secretKeyRef** construct. You can update the Secret’s values, and Telemetry Manager detects the changes and applies the new Secret to the setup. - -> [!TIP] -> If you use a Secret owned by the [SAP BTP Service Operator](https://github.com/SAP/sap-btp-service-operator), you can configure an automated rotation using a `credentialsRotationPolicy` with a specific `rotationFrequency` and don’t have to intervene manually. - -### 5. Deploy the Pipeline - -To activate the TracePipeline, apply the `tracepipeline.yaml` resource file in your cluster: - -```bash -kubectl apply -f tracepipeline.yaml -``` - -### Result - -You activated a TracePipeline and traces start streaming to your backend. - -To check that the pipeline is running, wait until the status conditions of the TracePipeline in your cluster have status `True`: - -```bash -kubectl get tracepipeline -NAME CONFIGURATION GENERATED GATEWAY HEALTHY FLOW HEALTHY -backend True True True -``` - -## Kyma Modules With Tracing Capabilities - -Kyma bundles several modules that can be involved in user flows. Applications involved in a distributed trace must propagate the trace context to keep the trace complete. Optionally, they can enrich the trace with custom spans, which requires reporting them to the backend. - -### Istio - -The Istio module is crucial in distributed tracing because it provides the [Ingress Gateway](https://istio.io/latest/docs/tasks/traffic-management/ingress/ingress-control/). Typically, this is where external requests enter the cluster scope and are enriched with trace context if it hasn’t happened earlier. Furthermore, every component that’s part of the Istio Service Mesh runs an Istio proxy, which propagates the context properly but also creates span data. If Istio tracing is activated and taking care of trace propagation in your application, you get a complete picture of a trace, because every component automatically contributes span data. Also, Istio tracing is pre-configured to be based on the vendor-neutral [W3C Trace Context](https://www.w3.org/TR/trace-context/) protocol. - -The Istio module is configured with an [extension provider](https://istio.io/latest/docs/tasks/observability/telemetry/) called `kyma-traces`. To activate the provider on the global mesh level using the Istio [Telemetry API](https://istio.io/latest/docs/reference/config/telemetry/#Tracing), place a resource to the `istio-system` namespace. The following code samples help setting up the Istio tracing feature: - - - -#### **Extension Provider** - -The following example configures all Istio proxies with the `kyma-traces` extension provider, which, by default, reports span data to the trace gateway of the Telemetry module. - -```yaml -apiVersion: telemetry.istio.io/v1 -kind: Telemetry -metadata: - name: mesh-default - namespace: istio-system -spec: - tracing: - - providers: - - name: "kyma-traces" -``` - -#### **Sampling Rate** - -By default, the sampling rate is configured to 1%. That means that only 1 trace out of 100 traces is reported to the trace gateway, and all others are dropped. The sampling decision itself is propagated as part of the [trace context](https://www.w3.org/TR/trace-context/#sampled-flag) so that either all involved components are reporting the span data of a trace, or none. - -> [!TIP] -> If you increase the sampling rate, you send more data your tracing backend and cause much higher network utilization in the cluster. -> To reduce costs and performance impacts in a production setup, a very low percentage of around 5% is recommended. - -To configure an "always-on" sampling, set the sampling rate to 100%: - -```yaml -apiVersion: telemetry.istio.io/v1 -kind: Telemetry -metadata: - name: mesh-default - namespace: istio-system -spec: - tracing: - - providers: - - name: "kyma-traces" - randomSamplingPercentage: 100.00 -``` - -#### **Namespaces or Workloads** - -If you need specific settings for individual namespaces or workloads, place additional Telemetry resources. If you don't want to report spans at all for a specific workload, activate the `disableSpanReporting` flag with the selector expression. - -```yaml -apiVersion: telemetry.istio.io/v1 -kind: Telemetry -metadata: - name: tracing - namespace: my-namespace -spec: - selector: - matchLabels: - app.kubernetes.io/name: "my-app" - tracing: - - providers: - - name: "kyma-traces" - randomSamplingPercentage: 100.00 -``` - -#### **Trace Context Without Spans** - -To enable the propagation of the [W3C Trace Context](https://www.w3.org/TR/trace-context/) only, without reporting any spans (so the actual tracing feature is disabled), you must enable the `kyma-traces` provider with a sampling rate of 0. With this configuration, you get the relevant trace context into the [access logs](https://kyma-project.io/#/istio/user/tutorials/01-45-enable-istio-access-logs) without any active trace reporting. - - ```yaml - apiVersion: telemetry.istio.io/v1 - kind: Telemetry - metadata: - name: mesh-default - namespace: istio-system - spec: - tracing: - - providers: - - name: "kyma-traces" - randomSamplingPercentage: 0 - ``` - - - -### Eventing - -The [Eventing](https://kyma-project.io/#/eventing-manager/user/README) module uses the [CloudEvents](https://cloudevents.io/) protocol (which natively supports the [W3C Trace Context](https://www.w3.org/TR/trace-context) propagation). Because of that, it propagates trace context properly. However, it doesn't enrich a trace with more advanced span data. - -### Serverless - -By default, all engines for the [Serverless](https://kyma-project.io/#/serverless-manager/user/README) module integrate the [Open Telemetry SDK](https://opentelemetry.io/docs/reference/specification/metrics/sdk/). Thus, the used middlewares are configured to automatically propagate the trace context for chained calls. - -Because the Telemetry endpoints are configured by default, Serverless also reports custom spans for incoming and outgoing requests. You can [customize Function traces](https://kyma-project.io/#/serverless-manager/user/tutorials/01-100-customize-function-traces) to add more spans as part of your Serverless source code. - -## Operations - -A TracePipeline runs several OTel Collector instances in your cluster. This Deployment serves OTLP endpoints and ships received data to the configured backend. - -The Telemetry module ensures that the OTel Collector instances are operational and healthy at any time, for example, with buffering and retries. However, there may be situations when the instances drop traces, or cannot handle the trace load. - -To detect and fix such situations, check the [pipeline status](./resources/04-tracepipeline.md#tracepipeline-status) and check out [Troubleshooting](#troubleshooting). If you have set up [pipeline health monitoring](./04-metrics.md#5-monitor-pipeline-health), check the alerts and reports in an integrated backend like [SAP Cloud Logging](./integration/sap-cloud-logging/README.md#use-sap-cloud-logging-alerts). - -> [! WARNING] -> It's not recommended to access the metrics endpoint of the used OTel Collector instances directly, because the exposed metrics are no official API of the Kyma Telemetry module. Breaking changes can happen if the underlying OTel Collector version introduces such. -> Instead, use the [pipeline status](./resources/04-tracepipeline.md#tracepipeline-status). - -## Limitations - -- **Throughput**: Assuming an average span with 40 attributes with 64 characters, the maximum throughput is 4200 span/sec ~= 15.000.000 spans/hour. If this limit is exceeded, spans are refused. To increase the maximum throughput, manually scale out the gateway by increasing the number of replicas for the trace gateway. See [Module Configuration and Status](https://kyma-project.io/#/telemetry-manager/user/01-manager?id=module-configuration). -- **Unavailability of Output**: For up to 5 minutes, a retry for data is attempted when the destination is unavailable. After that, data is dropped. -- **No Guaranteed Delivery**: The used buffers are volatile. If the OTel Collector instance crashes, trace data can be lost. -- **Multiple TracePipeline Support**: The maximum amount of TracePipeline resources is 5. -- **System Span Filtering**: System-related spans reported by Istio are filtered out without the opt-out option, for example: - - Any communication of applications to the Telemetry gateways - - Any communication from the gateways to backends - -## Troubleshooting - -### No Spans Arrive at the Backend - -**Symptom**: In the TracePipeline status, the `TelemetryFlowHealthy` condition has status **GatewayAllTelemetryDataDropped**. - -**Cause**: Incorrect backend endpoint configuration (such as using the wrong authentication credentials), or the backend is unreachable. - -**Solution**: - -1. Check the `telemetry-trace-gateway` Pods for error logs by calling `kubectl logs -n kyma-system {POD_NAME}`. -2. Check if the backend is up and reachable. -3. Fix the errors. - -### Not All Spans Arrive at the Backend - -**Symptom**: - -- The backend is reachable and the connection is properly configured, but some spans are refused. -- In the TracePipeline status, the `TelemetryFlowHealthy` condition has status **GatewaySomeTelemetryDataDropped**. - -**Cause**: It can happen due to a variety of reasons - for example, the backend is limiting the ingestion rate. - -**Solution**: - -1. Check the `telemetry-trace-gateway` Pods for error logs by calling `kubectl logs -n kyma-system {POD_NAME}`. Also, check your observability backend to investigate potential causes. -2. If the backend is limiting the rate by refusing spans, try the following options: - - Option 1: Increase maximum backend ingestion rate. For example, by scaling out the SAP Cloud Logging instances. - - Option 2: Reduce the emitted spans in your applications. -3. Otherwise, take the actions appropriate to the cause indicated in the logs. - -### Custom Spans Don’t Arrive at the Backend, but Istio Spans Do - -**Cause**: Your SDK version is incompatible with the OTel Collector version. - -**Solution**: - -1. Check which SDK version you are using for instrumentation. -2. Investigate whether it is compatible with the OTel Collector version. -3. If required, upgrade to a supported SDK version. - -### Trace Backend Shows Fewer Traces than Expected - -**Cause**: By [default](#istio), only 1% of the requests are sent to the trace backend for trace recording. - -**Solution**: - -To see more traces in the trace backend, increase the percentage of requests by changing the default settings. -If you just want to see traces for one particular request, you can manually force sampling: - -1. Create a `values.yaml` file. - The following example sets the value to `60`, which means 60% of the requests are sent to the tracing backend. - -```yaml - apiVersion: telemetry.istio.io/v1 - kind: Telemetry - metadata: - name: kyma-traces - namespace: istio-system - spec: - tracing: - - providers: - - name: "kyma-traces" - randomSamplingPercentage: 60 -``` - -2. To override the default percentage, change the value for the **randomSamplingPercentage** attribute. -3. Deploy the `values.yaml` to your existing Kyma installation. - -### Gateway Throttling - -**Symptom**: - -- In the TracePipeline status, the `TelemetryFlowHealthy` condition has status **GatewayThrottling**. -- Also, your application might have error logs indicating a refusal for sending traces to the gateway. - -**Cause**: Gateway cannot receive spans at the given rate. - -**Solution**: Manually scale out the gateway by increasing the number of replicas for the trace gateway. See [Module Configuration and Status](https://kyma-project.io/#/telemetry-manager/user/01-manager?id=module-configuration). +This content moved to [Collecting Traces](./collecting-traces/README.md). diff --git a/docs/user/04-metrics.md b/docs/user/04-metrics.md index 6b4746c62a..9b71100096 100644 --- a/docs/user/04-metrics.md +++ b/docs/user/04-metrics.md @@ -1,879 +1,3 @@ # Metrics -The goal of the Telemetry module is to support you in collecting all relevant metrics of a workload in a Kyma cluster and ship them to a backend for further analysis. Kyma modules like [Istio](https://kyma-project.io/#/istio/user/README) or [Serverless](https://kyma-project.io/#/serverless-manager/user/README) contribute metrics instantly, and the Telemetry module enriches the data. You can choose among multiple [vendors for OTLP-based backends](https://opentelemetry.io/ecosystem/vendors/). - -## Overview - -Observability is all about exposing the internals of the components belonging to a distributed application and making that data analysable at a central place. -While application logs and traces usually provide request-oriented data, metrics are aggregated statistics exposed by a component to reflect the internal state. Typical statistics like the amount of processed requests, or the amount of registered users, can be very useful to monitor the current state and also the health of a component. Also, you can define proactive and reactive alerts if metrics are about to reach thresholds, or if they already passed thresholds. - -The Telemetry module provides a metric gateway and, optionally, an agent for the collection and shipment of metrics of any container running in the Kyma runtime. - -You can configure the metric gateway with external systems using runtime configuration with a dedicated Kubernetes API ([CRD](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#customresourcedefinitions)) named MetricPipeline. -The Metric feature is optional. If you don't want to use it, simply don't set up a MetricPipeline. - -## Prerequisites - -- Before you can collect metrics data from a component, it must expose (or instrument) the metrics. Typically, it instruments specific metrics for the used language runtime (like Node.js) and custom metrics specific to the business logic. Also, the exposure can be in different formats, like the pull-based Prometheus format or the [push-based OTLP format](https://opentelemetry.io/docs/specs/otlp/). - -- If you want to use Prometheus-based metrics, you must have instrumented your application using a library like the [Prometheus client library](https://prometheus.io/docs/instrumenting/clientlibs/), with a port in your workload exposed serving as a Prometheus metrics endpoint. - -- For the instrumentation, you typically use an SDK, namely the [Prometheus client libraries](https://prometheus.io/docs/instrumenting/clientlibs/) or the [Open Telemetry SDKs](https://opentelemetry.io/docs/instrumentation/). Both libraries provide extensions to activate language-specific auto-instrumentation like for Node.js, and an API to implement custom instrumentation. - -## Architecture - -In the Telemetry module, a central in-cluster Deployment of an [OTel Collector](https://opentelemetry.io/docs/collector/) acts as a gateway. The gateway exposes endpoints for the [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/specs/otlp/) for GRPC and HTTP-based communication using the dedicated `telemetry-otlp-metrics` service, to which all Kyma modules and users’ applications send the metrics data. - -Optionally, the Telemetry module provides a DaemonSet of an OTel Collector acting as an agent. This agent can pull metrics of a workload and the Istio sidecar in the [Prometheus pull-based format](https://prometheus.io/docs/instrumenting/exposition_formats) and can provide runtime-specific metrics for the workload. - -![Architecture](./assets/metrics-arch.drawio.svg) - -1. An application (exposing metrics in OTLP) sends metrics to the central metric gateway service. -2. An application (exposing metrics in Prometheus protocol) activates the agent to scrape the metrics with an annotation-based configuration. -3. Additionally, you can activate the agent to pull metrics of each Istio sidecar. -4. The agent supports collecting metrics from the Kubelet and Kubernetes APIServer. -5. The agent converts and sends all collected metric data to the gateway in OTLP. -6. The gateway discovers the metadata and enriches all received data with typical metadata of the source by communicating with the Kubernetes APIServer. Furthermore, it filters data according to the pipeline configuration. -7. Telemetry Manager configures the agent and gateway according to the `MetricPipeline` resource specification, including the target backend for the metric gateway. Also, it observes the metrics flow to the backend and reports problems in the MetricPipeline status. -8. The metric gateway sends the data to the observability system that's specified in your `MetricPipeline` resource - either within the Kyma cluster, or, if authentication is set up, to an external observability backend. -9. You can analyze the metric data with your preferred backend system. - -### Telemetry Manager - -The MetricPipeline resource is watched by Telemetry Manager, which is responsible for generating the custom parts of the OTel Collector configuration. - -![Manager resources](./assets/metrics-resources.drawio.svg) - -1. Telemetry Manager watches all MetricPipeline resources and related Secrets. -2. Furthermore, Telemetry Manager takes care of the full lifecycle of the gateway Deployment and the agent DaemonSet. Only if you defined a MetricPipeline, the gateway and agent are deployed. -3. Whenever the user configuration changes, Telemetry Manager validates it and generates a single configuration for the gateway and agent. -4. Referenced Secrets are copied into one Secret that is mounted to the gateway as well. - -### Metric Gateway - -In a Kyma cluster, the metric gateway is the central component to which all components can send their individual metrics. The gateway collects, enriches, and dispatches the data to the configured backend. For more information, see [Telemetry Gateways](./gateways.md). - -### Metric Agent - -If a MetricPipeline configures a feature in the `input` section, an additional DaemonSet is deployed acting as an agent. The agent is also based on an [OTel Collector](https://opentelemetry.io/docs/collector/) and encompasses the collection and conversion of Prometheus-based metrics. Hereby, the workload puts a `prometheus.io/scrape` annotation on the specification of the Pod or service, and the agent collects it. The agent sends all data in OTLP to the central gateway. - -## Setting up a MetricPipeline - -In the following steps, you can see how to construct and deploy a typical MetricPipeline. Learn more about the available [parameters and attributes](resources/05-metricpipeline.md). - -### 1. Create a MetricPipeline - -To ship metrics to a new OTLP output, create a resource of the kind `MetricPipeline` and save the file (named, for example, `metricpipeline.yaml`). - -This configures the underlying OTel Collector with a pipeline for metrics and opens a push endpoint that is accessible with the `telemetry-otlp-metrics` service. For details, see [Gateway Usage](./gateways.md#usage). - -The following push URLs are set up: - -- GRPC: `http://telemetry-otlp-metrics.kyma-system:4317` -- HTTP: `http://telemetry-otlp-metrics.kyma-system:4318` - -The default protocol for shipping the data to a backend is GRPC, but you can choose HTTP instead. Depending on the configured protocol, an `otlp` or an `otlphttp` exporter is used. Ensure that the correct port is configured as part of the endpoint. - - - -#### **GRPC** - -For GRPC, use: - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: MetricPipeline -metadata: - name: backend -spec: - output: - otlp: - endpoint: - value: https://backend.example.com:4317 -``` - -#### **HTTP** - -For HTTP, use the `protocol` attribute: - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: MetricPipeline -metadata: - name: backend -spec: - output: - otlp: - protocol: http - endpoint: - value: https://backend.example.com:4318 -``` - - - -### 2a. Add Authentication Details From Plain Text - -To integrate with external systems, you must configure authentication details. You can use mutual TLS (mTLS), Basic Authentication, or custom headers: - - - -#### **mTLS** - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: MetricPipeline -metadata: - name: backend -spec: - output: - otlp: - endpoint: - value: https://backend.example.com/otlp:4317 - tls: - cert: - value: | - -----BEGIN CERTIFICATE----- - ... - key: - value: | - -----BEGIN RSA PRIVATE KEY----- - ... -``` - -#### **Basic Authentication** - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: MetricPipeline -metadata: - name: backend -spec: - output: - otlp: - endpoint: - value: https://backend.example.com/otlp:4317 - authentication: - basic: - user: - value: myUser - password: - value: myPwd -``` - -#### **Token-based authentication with custom headers** - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: MetricPipeline -metadata: - name: backend -spec: - output: - otlp: - endpoint: - value: https://backend.example.com/otlp:4317 - headers: - - name: Authorization - prefix: Bearer - value: "myToken" -``` - - -### 2b. Add Authentication Details From Secrets - -Integrations into external systems usually need authentication details dealing with sensitive data. To handle that data properly in Secrets, MetricsPipeline supports the reference of Secrets. - -Using the **valueFrom** attribute, you can map Secret keys for mutual TLS (mTLS), Basic Authentication, or with custom headers. - -You can store the value of the token in the referenced Secret without any prefix or scheme, and you can configure it in the headers section of the MetricPipeline. In this example, the token has the prefix “Bearer”. - - - -#### **mTLS** - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: MetricPipeline -metadata: - name: backend -spec: - output: - otlp: - endpoint: - value: https://backend.example.com/otlp:4317 - tls: - cert: - valueFrom: - secretKeyRef: - name: backend - namespace: default - key: cert - key: - valueFrom: - secretKeyRef: - name: backend - namespace: default - key: key -``` - -#### **Basic Authentication** - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: MetricPipeline -metadata: - name: backend -spec: - output: - otlp: - endpoint: - valueFrom: - secretKeyRef: - name: backend - namespace: default - key: endpoint - authentication: - basic: - user: - valueFrom: - secretKeyRef: - name: backend - namespace: default - key: user - password: - valueFrom: - secretKeyRef: - name: backend - namespace: default - key: password -``` - -#### **Token-based authentication with custom headers** - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: MetricPipeline -metadata: - name: backend -spec: - output: - otlp: - endpoint: - value: https://backend.example.com:4317 - headers: - - name: Authorization - prefix: Bearer - valueFrom: - secretKeyRef: - name: backend - namespace: default - key: token -``` - - - -The related Secret must have the referenced name, be located in the referenced namespace, and contain the mapped key. See the following example: - -```yaml -kind: Secret -apiVersion: v1 -metadata: - name: backend - namespace: default -stringData: - endpoint: https://backend.example.com:4317 - user: myUser - password: XXX - token: YYY -``` - -### 3. Rotate the Secret - -Telemetry Manager continuously watches the Secret referenced with the **secretKeyRef** construct. You can update the Secret’s values, and Telemetry Manager detects the changes and applies the new Secret to the setup. - -> [!TIP] -> If you use a Secret owned by the [SAP BTP Service Operator](https://github.com/SAP/sap-btp-service-operator), you can configure an automated rotation using a `credentialsRotationPolicy` with a specific `rotationFrequency` and don’t have to intervene manually. - -### 4. Activate Prometheus-Based Metrics - -> [!NOTE] -> For the following approach, you must have instrumented your application using a library like the [Prometheus client library](https://prometheus.io/docs/instrumenting/clientlibs/), with a port in your workload exposed serving as a Prometheus metrics endpoint. - -To enable collection of Prometheus-based metrics, define a MetricPipeline that has the `prometheus` section enabled as input: - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: MetricPipeline -metadata: - name: backend -spec: - input: - prometheus: - enabled: true - output: - otlp: - endpoint: - value: https://backend.example.com:4317 -``` - -The Metric agent is configured with a generic scrape configuration, which uses annotations to specify the endpoints to scrape in the cluster. - -For metrics ingestion to start automatically, use the annotations of the following table. -If an Istio sidecar is present, apply them to a Service that resolves your metrics port. -By annotating the Service, all endpoints targeted by the Service are resolved and scraped by the Metric agent bypassing the Service itself. -Only if Istio sidecar is not present, you can alternatively apply the annotations directly to the Pod. - -| Annotation Key | Example Values | Default Value | Description | -|------------------------------------------------------------------|-------------------|-------------- |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `prometheus.io/scrape` (mandatory) | `true`, `false` | none | Controls whether Prometheus Receiver automatically scrapes metrics from this target. | -| `prometheus.io/port` (mandatory) | `8080`, `9100` | none | Specifies the port where the metrics are exposed. | -| `prometheus.io/path` | `/metrics`, `/custom_metrics` | `/metrics` | Defines the HTTP path where Prometheus Receiver can find metrics data. | -| `prometheus.io/scheme` (only relevant when annotating a Service) | `http`, `https` | If Istio is active, `https` is supported; otherwise, only `http` is available. The default scheme is `http` unless an Istio sidecar is present, denoted by the label `security.istio.io/tlsMode=istio`, in which case `https` becomes the default. | Determines the protocol used for scraping metrics — either HTTPS with mTLS or plain HTTP. | -| `prometheus.io/param_: ` | `prometheus.io/param_format: prometheus` | none | Instructs Prometheus Receiver to pass name-value pairs as URL parameters when calling the metrics endpoint. | - -If you're running the Pod targeted by a Service with Istio, Istio must be able to derive the [appProtocol](https://kubernetes.io/docs/concepts/services-networking/service/#application-protocol) from the Service port definition; otherwise the communication for scraping the metric endpoint cannot be established. You must either prefix the port name with the protocol like in `http-metrics`, or explicitly define the `appProtocol` attribute. - -For example, see the following `Service` configuration: - -```yaml -apiVersion: v1 -kind: Service -metadata: - annotations: - prometheus.io/port: "8080" - prometheus.io/scrape: "true" - name: sample -spec: - ports: - - name: http-metrics - appProtocol: http - port: 8080 - protocol: TCP - targetPort: 8080 - selector: - app: sample - type: ClusterIP -``` - -> [!NOTE] -> The Metric agent can scrape endpoints even if the workload is a part of the Istio service mesh and accepts mTLS communication. However, there's a constraint: For scraping through HTTPS, Istio must configure the workload using 'STRICT' mTLS mode. Without 'STRICT' mTLS mode, you can set up scraping through HTTP by applying the annotation `prometheus.io/scheme=http`. For related troubleshooting, see [Log Entry: Failed to Scrape Prometheus Endpoint](#log-entry-failed-to-scrape-prometheus-endpoint). - -### 5. Monitor Pipeline Health - -By default, a MetricPipeline emits metrics about the health of all pipelines managed by the Telemetry module. Based on these metrics, you can track the status of every individual pipeline and set up alerting for it. - -Metrics for Pipelines and the Telemetry Module: - -| Metric | Description | Availability | -|---------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------| -| kyma.resource.status.conditions | Value represents status of different conditions reported by the resource. Possible values are 1 ("True"), 0 ("False"), and -1 (other status values) | Available for both, the pipelines and the Telemetry resource | -| kyma.resource.status.state | Value represents the state of the resource (if present) | Available for the Telemetry resource | - -Metric Attributes for Monitoring: - -| Name | Description | -|--------------------------|----------------------------------------------------------------------------------------------| -| metric.attributes.Type | Type of the condition | -| metric.attributes.status | Status of the condition | -| metric.attributes.reason | Contains a programmatic identifier indicating the reason for the condition's last transition | - -To set up alerting, use an alert rule. In the following example, the alert is triggered if metrics are not delivered to the backend: - -```text - min by (k8s_resource_name) ((kyma_resource_status_conditions{type="TelemetryFlowHealthy",k8s_resource_kind="metricpipelines"})) == 0 -``` - -### 6. Activate Runtime Metrics - -To enable collection of runtime metrics, define a MetricPipeline that has the `runtime` section enabled as input: - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: MetricPipeline -metadata: - name: backend -spec: - input: - runtime: - enabled: true - output: - otlp: - endpoint: - value: https://backend.example.com:4317 -``` - -By default, metrics for all resources (Pod, container, Node, Volume, DaemonSet, Deployment, StatefulSet, and Job) are collected. -To enable or disable the collection of metrics for a specific resource, use the `resources` section in the `runtime` input. - -The following example collects only DaemonSet, Deployment, StatefulSet, and Job metrics: - - ```yaml - apiVersion: telemetry.kyma-project.io/v1alpha1 - kind: MetricPipeline - metadata: - name: backend - spec: - input: - runtime: - enabled: true - resources: - pod: - enabled: false - container: - enabled: false - node: - enabled: false - volume: - enabled: false - daemonset: - enabled: true - deployment: - enabled: true - statefulset: - enabled: true - job: - enabled: true - output: - otlp: - endpoint: - value: https://backend.example.com:4317 - ``` - -If Pod metrics are enabled, the following metrics are collected: - -- From the [kubletstatsreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/kubeletstatsreceiver): - - `k8s.pod.cpu.capacity` - - `k8s.pod.cpu.usage` - - `k8s.pod.filesystem.available` - - `k8s.pod.filesystem.capacity` - - `k8s.pod.filesystem.usage` - - `k8s.pod.memory.available` - - `k8s.pod.memory.major_page_faults` - - `k8s.pod.memory.page_faults` - - `k8s.pod.memory.rss` - - `k8s.pod.memory.usage` - - `k8s.pod.memory.working_set` - - `k8s.pod.network.errors` - - `k8s.pod.network.io` -- From the [k8sclusterreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sclusterreceiver): - - `k8s.pod.phase` - -If container metrics are enabled, the following metrics are collected: - -- From the [kubletstatsreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/kubeletstatsreceiver): - - `container.cpu.time` - - `container.cpu.usage` - - `container.filesystem.available` - - `container.filesystem.capacity` - - `container.filesystem.usage` - - `container.memory.available` - - `container.memory.major_page_faults` - - `container.memory.page_faults` - - `container.memory.rss` - - `container.memory.usage` - - `container.memory.working_set` -- From the [k8sclusterreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sclusterreceiver): - - `k8s.container.cpu_request` - - `k8s.container.cpu_limit` - - `k8s.container.memory_request` - - `k8s.container.memory_limit` - - `k8s.container.restarts` - -If Node metrics are enabled, the following metrics are collected: - -- From the [kubletstatsreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/kubeletstatsreceiver): - - `k8s.node.cpu.usage` - - `k8s.node.filesystem.available` - - `k8s.node.filesystem.capacity` - - `k8s.node.filesystem.usage` - - `k8s.node.memory.available` - - `k8s.node.memory.usage` - - `k8s.node.network.errors`, - - `k8s.node.network.io`, - - `k8s.node.memory.rss` - - `k8s.node.memory.working_set` - -If Volume metrics are enabled, the following metrics are collected: - -- From the [kubletstatsreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/kubeletstatsreceiver): - - `k8s.volume.available` - - `k8s.volume.capacity` - - `k8s.volume.inodes` - - `k8s.volume.inodes.free` - - `k8s.volume.inodes.used` - -If Deployment metrics are enabled, the following metrics are collected: - -- From the [k8sclusterreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sclusterreceiver): - - `k8s.deployment.available` - - `k8s.deployment.desired` - -If DaemonSet metrics are enabled, the following metrics are collected: - -- From the [k8sclusterreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sclusterreceiver): - - `k8s.daemonset.current_scheduled_nodes` - - `k8s.daemonset.desired_scheduled_nodes` - - `k8s.daemonset.misscheduled_nodes` - - `k8s.daemonset.ready_nodes` - -If StatefulSet metrics are enabled, the following metrics are collected: - -- From the [k8sclusterreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sclusterreceiver): - - `k8s.statefulset.current_pods` - - `k8s.statefulset.desired_pods` - - `k8s.statefulset.ready_pods` - - `k8s.statefulset.updated_pods` - -If Job metrics are enabled, the following metrics are collected: - -- From the [k8sclusterreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sclusterreceiver): - - `k8s.job.active_pods` - - `k8s.job.desired_successful_pods` - - `k8s.job.failed_pods` - - `k8s.job.max_parallel_pods` - - `k8s.job.successful_pods` - -### 7. Activate Istio Metrics - -To enable collection of Istio metrics, define a MetricPipeline that has the `istio` section enabled as input: - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: MetricPipeline -metadata: - name: backend -spec: - input: - istio: - enabled: true - output: - otlp: - endpoint: - value: https://backend.example.com:4317 -``` - -With this, the agent starts collecting all Istio metrics from Istio sidecars. - -If you are using the `istio` input, you can also collect Envoy metrics. Envoy metrics provide insights into the performance and behavior of the Envoy proxy, such as request rates, latencies, and error counts. These metrics are useful for observability and troubleshooting service mesh traffic. - -For details, see the list of available [Envoy metrics](https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats) and [server metrics](https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/statistics). - -> [!NOTE] -> Envoy metrics are only available for the `istio` input. Ensure that Istio sidecars are correctly injected into your workloads for Envoy metrics to be available. - -By default, Envoy metrics collection is disabled. - -To activate Envoy metrics, enable the `envoyMetrics` section in the MetricPipeline specification under the `istio` input: - - ```yaml - apiVersion: telemetry.kyma-project.io/v1alpha1 - kind: MetricPipeline - metadata: - name: envoy-metrics - spec: - input: - istio: - enabled: true - envoyMetrics: - enabled: true - output: - otlp: - endpoint: - value: https://backend.example.com:4317 - ``` - -### 8. Deactivate OTLP Metrics - -By default, `otlp` input is enabled. - -To drop the push-based OTLP metrics that are received by the metric gateway, define a MetricPipeline that has the `otlp` section disabled as an input: - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: MetricPipeline -metadata: - name: backend -spec: - input: - istio: - enabled: true - otlp: - disabled: true - output: - otlp: - endpoint: - value: https://backend.example.com:4317 -``` - -With this, the agent starts collecting all Istio metrics from Istio sidecars, and the push-based OTLP metrics are dropped. - -### 9. Add Filters - -To filter metrics by namespaces, define a MetricPipeline that has the `namespaces` section defined in one of the inputs. For example, you can specify the namespaces from which metrics are collected or the namespaces from which metrics are dropped. Learn more about the available [parameters and attributes](resources/05-metricpipeline.md). - -The following example collects runtime metrics **only** from the `foo` and `bar` namespaces: - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: MetricPipeline -metadata: - name: backend -spec: - input: - runtime: - enabled: true - namespaces: - include: - - foo - - bar - output: - otlp: - endpoint: - value: https://backend.example.com:4317 -``` - -The following example collects runtime metrics from all namespaces **except** the `foo` and `bar` namespaces: - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: MetricPipeline -metadata: - name: backend -spec: - input: - runtime: - enabled: true - namespaces: - exclude: - - foo - - bar - output: - otlp: - endpoint: - value: https://backend.example.com:4317 -``` -The following example collects metrics from all namespaces including system namespaces: - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: MetricPipeline -metadata: - name: backend -spec: - input: - istio: - enabled: true - namespaces: {} - prometheus: - enabled: true - namespaces: {} - runtime: - enabled: true - namespaces: {} - output: - otlp: - endpoint: - value: https://backend.example.com:4317 -``` -To collect metrics with specific system namespaces, add them to the `include` section of namespaces. - -> [!NOTE] -> The default settings depend on the input: -> -> If no namespace selector is defined for the `prometheus` or `runtime` input, then metrics from system namespaces are excluded by default. -> -> However, if the namespace selector is not defined for the `istio` and `otlp` input, then metrics from system namespaces are included by default. - -### 10. Activate Diagnostic Metrics - -If you use the `prometheus` or `istio` input, for every metric source typical scrape metrics are produced, such as `up`, `scrape_duration_seconds`, `scrape_samples_scraped`, `scrape_samples_post_metric_relabeling`, and `scrape_series_added`. - -By default, they are disabled. - -If you want to use them for debugging and diagnostic purposes, you can activate them. To activate diagnostic metrics, define a MetricPipeline that has the `diagnosticMetrics` section defined. - -- The following example collects diagnostic metrics **only** for input `istio`: - - ```yaml - apiVersion: telemetry.kyma-project.io/v1alpha1 - kind: MetricPipeline - metadata: - name: backend - spec: - input: - istio: - enabled: true - diagnosticMetrics: - enabled: true - output: - otlp: - endpoint: - value: https://backend.example.com:4317 - ``` - -- The following example collects diagnostic metrics **only** for input `prometheus`: - - ```yaml - apiVersion: telemetry.kyma-project.io/v1alpha1 - kind: MetricPipeline - metadata: - name: backend - spec: - input: - prometheus: - enabled: true - diagnosticMetrics: - enabled: true - output: - otlp: - endpoint: - value: https://backend.example.com:4317 - ``` - -> [!NOTE] -> Diagnostic metrics are only available for inputs `prometheus` and `istio`. Learn more about the available [parameters and attributes](resources/05-metricpipeline.md). - -### 11. Deploy the Pipeline - -To activate the MetricPipeline, apply the `metricpipeline.yaml` resource file in your cluster: - -```bash -kubectl apply -f metricpipeline.yaml -``` - -### Result - -You activated a MetricPipeline and metrics start streaming to your backend. - -To check that the pipeline is running, wait until the status conditions of the MetricPipeline in your cluster have status `True`: - -```bash -kubectl get metricpipeline -NAME CONFIGURATION GENERATED GATEWAY HEALTHY AGENT HEALTHY FLOW HEALTHY -backend True True True True -``` - -## Operations - -A MetricPipeline runs several OTel Collector instances in your cluster. This Deployment serves OTLP endpoints and ships received data to the configured backend. - -The Telemetry module ensures that the OTel Collector instances are operational and healthy at any time, for example, with buffering and retries. However, there may be situations when the instances drop metrics, or cannot handle the metric load. - -To detect and fix such situations, check the [pipeline status](./resources/05-metricpipeline.md#metricpipeline-status) and check out [Troubleshooting](#troubleshooting). If you have set up [pipeline health monitoring](./04-metrics.md#5-monitor-pipeline-health), check the alerts and reports in an integrated backend like [SAP Cloud Logging](./integration/sap-cloud-logging/README.md#use-sap-cloud-logging-alerts). - -> [! WARNING] -> It's not recommended to access the metrics endpoint of the used OTel Collector instances directly, because the exposed metrics are no official API of the Kyma Telemetry module. Breaking changes can happen if the underlying OTel Collector version introduces such. -> Instead, use the [pipeline status](./resources/05-metricpipeline.md#metricpipeline-status). - -## Limitations - -- **Throughput**: Assuming an average metric with 20 metric data points and 10 labels, the default metric **gateway** setup has a maximum throughput of 34K metric data points/sec. If more data is sent to the gateway, it is refused. To increase the maximum throughput, manually scale out the gateway by increasing the number of replicas for the metric gateway. See [Module Configuration and Status](https://kyma-project.io/#/telemetry-manager/user/01-manager?id=module-configuration). - The metric **agent** setup has a maximum throughput of 14K metric data points/sec per instance. If more data must be ingested, it is refused. If a metric data endpoint emits more than 50.000 metric data points per scrape loop, the metric agent refuses all the data. -- **Load Balancing With Istio**: To ensure availability, the metric gateway runs with multiple instances. If you want to increase the maximum throughput, use manual scaling and enter a higher number of instances. - By design, the connections to the gateway are long-living connections (because OTLP is based on gRPC and HTTP/2). For optimal scaling of the gateway, the clients or applications must balance the connections across the available instances, which is automatically achieved if you use an Istio sidecar. If your application has no Istio sidecar, the data is always sent to one instance of the gateway. -- **Unavailability of Output**: For up to 5 minutes, a retry for data is attempted when the destination is unavailable. After that, data is dropped. -- **No Guaranteed Delivery**: The used buffers are volatile. If the gateway or agent instances crash, metric data can be lost. -- **Multiple MetricPipeline Support**: The maximum amount of MetricPipeline resources is 5. - -## Troubleshooting - -### No Metrics Arrive at the Backend - -**Symptom**: - -- No metrics arrive at the backend. -- In the MetricPipeline status, the `TelemetryFlowHealthy` condition has status **GatewayAllTelemetryDataDropped** or **AgentAllTelemetryDataDropped**. - -**Cause**: Incorrect backend endpoint configuration (such as using the wrong authentication credentials) or the backend is unreachable. - -**Solution**: - -1. Check the error logs for the affected Pod by calling `kubectl logs -n kyma-system {POD_NAME}` - - For **GatewayAllTelemetryDataDropped**, check pods `telemetry-metric-gateway`. - - For **AgentAllTelemetryDataDropped**, check pods `telemetry-metric-agent`. -2. Check if the backend is up and reachable. -3. Fix the errors. - -### Not All Metrics Arrive at the Backend - -**Symptom**: - -- The backend is reachable and the connection is properly configured, but some metrics are refused. -- In the MetricPipeline status, the `TelemetryFlowHealthy` condition has status **GatewaySomeTelemetryDataDropped** or **AgentSomeTelemetryDataDropped**. - -**Cause**: It can happen due to a variety of reasons - for example, the backend is limiting the ingestion rate. - -**Solution**: - -1. Check the `telemetry-metric-gateway` Pods for error logs by calling `kubectl logs -n kyma-system {POD_NAME}`. Also, check your observability backend to investigate potential causes. -2. If the backend is limiting the rate by refusing metrics, try the following options: - - Option 1: Increase maximum backend ingestion rate. For example, by scaling out the SAP Cloud Logging instances. - - Option 2: Reduce emitted metrics by re-configuring the MetricPipeline. For example, by disabling certain inputs or applying namespace filters. - - Option 3: Reduce emitted metrics in your applications. -3. Otherwise, take the actions appropriate to the cause indicated in the logs. - -### Only Istio Metrics Arrive at the Backend - -**Symptom**: Custom metrics don't arrive at the backend, but Istio metrics do. - -**Cause**: Your SDK version is incompatible with the OTel Collector version. - -**Solution**: - -1. Check which SDK version you are using for instrumentation. -2. Investigate whether it is compatible with the OTel Collector version. -3. If required, upgrade to a supported SDK version. - -### Log Entry: Failed to Scrape Prometheus Endpoint - -**Symptom**: Custom metrics don't arrive at the destination. The OTel Collector produces log entries saying "Failed to scrape Prometheus endpoint", such as the following example: - -```bash -2023-08-29T09:53:07.123Z warn internal/transaction.go:111 Failed to scrape Prometheus endpoint {"kind": "receiver", "name": "prometheus/app-pods", "data_type": "metrics", "scrape_timestamp": 1693302787120, "target_labels": "{__name__=\"up\", instance=\"10.42.0.18:8080\", job=\"app-pods\"}"} -``` - -**Cause 1**: The workload is not configured to use 'STRICT' mTLS mode. For details, see [Activate Prometheus-Based Metrics](#_4-activate-prometheus-based-metrics). - -**Solution 1**: You can either set up 'STRICT' mTLS mode or HTTP scraping: - -- Configure the workload using “STRICT” mTLS mode (for example, by applying a corresponding PeerAuthentication). -- Set up scraping through HTTP by applying the `prometheus.io/scheme=http` annotation. - -**Cause 2**: The Service definition enabling the scrape with Prometheus annotations does not reveal the application protocol to use in the port definition. For details, see [Activate Prometheus-Based Metrics](#_4-activate-prometheus-based-metrics). - -**Solution 2**: Define the application protocol in the Service port definition by either prefixing the port name with the protocol, like in `http-metrics` or define the `appProtocol` attribute. - -**Cause 3**: A deny-all `NetworkPolicy` was created in the workload namespace, which prevents that the agent can scrape metrics from annotated workloads. - -**Solution 3**: Create a separate `NetworkPolicy` to explicitly let the agent scrape your workload using the `telemetry.kyma-project.io/metric-scrape` label. - -For example, see the following `NetworkPolicy` configuration: - -```yaml -apiVersion: networking.k8s.io/v1 -kind: NetworkPolicy -metadata: - name: allow-traffic-from-agent -spec: - podSelector: - matchLabels: - app.kubernetes.io/name: "annotated-workload" # - ingress: - - from: - - namespaceSelector: - matchLabels: - kubernetes.io/metadata.name: kyma-system - podSelector: - matchLabels: - telemetry.kyma-project.io/metric-scrape: "true" - policyTypes: - - Ingress -``` - -### Gateway Throttling - -**Symptom**: In the MetricPipeline status, the `TelemetryFlowHealthy` condition has status **GatewayThrottling**. - -**Cause**: Gateway cannot receive metrics at the given rate. - -**Solution**: Manually scale out the gateway by increasing the number of replicas for the metric gateway. See [Module Configuration and Status](https://kyma-project.io/#/telemetry-manager/user/01-manager?id=module-configuration). +This content moved to [Collecting Metrics](./collecting-metrics/README.md). diff --git a/docs/user/README.md b/docs/user/README.md index 4178e51e8c..a21262ed6c 100644 --- a/docs/user/README.md +++ b/docs/user/README.md @@ -1,97 +1,82 @@ # Telemetry Module -Learn more about the Telemetry Module. Use it to enable observability for your application. +Use the Telemetry module to collect telemetry signals (logs, traces, and metrics) from your applications and send them to your preferred observability backend. ## What Is Telemetry? -Fundamentally, "Observability" is a measure of how well the application's external outputs can reflect the internal states of single components. The insights that an application and the surrounding infrastructure expose are displayed in the form of metrics, traces, and logs - collectively, that's called "telemetry" or ["signals"](https://opentelemetry.io/docs/concepts/signals/). These can be exposed by employing modern instrumentation. +With telemetry signals, you can understand the behavior and health of your applications and infrastructure. The Telemetry module provides a standardized way to collect these signals and send them to your observability backend, where you can analyze them and troubleshoot issues. + +The Telemetry module processes three types of signals: + +- Logs: Time-stamped records of events that happen over time. +- Traces: The path of a request as it travels through your application's components. +- Metrics: Aggregated numerical data about the performance or state of a component over time. ![Stages of Observability](./assets/telemetry-stages.drawio.svg) -1. In order to implement Day-2 operations for a distributed application running in a container runtime, the single components of an application must expose these signals by employing modern instrumentation. -2. Furthermore, the signals must be collected and enriched with the infrastructural metadata in order to ship them to a target system. -3. Instead of providing a one-size-for-all backend solution, the Telemetry module supports you with instrumenting and shipping your telemetry data in a vendor-neutral way. -4. This way, you can conveniently enable observability for your application by integrating it into your existing or desired backends. Pick your favorite among many observability backends (available either as a service or as a self-manageable solution) that focus on different aspects and scenarios. +Telemetry signals flow through the following stages: + +1. You instrument your application so that its components expose telemetry signals. +2. The signals are collected and enriched with infrastructural metadata. +3. You send the enriched signals to your preferred observability backend. +4. The backend stores your data, where you can analyze and visualize it. -The Telemetry module focuses exactly on the aspects of instrumentation, collection, and shipment that happen in the runtime and explicitly defocuses on backends. +The Telemetry module focuses on the collection, processing, and shipment stages of the observability workflow. It offers a vendor-neutral approach based on [OpenTelemetry](https://opentelemetry.io/) and doesn't force you into a specific backend. This means you can integrate with your existing observability platforms or choose from a wide range of available backends that best suit your operational needs. > [!TIP] -> An enterprise-grade setup demands a central solution outside the cluster, so we recommend in-cluster solutions only for testing purposes. If you want to install lightweight in-cluster backends for demo or development purposes, see [Integration Guides](#integration-guides). +> Build your first telemetry pipeline with the hands-on lesson [Collecting Application Logs and Shipping them to SAP Cloud Logging](https://learning.sap.com/learning-journeys/developing-applications-in-sap-btp-kyma-runtime/collecting-application-logs-and-shipping-to-sap-cloud-logging). ## Features To support telemetry for your applications, the Telemetry module provides the following features: -- Tooling for collection, filtering, and shipment: Based on the [Open Telemetry Collector](https://opentelemetry.io/docs/collector/), you can configure basic pipelines to filter and ship telemetry data. -- Integration in a vendor-neutral way to a vendor-specific observability system: Based on the [OpenTelemetry protocol (OTLP)](https://opentelemetry.io/docs/reference/specification/protocol/), you can integrate backend systems. -- Guidance for the instrumentation: Based on [Open Telemetry](https://opentelemetry.io/), you get community samples on how to instrument your code using the [Open Telemetry SDKs](https://opentelemetry.io/docs/instrumentation/) in nearly every programming language. -- Enriching telemetry data by automatically adding common attributes. This is done in compliance with established semantic conventions, ensuring that the enriched data adheres to industry best practices and is more meaningful for analysis. For details, see [Data Enrichment](gateways.md#data-enrichment). -- Opt-out of features for advanced scenarios: At any time, you can opt out for each data type, and use custom tooling to collect and ship the telemetry data. -- SAP BTP as first-class integration: Integration into SAP BTP Observability services, such as SAP Cloud Logging, is prioritized. For more information, see [Integrate with SAP Cloud Logging](integration/sap-cloud-logging/README.md). - -## Scope - -The Telemetry module focuses only on the signals of application logs, distributed traces, and metrics. Other kinds of signals are not considered. Also, audit logs are not in scope. +- **Consistent Telemetry Pipeline API**: Use a streamlined set of APIs based on the [OTel Collector](https://opentelemetry.io/docs/collector/) to collect, filter, and ship your logs, metrics, and traces (see [Telemetry Pipeline API](pipelines.md)). You define a pipeline for each signal type to control how the data is processed and where it's sent. For details, see [Collecting Logs](./collecting-logs/README.md), [Collecting Traces](./collecting-traces/README.md), and [Collecting Metrics](./collecting-metrics/README.md). -Supported integration scenarios are neutral to the vendor of the target system. - -## Architecture +- **Flexible Backend Integration**: The Telemetry module is optimized for integration with SAP BTP observability services, such as SAP Cloud Logging. You can also send data to any backend that supports the [OpenTelemetry protocol (OTLP)](https://opentelemetry.io/docs/specs/otel/protocol/), giving you the freedom to choose your preferred solution (see [Integrate With Your OTLP Backend](./integrate-otlp-backend/)). -![Components](./assets/telemetry-arch.drawio.svg) + > [!TIP] + > For production deployments, we recommend using a central telemetry solution located outside your cluster. For an example with SAP Cloud Logging, see [Integrate With SAP Cloud Logging](./integration/sap-cloud-logging/README.md). + > + > For testing or development, in-cluster solutions may be suitable. For examples such as Dynatrace (or to learn how to collect data from applications based on the OpenTelemetry Demo App), see [Integration Guides](https://kyma-project.io/#/telemetry-manager/user/integration/README). -### Telemetry Manager +- **Seamless Istio Integration**: The Telemetry module seamlessly integrates with the Istio module when both are present in your cluster. For details, see [Istio Integration](./architecture/istio-integration.md). -The Telemetry module ships Telemetry Manager as its core component. Telemetry Manager is a Kubernetes [operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) that implements the Kubernetes controller pattern and manages the whole lifecycle of all other components covered in the Telemetry module. Telemetry Manager watches for the user-created Kubernetes resources: LogPipeline, TracePipeline, and MetricPipeline. In these resources, you specify what data of a signal type to collect and where to ship it. -If Telemetry Manager detects a configuration, it deploys the related gateway and agent components accordingly and keeps them in sync with the requested pipeline definition. +- **Automatic Data Enrichment**: The Telemetry module adds resource attributes as metadata, following OTel semantic conventions. This makes your data more consistent, meaningful, and ready for analysis in your observability backend. For details, see [Automatic Data Enrichment](./filter-and-process/automatic-data-enrichment.md). -For more information, see [Telemetry Manager](01-manager.md). +- **Instrumentation Guidance**: To generate telemetry data, you must instrument your code. Based on [Open Telemetry](https://opentelemetry.io/) (OTel), you get community samples on how to instrument your code using the [Open Telemetry SDKs](https://opentelemetry.io/docs/languages/) in most programming languages. -### Gateways +- **Custom Tooling Support**: For advanced scenarios, you can opt out of the module's default collection and shipment mechanisms for individual data types. This enables you to use custom tooling to collect and ship the telemetry data. -The log, trace, and metrics features provide gateways based on an [OTel Collector](https://opentelemetry.io/docs/collector/) [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/). The gateways act as central endpoints in the cluster to which your applications push data in the [OTLP](https://opentelemetry.io/docs/reference/specification/protocol/) format. From here, the data is enriched and filtered, and then dispatched as configured in your pipeline resources. - -For more information, see [Telemetry Gateways](gateways.md). - -### Log Gateway and Agent - -In addition to the log gateway, you can also use the log agent based on a [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/), which collects logs of any container printing logs to `stdout/stderr`. For more information, see [Application Logs (OTLP)](logs.md). - -As an alternative to the OTLP-based log feature, you can choose using a log agent based on a [Fluent Bit](https://fluentbit.io/) installation running as a [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/). It reads all containers' logs in the runtime and ships them according to your LogPipeline configuration. For more information, see [Application Logs (Fluent Bit)](02-logs.md). - -### Trace Gateway +## Scope -The trace gateway provides an [OTLP-based](https://opentelemetry.io/docs/reference/specification/protocol/) endpoint to which applications can push the trace signals. Kyma modules like Istio or Serverless contribute traces transparently. For more information, see [Traces](03-traces.md). +The Telemetry module focuses only on the signals of application logs, distributed traces, and metrics. Other kinds of signals are not considered. Also, audit logs are not in scope. -### Metric Gateway and Agent +Supported integration scenarios are neutral to the vendor of the target system. -In addition to the metric gateway, you can also use the metric agent based on a [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/), which scrapes annotated Prometheus-based workloads. For more information, see [Metrics](04-metrics.md). +## Architecture -## Integration Guides +The Telemetry module is built around a central controller, Telemetry Manager, which dynamically configures and deploys data collection components based on your pipeline resources. -To learn about integration with SAP Cloud Logging, read [Integrate with SAP Cloud Logging](./integration/sap-cloud-logging/README.md). +To understand how the core components interact, see [Architecture](architecture/README.md). -For integration with other backends, such as Dynatrace, see: +To learn how this model applies to each signal type, see: -- [Dynatrace](./integration/dynatrace/README.md) -- [Prometheus](./integration/prometheus/README.md) -- [Loki](./integration/loki/README.md) -- [Jaeger](./integration/jaeger/README.md) -- [Amazon CloudWatch](./integration/aws-cloudwatch/README.md) +- [Logs Architecture](./architecture/logs-architecture.md) +- [Traces Architecture](./architecture/traces-architecture.md) +- [Metrics Architecture](./architecture/metrics-architecture.md) -To learn how to collect data from applications based on the OpenTelemetry SDK, see: +## API/Custom Resource Definitions -- [OpenTelemetry Demo App](./integration/opentelemetry-demo/README.md) -- [Sample App](./integration/sample-app/) +You configure the Telemetry module and its pipelines by creating and applying Kubernetes CustomResourceDefinitions (CRDs), which extend the Kubernetes API with custom additions. -## API / Custom Resource Definitions +To understand and configure the module's global settings, refer to the [Telemetry CRD](./resources/01-telemetry.md). -The API of the Telemetry module is based on Kubernetes Custom Resource Definitions (CRD), which extend the Kubernetes API with custom additions. To inspect the specification of the Telemetry module API, see: +To define how to collect, process, and ship a specific signal, use the pipeline CRDs: -- [Telemetry CRD](./resources/01-telemetry.md) - [LogPipeline CRD](./resources/02-logpipeline.md) - [TracePipeline CRD](./resources/04-tracepipeline.md) - [MetricPipeline CRD](./resources/05-metricpipeline.md) -## Resource Usage +## Resource Consumption To learn more about the resources used by the Telemetry module, see [Kyma Modules' Sizing](https://help.sap.com/docs/btp/sap-business-technology-platform/kyma-modules-sizing#telemetry). diff --git a/docs/user/_sidebar.md b/docs/user/_sidebar.md index b6e625e299..5ea3423c7f 100644 --- a/docs/user/_sidebar.md +++ b/docs/user/_sidebar.md @@ -1,12 +1,32 @@ - [Back to Kyma Home](/) - [Kyma Telemetry Module](/telemetry-manager/user/README.md) -- [Telemetry Manager](/telemetry-manager/user/01-manager.md) -- [Gateways](/telemetry-manager/user/gateways.md) -- [Application Logs (Fluent Bit)](/telemetry-manager/user/02-logs.md) -- [Application Logs (OTLP)](/telemetry-manager/user/logs.md) -- [Traces](/telemetry-manager/user/03-traces.md) -- [Metrics](/telemetry-manager/user/04-metrics.md) +- [Telemetry Pipeline API](/telemetry-manager/user/pipelines.md) +- [Set Up the OTLP Input](/telemetry-manager/user/otlp-input.md) +- [Collecting Logs](/telemetry-manager/user/collecting-logs/README.md) + - [Configure Application Logs](/telemetry-manager/user/collecting-logs/application-input.md) + - [Configure Istio Access Logs](/telemetry-manager/user/collecting-logs/istio-support.md) +- [Collecting Traces](/telemetry-manager/user/collecting-traces/README.md) + - [Configure Istio Tracing](/telemetry-manager/user/collecting-traces/istio-support.md) +- [Collecting Metrics](/telemetry-manager/user/collecting-metrics/README.md) + - [Collect Prometheus Metrics](/telemetry-manager/user/collecting-metrics/prometheus-input.md) + - [Collect Istio Metrics](/telemetry-manager/user/collecting-metrics/istio-input.md) + - [Collect Runtime Metrics](/telemetry-manager/user/collecting-metrics/runtime-input.md) +- [Filtering and Processing Data](/telemetry-manager/user/filter-and-process/README.md) + - [Filter Logs](/telemetry-manager/user/filter-and-process/filter-logs.md) + - [Filter Traces](/telemetry-manager/user/filter-and-process/filter-traces.md) + - [Filter Metrics](/telemetry-manager/user/filter-and-process/filter-metrics.md) + - [Transformation to OTLP Logs](/telemetry-manager/user/filter-and-process/transformation-to-otlp-logs.md) + - [Automatic Data Enrichment](/telemetry-manager/user/filter-and-process/automatic-data-enrichment.md) +- [Integrate with your OTLP Backend](/telemetry-manager/user/integrate-otlp-backend/README.md) + - [Migrate Your LogPipeline from HTTP to OTLP Logs](/telemetry-manager/user/integrate-otlp-backend/migration-to-otlp-logs.md) +- [Monitor Pipeline Health](/telemetry-manager/user/monitor-pipeline-health.md) +- [Troubleshooting the Telemetry Module](/telemetry-manager/user/troubleshooting.md) +- [Architecture](/telemetry-manager/user/architecture/README.md) + - [Logs Architecture](/telemetry-manager/user/architecture/logs-architecture.md) + - [Traces Architecture](/telemetry-manager/user/architecture/traces-architecture.md) + - [Metrics Architecture](/telemetry-manager/user/architecture/metrics-architecture.md) + - [Istio Integration](/telemetry-manager/user/architecture/istio-integration.md) - [Integration Guides](/telemetry-manager/user/integration/README.md) - [SAP Cloud Logging](/telemetry-manager/user/integration/sap-cloud-logging/README.md) - [Dynatrace](/telemetry-manager/user/integration/dynatrace/README.md) @@ -21,4 +41,5 @@ - [LogPipeline](/telemetry-manager/user/resources/02-logpipeline.md) - [TracePipeline](/telemetry-manager/user/resources/04-tracepipeline.md) - [MetricPipeline](/telemetry-manager/user/resources/05-metricpipeline.md) +- [Logs (Fluent Bit)](/telemetry-manager/user/02-logs.md) diff --git a/docs/user/_sidebar.ts b/docs/user/_sidebar.ts index 7eddf96739..79146ae36d 100644 --- a/docs/user/_sidebar.ts +++ b/docs/user/_sidebar.ts @@ -1,24 +1,76 @@ export default [ - { text: 'Telemetry Manager', link: './01-manager' }, - { text: 'Gateways', link: './gateways' }, - { text: 'Application Logs (Fluent Bit)', link: './02-logs' }, - { text: 'Application Logs (OTLP)', link: './logs' }, - { text: 'Traces', link: './03-traces' }, - { text: 'Metrics', link: './04-metrics' }, - { text: 'Integration Guides', link: './integration/README', collapsed: true, items: [ - { text: 'SAP Cloud Logging', link: './integration/sap-cloud-logging/README' }, - { text: 'Dynatrace', link: './integration/dynatrace/README' }, - { text: 'Prometheus', link: './integration/prometheus/README' }, - { text: 'Loki', link: './integration/loki/README' }, - { text: 'Jaeger', link: './integration/jaeger/README' }, - { text: 'Amazon CloudWatch', link: './integration/aws-cloudwatch/README' }, - { text: 'OpenTelemetry Demo App', link: './integration/opentelemetry-demo/README' }, - { text: 'Sample App', link: './integration/sample-app/README' } - ]}, - { text: 'Resources', link: './resources/README', collapsed: true, items: [ - { text: 'Telemetry', link: './resources/01-telemetry' }, - { text: 'LogPipeline', link: './resources/02-logpipeline' }, - { text: 'TracePipeline', link: './resources/04-tracepipeline' }, - { text: 'MetricPipeline', link: './resources/05-metricpipeline' } - ]} + { text: 'Telemetry Pipeline API', link: '/telemetry-manager/user/pipelines.md' }, + { text: 'Set Up the OTLP Input', link: '/telemetry-manager/user/collecting-logs/README.md' }, + { + text: 'Collecting Logs', link: '/telemetry-manager/user/collecting-logs/README.md', collapsed: true, items: [ + { text: 'Configure Application Logs', link: '/telemetry-manager/user/collecting-logs/application-input.md' }, + { text: 'Configure Istio Access Logs', link: '/telemetry-manager/user/collecting-logs/istio-support.md' }, + ] + }, + { + text: 'Collecting Traces', link: '/telemetry-manager/user/collecting-traces/README.md', collapsed: true, items: [ + { text: 'Configure Istio Tracing', link: '/telemetry-manager/user/collecting-traces/istio-support.md' }, + ] + }, + { + text: 'Collecting Metrics', link: '/telemetry-manager/user/collecting-metrics/README.md', collapsed: true, items: [ + { text: 'Collect Prometheus Metrics', link: '/telemetry-manager/user/collecting-metrics/prometheus-input.md' }, + { text: 'Collect Istio Metrics', link: '/telemetry-manager/user/collecting-metrics/istio-input.md' }, + { text: 'Collect Runtime Metrics', link: '/telemetry-manager/user/collecting-metrics/runtime-input.md' }, + ] + }, + { + text: 'Filtering and Processing Data', link: '/telemetry-manager/user/filter-and-process/README.md', collapsed: true, items: [ + { text: 'Filter Logs', link: '/telemetry-manager/user/filter-and-process/filter-logs.md' }, + { text: 'Filter Traces', link: '/telemetry-manager/user/filter-and-process/filter-traces.md' }, + { text: 'Filter Metrics', link: '/telemetry-manager/user/filter-and-process/filter-metrics.md' }, + { text: 'Transformation to OTLP Logs', link: '/telemetry-manager/user/filter-and-process/transformation-to-otlp-logs.md' }, + { text: 'Automatic Data Enrichment', link: '/telemetry-manager/user/filter-and-process/automatic-data-enrichment.md' }, + ] + }, + { + text: 'Integrate with your OTLP Backend', link: '/telemetry-manager/user/integrate-otlp-backend/README.md', collapsed: true, items: [ + { text: 'Migrate Your LogPipeline from HTTP to OTLP Logs', link: '/telemetry-manager/user/integrate-otlp-backend/migration-to-otlp-logs.md' }, + ] + }, + { text: 'Monitor Pipeline Health', link: '/telemetry-manager/user/monitor-pipeline-health.md' }, + { text: 'Troubleshooting the Telemetry Module', link: '/telemetry-manager/user/troubleshooting.md' }, + { + text: 'Architecture', + link: '/telemetry-manager/user/architecture/README.md', + collapsed: true, + items: [ + { text: 'Logs Architecture', link: '/telemetry-manager/user/architecture/logs-architecture.md' }, + { text: 'Traces Architecture', link: '/telemetry-manager/user/architecture/traces-architecture.md' }, + { text: 'Metrics Architecture', link: '/telemetry-manager/user/architecture/metrics-architecture.md' }, + { text: 'Istio Integration', link: '/telemetry-manager/user/architecture/istio-integration.md' }, + ] + }, + { + text: 'Integration Guides', + link: '/telemetry-manager/user/integration/README.md', + collapsed: true, + items: [ + { text: 'SAP Cloud Logging', link: '/telemetry-manager/user/integration/sap-cloud-logging/README.md' }, + { text: 'Dynatrace', link: '/telemetry-manager/user/integration/dynatrace/README.md' }, + { text: 'Prometheus', link: '/telemetry-manager/user/integration/prometheus/README.md' }, + { text: 'Loki', link: '/telemetry-manager/user/integration/loki/README.md' }, + { text: 'Jaeger', link: '/telemetry-manager/user/integration/jaeger/README.md' }, + { text: 'Amazon CloudWatch', link: '/telemetry-manager/user/integration/aws-cloudwatch/README.md' }, + { text: 'OpenTelemetry Demo App', link: '/telemetry-manager/user/integration/opentelemetry-demo/README.md' }, + { text: 'Sample App', link: '/telemetry-manager/user/integration/sample-app/README.md' }, + ] + }, + { + text: 'Resources', + link: '/telemetry-manager/user/resources/README.md', + collapsed: true, + items: [ + { text: 'Telemetry', link: '/telemetry-manager/user/resources/01-telemetry.md' }, + { text: 'LogPipeline', link: '/telemetry-manager/user/resources/02-logpipeline.md' }, + { text: 'TracePipeline', link: '/telemetry-manager/user/resources/04-tracepipeline.md' }, + { text: 'MetricPipeline', link: '/telemetry-manager/user/resources/05-metricpipeline.md' }, + ] + }, + { text: 'Logs (Fluent Bit)', link: '/telemetry-manager/user/02-logs.md' } ]; diff --git a/docs/user/architecture/README.md b/docs/user/architecture/README.md new file mode 100644 index 0000000000..cd63bed54d --- /dev/null +++ b/docs/user/architecture/README.md @@ -0,0 +1,55 @@ +# Architecture + +The Telemetry module consists of a manager component, which continuosly watches the user-provided pipeline resources and deploys the respective OTel Collectors. Learn more about the architecture and how the components interact. + +## Overview + +The Telemetry API provides a robust, pre-configured OpenTelemetry (OTel) Collector setup that abstracts its underlying complexities. This approach delivers several key benefits: + +- Compatibility: Maintains stability and functionality even as underlying OTel Collector features evolve, reducing the need for constant updates on your end. +- Migratability: Facilitates smooth transitions when you switch underlying technologies or architectures. +- Native Kubernetes Support: Offers seamless integration with Secrets, for example, served by the SAP BTP Service Operator, and the Telemetry Manager automatically handles the full lifecycle of all components. +- Focus: Reduces the need to understand intricate underlying OTel Collector concepts, allowing you to focus on your application development. + +![Components](./../assets/telemetry-arch.drawio.svg) + +## Telemetry Manager + +Telemetry Manager, the core component of the module, is a Kubernetes [operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) that implements the Kubernetes controller pattern and manages the whole lifecycle of all other Telemetry components. It performs the following tasks: + +1. Watch the module configuration for changes and sync the module status to it. +2. Watch the user-created Kubernetes resources LogPipeline, TracePipeline, and MetricPipeline. In these resources, you specify what data of a signal type to collect and where to ship it. +3. Manage the lifecycle of the self monitor and the user-configured agents and gateways. + For example, only if you defined a LogPipeline resource, the log gateway is deployed. + +![Manager](./../assets/manager-resources.drawio.svg) + +## Gateways and Agents + +Gateways and agents handle the incoming telemetry data. The Telemetry Manager deploys them based on your pipeline configuration. + +The gateways are based on an [OTel Collector](https://opentelemetry.io/docs/collector/) [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) and act as central endpoints in the cluster to which your applications push data in the OTLP format. From here, the data is enriched and filtered, and then dispatched configured in your pipeline resources. + +Agents run as [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) and pull data from the respective Node. + +- Log Gateway and Agent + + The log gateway provides a central OTLP endpoint for logs. You can also enable the log agent, which collects logs from the stdout/stderr output of all containers on a Node. For details, see [Logs Architecture](./logs-architecture.md). + + As an alternative to the OTLP-based log feature, you can choose using a log agent based on a [Fluent Bit](https://fluentbit.io/) installation running as a DaemonSet. It reads all containers’ logs in the runtime and ships them according to your LogPipeline configuration. For details, see [Application Logs (Fluent Bit)](./../02-logs.md). + +- Trace Gateway + +The trace gateway provides a central [OTLP](https://opentelemetry.io/docs/specs/otel/protocol/) endpoint to which your applications can push the trace signals. Kyma modules like Istio or Serverless contribute traces transparently. For details, see [Traces Architecture](./traces-architecture.md). + +- Metric Gateway and Agent + + The metric gateway provides a central OTLP endpoint for metrics. You can also enable the metric agent, which scrapes Prometheus-annotated workloads on each Node. For details, see [Metrics Architecture](./metrics-architecture.md). + +## Self Monitor + +The Telemetry module includes a [Prometheus](https://prometheus.io/)-based self-monitor that collects and evaluates health metrics from the gateways and agents. Telemetry Manager uses this data to report the current health status in your pipeline resources. + +You can also use these health metrics in your own observability backend to set up alerts and dashboards for your telemetry pipelines. For details, see [Monitor Pipeline Health](./../monitor-pipeline-health.md). + +![Self-Monitor](./../assets/manager-arch.drawio.svg) diff --git a/docs/user/architecture/istio-integration.md b/docs/user/architecture/istio-integration.md new file mode 100644 index 0000000000..2e473eeff0 --- /dev/null +++ b/docs/user/architecture/istio-integration.md @@ -0,0 +1,29 @@ +# Istio Integration + +When you have the Istio module in your cluster, the Telemetry module automatically integrates with it. It detects the Istio installation and injects sidecars into the Telemetry components, adding them to the service mesh. This enables secure mTLS communication for your Telemetry pipelines by default. + +## Receiving Data from Your Applications + +The Telemetry gateways are automatically configured to accept OTLP data from applications both inside and outside the Istio service mesh. To achieve this, the ingestion endpoints of gateways are set to Istio's permissive mode, so they accept mTLS-based communication as well as plain text. + +- Applications within the mesh automatically send data to the gateways using mTLS for a secure, encrypted connection. +- Applications outside the mesh can send data to the gateway using a standard plain text connection. + +> [!TIP] +> Learn more about Istio-specific input configuration for logs, traces, and metrics: +> +> - Configure Istio Access Logs +> - Configure Istio Tracing +> - Collect Istio Metrics + +![arch](./../assets/istio-input.drawio.svg) + +## Sending Data to In-Cluster Backends + +Telemetry gateways automatically secure the connection when sending data to your observability backends. + +If you're using an in-cluster backend that is part of the Istio mesh, the Telemetry gateways automatically use mTLS to send data to the backend securely. You don't need any special configuration for this. + +For sending data to backends outside the cluster, see [Integrate With Your OTLP Backend](./../integrate-otlp-backend/README.md). + +![arch](./../assets/istio-output.drawio.svg) diff --git a/docs/user/architecture/logs-architecture.md b/docs/user/architecture/logs-architecture.md new file mode 100644 index 0000000000..1169a5f2ce --- /dev/null +++ b/docs/user/architecture/logs-architecture.md @@ -0,0 +1,32 @@ +# Logs Architecture + +The Telemetry module provides a central Deployment of an [OTel Collector](https://opentelemetry.io/docs/collector/) acting as a gateway, and an optional DaemonSet acting as an agent. The gateway exposes endpoints that receive OTLP logs from your applications, while the agent collects container logs from each node. To control their behavior and data destination, you define a LogPipeline. + +![Architecture](./../assets/logs-arch.drawio.svg) + +1. Application containers print JSON logs to the `stdout/stderr` channel and are stored by the Kubernetes container runtime under the `var/log` directory and its subdirectories at the related Node. Istio is configured to write access logs to `stdout` as well. +2. If you choose to use the agent, an OTel Collector runs as a [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) (one instance per Node), detects any new log files in the folder, and tails and parses them. +3. An application (exposing logs in OTLP) sends logs to the central log gateway using the `telemetry-otlp-logs` service. Istio is configured to push access logs with OTLP as well. +4. The gateway and agent discover the metadata and enrich all received data with metadata of the source by communicating with the Kubernetes APIServer. Furthermore, they filter data according to the pipeline configuration. +5. Telemetry Manager configures the agent and gateway according to the LogPipeline resource specification, including the target backend. Also, it observes the logs flow to the backend and reports problems in the LogPipeline status. +8. The log agent and gateway send the data to the observability backend that's specified in your LogPipeline resource - either within your cluster, or, if authentication is set up, to an external observability backend. +9. You can analyze the logs data with your preferred backend. + +## Telemetry Manager + +The LogPipeline resource is watched by Telemetry Manager, which is responsible for generating the custom parts of the OTel Collector configuration. + +![Manager resources](./../assets/logs-resources.drawio.svg) + +1. Telemetry Manager watches all LogPipeline resources and related Secrets. +2. Furthermore, Telemetry Manager takes care of the full lifecycle of the gateway Deployment and the agent DaemonSet. Only if you defined a LogPipeline, the gateway and agent are deployed. +3. Whenever the user configuration changes, Telemetry Manager validates it and generates a single configuration for the gateway and agent. +4. Referenced Secrets are copied into one Secret that is mounted to the gateway as well. + +## Log Gateway + +In your cluster, the log gateway is the central component to which all components can send their individual logs. The gateway collects, enriches, and dispatches the data to the configured backend. For more information, see [Set Up the OTLP Input](./../otlp-input.md). + +## Log Agent + +If you configure a feature in the `input` section of your LogPipeline, an additional DaemonSet is deployed acting as an agent. The agent is based on an [OTel Collector](https://opentelemetry.io/docs/collector/) and encompasses the collection and conversion of logs from the container runtime. Hereby, the workload container just prints the structured log to the `stdout/stderr` channel. The agent picks them up, parses and enriches them, and sends all data in OTLP to the configured backend. diff --git a/docs/user/architecture/metrics-architecture.md b/docs/user/architecture/metrics-architecture.md new file mode 100644 index 0000000000..6f716d7d5c --- /dev/null +++ b/docs/user/architecture/metrics-architecture.md @@ -0,0 +1,33 @@ +# Metrics Architecture + +The Telemetry module provides a central Deployment of an [OTel Collector](https://opentelemetry.io/docs/collector/) acting as a gateway, and an optional DaemonSet acting as an agent. The gateway exposes endpoints that receive OTLP metrics from your applications, while the agent pulls metrics from Prometheus-annotated endpoints. To control their behavior and data destination, you define a MetricPipeline. + +![Architecture](./../assets/metrics-arch.drawio.svg) + +1. An application (exposing metrics in [OTLP](https://opentelemetry.io/docs/specs/otlp/)) sends metrics to the central metric gateway using the `telemetry-otlp-metrics` service. +2. An application (exposing metrics in [Prometheus](https://prometheus.io/docs/instrumenting/exposition_formats) protocol) activates the agent to scrape the metrics with an annotation-based configuration. +3. Additionally, you can activate the agent to pull metrics of each Istio sidecar. +4. The agent supports collecting metrics from the Kubelet and Kubernetes APIServer. +5. The gateway and the agent discover the metadata and enrich all received data with typical metadata of the source by communicating with the Kubernetes APIServer. Furthermore, they filter data according to the pipeline configuration. +6. Telemetry Manager configures the agent and gateway according to the MetricPipeline resource specification, including the target backend for the metric gateway. Also, it observes the metrics flow to the backend and reports problems in the MetricPipeline status. +7. The gateway and the agent send the data to the observability backend that's specified in your MetricPipeline resource - either within your cluster, or, if authentication is set up, to an external observability backend. +8. You can analyze the metric data with your preferred observability backend. + +## Telemetry Manager + +The MetricPipeline resource is watched by Telemetry Manager, which is responsible for generating the custom parts of the OTel Collector configuration. + +![Manager resources](./../assets/metrics-resources.drawio.svg) + +1. Telemetry Manager watches all MetricPipeline resources and related Secrets. +2. Furthermore, Telemetry Manager takes care of the full lifecycle of the gateway Deployment and the agent DaemonSet. Only if you defined a MetricPipeline, the gateway and agent are deployed. +3. Whenever the user configuration changes, Telemetry Manager validates it and generates a single configuration for the gateway and agent. +4. Referenced Secrets are copied into one Secret that is mounted to the gateway as well. + +## Metric Gateway + +In your cluster, the metric gateway is the central component to which all applications can send their individual metrics. The gateway collects, enriches, and dispatches the data to the configured backend. For more information, see [Set Up the OTLP Input](./../otlp-input.md). + +## Metric Agent + +If a MetricPipeline configures a feature in the `input` section, an additional DaemonSet is deployed acting as an agent. The agent is also based on an [OTel Collector](https://opentelemetry.io/docs/collector/) and encompasses the collection and conversion of Prometheus-based metrics. Hereby, the workload puts a `prometheus.io/scrape` annotation on the specification of the Pod or service, and the agent collects it. diff --git a/docs/user/architecture/traces-architecture.md b/docs/user/architecture/traces-architecture.md new file mode 100644 index 0000000000..d0062c106f --- /dev/null +++ b/docs/user/architecture/traces-architecture.md @@ -0,0 +1,28 @@ +# Traces Architecture + +The Telemetry module provides a central Deployment of an [OTel Collector](https://opentelemetry.io/docs/collector/) acting as a gateway in the cluster. The gateway exposes endpoints that receive trace data from your applications and the service mesh. To control the gateway's behavior and data destination, you define a TracePipeline. + +![Architecture](./../assets/traces-arch.drawio.svg) + +1. An end-to-end request is triggered and populated across the distributed application. Every involved component propagates the trace context using the [W3C Trace Context](https://www.w3.org/TR/trace-context/) protocol. +2. After contributing a new span to the trace, the involved components send the related span data ([OTLP](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md)) to the central trace gateway using the `telemetry-otlp-traces` service. +3. Istio sends the related span data to the trace gateway as well. +4. The trace gateway discovers metadata that's typical for sources running on Kubernetes, like Pod identifiers, and then enriches the span data with that metadata. +5. Telemetry Manager configures the gateway according to the TracePipeline resource, including the target backend for the trace gateway. Also, it observes the trace flow to the backend and reports problems in the TracePipeline status. +6. The trace gateway sends the data to the observability backend that's specified in your TracePipeline resource - either within your cluster, or, if authentication is set up, to an external observability backend. +7. You can analyze the trace data with your preferred observability backend. + +## Telemetry Manager + +The TracePipeline resource is watched by Telemetry Manager, which is responsible for generating the custom parts of the OTel Collector configuration. + +![Manager resources](./../assets/traces-resources.drawio.svg) + +1. Telemetry Manager watches all TracePipeline resources and related Secrets. +2. Furthermore, Telemetry Manager takes care of the full lifecycle of the OTel Collector Deployment itself. Only if you defined a TracePipeline, the collector is deployed. +3. Whenever the configuration changes, it validates the configuration and generates a new configuration for OTel Collector, where a ConfigMap for the configuration is generated. +4. Referenced Secrets are copied into one Secret that is mounted to the OTel Collector as well. + +## Trace Gateway + +In your cluster, the trace gateway is the central component to which all components can send their individual spans. The gateway collects, enriches, and dispatches the data to the configured backend. For more information, see [Set Up the OTLP Input](./../otlp-input.md). diff --git a/docs/user/assets/istio-input.drawio.svg b/docs/user/assets/istio-input.drawio.svg new file mode 100644 index 0000000000..f138b94fda --- /dev/null +++ b/docs/user/assets/istio-input.drawio.svg @@ -0,0 +1,325 @@ + + + + + + + + + + + + + +
+
+
+ + + Kyma Cluster +
+ + +
+
+
+ + + Kyma Cluster + + + + + + + + + + + +
+
+
+ + Kyma-System Namespace +
+ +
+
+
+ + + Kyma-System Namespace + + + + + + + + + + + +
+
+
+ + User Namespace +
+ +
+
+
+ + + User Namespace + + + + + + + + + + + +
+
+
+ + Service Mesh / Istio +
+ +
+
+
+ + + Service Mesh / Istio + + + + + + + + + + + + +
+
+
+ HTTP/gRPC +
+
+
+ + + HTTP/gRPC + + + + + + + + + + + +
+
+
+ + Application + +
+
+
+ + + Application + + + + + + + + + + + +
+
+
+ + Istio +
+ Proxy + +
+
+
+ + + Istio... + + + + + + + + + + + +
+
+
+ Gateway +
+
+
+ + + Gateway + + + + + + + + + + + +
+
+
+ + + Reference key: + + +
+
+
+ + + Reference key: + + + + + + + + + + + +
+
+
+ + User-Managed Resource + +
+
+
+ + + User-Managed Resource + + + + + + + + + + + +
+
+
+ + Application + +
+
+
+ + + Application + + + + + + + + + + + + +
+
+
+ mTLS +
+
+
+ + + mTLS + + + + + + + + + + + + + + + +
+
+
+ + Kyma-Managed Resource + +
+
+
+ + + Kyma-Managed Resource + + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/docs/user/assets/gateways-plain.drawio.svg b/docs/user/assets/istio-output.drawio.svg similarity index 60% rename from docs/user/assets/gateways-plain.drawio.svg rename to docs/user/assets/istio-output.drawio.svg index 9144727f3f..ca50e2e92a 100644 --- a/docs/user/assets/gateways-plain.drawio.svg +++ b/docs/user/assets/istio-output.drawio.svg @@ -1,17 +1,17 @@ - + - + - + -
+
@@ -24,20 +24,20 @@
- + Kyma Cluster - + -
+
@@ -48,22 +48,22 @@
- + Kyma-System Namespace - + -
+
-
+
User Namespace @@ -71,87 +71,65 @@
- + User Namespace - + -
+
-
- - User Namespace -
- -
-
-
- - - User Namespace - - - - - - - - - - - - -
-
-
- HTTP/GRPC +
+ + Service Mesh / Istio +
+
- - HTTP/GRPC + + Service Mesh / Istio - + -
+
-
+
Gateway
- + Gateway - + -
+
@@ -163,20 +141,20 @@
- + Reference key: - + -
+
@@ -186,88 +164,149 @@
- + User-Managed Resource - + -
+
-
+
- Application + Observability +
+ + Backend + +
- - Application + + Observability... - + -
+
- Backend + Observability +
+ + Backend + +
- - Backend + + Observability... - - + + -
+
-
- HTTP/GRPC +
+ HTTP/gRPC
- - HTTP/GRPC + + HTTP/gRPC - + + -
+
+
+
+ mTLS +
+
+
+ + + mTLS + + + + + + + + + + + +
+
+
+ + Istio +
+ Proxy + +
+
+
+ + + Istio... + + + + + + + + + + + + + + + +
@@ -276,86 +315,85 @@
Environment
-
-
-
-
- + External... - + -
+
- Observability System -
- (like Datadog/Splunk/...) + Observability +
+ Backend +
- - Observability System... + + Observability... - + + -
+
-
- - Kyma-Managed Resource +
+ + HTTP/gRPC
- - Kyma-Managed Resource + + HTTP/gRPC - - + -
+
-
- HTTP/GRPC +
+ + Kyma-Managed Resource +
- - HTTP/GRPC + + Kyma-Managed Resource diff --git a/docs/user/assets/logs-arch.drawio.svg b/docs/user/assets/logs-arch.drawio.svg index ccd1f1ab69..865b195cca 100644 --- a/docs/user/assets/logs-arch.drawio.svg +++ b/docs/user/assets/logs-arch.drawio.svg @@ -1,4 +1,962 @@ - - - -
Kyma Cluster
Kyma Cluster
Kyma-System Namespace
Kyma-System Namespace
User Namespace
User Namespace
Instrumentation
Instrumentation
Collection
Collection
Kyma User
Kyma User
Storage and Analysis
Storage and Analysis
External
Environment
External...
Observability System
(like Datadog/Splunk/...)
Observability System...
Observability System
(like Loki/Jaeger/...)
Observability System...
Telemetry
Manager
Telemetry...
Reference key:
Reference key:
Processes Logs
Processes Logs
Kubernetes
System
Kubernetes...
6
6
stdout/stderr
stdout/stderr
5
5
Kubernetes Node
Kubernetes Node
var/log
var/log
1
1
Application
Application
 Istio
 Proxy
Istio...
User-Managed Resource
User-Managed Resource
Kyma-Managed Resource
Kyma-Managed Resource
analyzes
logs
analyzes...
pushes logs
using OTLP
pushes logs...
3
3
2
2
7
7
tails
logs
tails...
manages
and observes
manages...
4
4
discovers
Pod
metadata
discovers...
Log Agent
(Fluent Bit)
Log Agent...
Log Gateway
(OTel Collector)
Log Gateway...
Log Agent
(Fluent Bit)
Log Agent...
Log Agent
(OTel Collector)
Log Agent...
sends logs 
using OTLP
sends logs...Text is not SVG - cannot display \ No newline at end of file + + + + + + + + + + + + + +
+
+
+ + + Kyma Cluster +
+ + +
+
+
+ + + Kyma Cluster + + + + + + + + + + + +
+
+
+ + Kyma-System Namespace +
+ +
+
+
+ + + Kyma-System Namespace + + + + + + + + + + + + + + +
+
+
+ + User Namespace +
+ +
+
+
+ + + User Namespace + + + + + + + + + + + +
+
+
+ + Instrumentation +
+ +
+
+
+ + + Instrumentation + + + + + + + + + + + +
+
+
+ + Collection and Processing +
+ +
+
+
+ + + Collection and Processing + + + + + + + + + + + + + + + + + + +
+
+
+ + + Kyma User + + +
+
+
+ + + Kyma User + + + + + + + + + + + +
+
+
+ + Storage and Analysis +
+ +
+
+
+ + + Storage and Analysis + + + + + + + + + + + +
+
+
+ Observability +
+
+ Backend +
+
+
+
+ + + Observability... + + + + + + + + + + + + +
+
+
+ analyzes +
+ logs +
+
+
+ + + analyzes... + + + + + + + + + + + +
+
+
+ + + External +
+ Environment +
+ + +
+
+
+ + + External... + + + + + + + + + + + +
+
+
+ Observability +
+ Backend +
+
+
+
+ + + Observability... + + + + + + + + + + + +
+
+
+ Telemetry +
+ Manager +
+
+
+ + + Telemetry... + + + + + + + + + + + +
+
+
+ + + Reference key: + + +
+
+
+ + + Reference key: + + + + + + + + + + + +
+
+
+ + Processes Logs + +
+
+
+ + + Processes Logs + + + + + + + + + + + +
+
+
+ Kubernetes +
+ System +
+
+
+ + + Kubernetes... + + + + + + + + + + + +
+
+
+ + 6 + +
+
+
+ + + 6 + + + + + + + + + + + +
+
+
+ stdout/stderr +
+
+
+ + + stdout/stderr + + + + + + + + + + + +
+
+
+ + 5 + +
+
+
+ + + 5 + + + + + + + + + + + +
+
+
+ + Kubernetes Node +
+ +
+
+
+ + + Kubernetes Node + + + + + + + + + + + +
+
+
+ var/log +
+
+
+ + + var/log + + + + + + + + + + + +
+
+
+ + 1 + +
+
+
+ + + 1 + + + + + + + + + + + + + + +
+
+
+ + Application + +
+
+
+ + + Application + + + + + + + + + + + +
+
+
+ + Istio +
+ Proxy + +
+
+
+ + + Istio... + + + + + + + + + + + + + + + +
+
+
+ + User-Managed Resource + +
+
+
+ + + User-Managed Resource + + + + + + + + + + + +
+
+
+ + Kyma-Managed Resource + +
+
+
+ + + Kyma-Managed Resource + + + + + + + + + + + + + + + + +
+
+
+ pushes logs +
+ using OTLP +
+
+
+ + + pushes logs... + + + + + + + + + + + +
+
+
+ + 3 + +
+
+
+ + + 3 + + + + + + + + + + + +
+
+
+ + 2 + +
+
+
+ + + 2 + + + + + + + + + + + +
+
+
+ + 7 + +
+
+
+ + + 7 + + + + + + + + + + + + +
+
+
+ + tails + +
+ + logs + +
+
+
+ + + tails... + + + + + + + + + + + + + + + + +
+
+
+ + manages +
+ and observes + +
+
+
+ + + manages... + + + + + + + + + + + +
+
+
+ + 4 + +
+
+
+ + + 4 + + + + + + + + + + + + + + + + +
+
+
+ + discovers +
+ + + Pod +
+ metadata + +
+
+
+ + + discovers... + + + + + + + + + + + + +
+
+
+ Log Agent +
+ (Fluent Bit) +
+
+
+ + + Log Agent... + + + + + + + + + + + +
+
+
+ Log Gateway +
+ (OTel Collector) +
+
+
+ + + Log Gateway... + + + + + + + + + + + + +
+
+
+ Log Agent +
+ (Fluent Bit) +
+
+
+ + + Log Agent... + + + + + + + + + + + +
+
+
+ Log Agent +
+ (OTel Collector) +
+
+
+ + + Log Agent... + + + + + + + + + + + + + + + + +
+
+
+ sends logs +
+ using OTLP +
+
+
+ + + sends logs... + + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/docs/user/assets/logs-fluentbit-arch.drawio.svg b/docs/user/assets/logs-fluentbit-arch.drawio.svg index aebe89ca3c..9e947bfcb1 100644 --- a/docs/user/assets/logs-fluentbit-arch.drawio.svg +++ b/docs/user/assets/logs-fluentbit-arch.drawio.svg @@ -1,697 +1,854 @@ - + - - - - - -
-
-
- - - Kyma Cluster + + + + + + + + + + +
+
+
+ + + Kyma Cluster +
+ + +
+
+
+ + + Kyma Cluster + + + + + + + + + + + + + + + +
+
+
+ + + Kyma User + + +
+
+
+ + + Kyma User + + + + + + + + + + + +
+
+
+ + 6 + +
+
+
+ + + 6 + + + + + + + + + + + +
+
+
+ + Kyma-System Namespace
+ +
+
+
+ + + Kyma-System Namespace + + + + + + + + + + + + + + +
+
+
+ + User Namespace +
+ +
+
+
+ + + User Namespace + + + + + + + + + + + + - - - Kyma Cluster - - - - - - - -
-
-
- - - Kyma User +
+
+
+ + + Storage and Analysis + + + + + + + + + + + + - - - Kyma User - - - - - - - -
-
-
- - 6 - -
-
-
- - - 6 - - - - - - - -
-
-
- - Kyma-System Namespace -
- -
-
-
- - - Kyma-System Namespace - - - - - - - - -
-
-
- - User Namespace -
- -
-
-
- - - User Namespace - - - - - - - -
-
-
- - Storage and Analysis +
+
+
+ + + External... + + + + + + + + + + + +
+
+
+ Observability System +
+ (like Datadog/Splunk/...) +
+
+
+ + + Observability System... + + + + + + + + + + + +
+
+
+ Observability System
- -
-
-
- - - Storage and Analysis - - - - - - - -
-
-
- + (like Loki/Jaeger/...) +
+
+
+ + + Observability System... + + + + + + + + + + + +
+
+
+ + Instrumentation +
+ +
+
+
+ + + Instrumentation + + + + + + + + + + + +
+
+
+ + Collection +
+ +
+
+
+ + + Collection + + + + + + + + + + + +
+
+
+ Telemetry +
+ Manager +
+
+
+ + + Telemetry... + + + + + + + + + + + +
+
+
- External -
- Environment -
+ + Reference key: + - -
-
-
- - - External... - - - - - - - -
-
-
- Observability System -
- (like Datadog/Splunk/...) -
-
-
- - - Observability System... - - - - - - - -
-
-
- Observability System -
- (like Loki/Jaeger/...) -
-
-
- - - Observability System... - - - - - - - -
-
-
- - Instrumentation +
+
+
+ + + Reference key: + + + + + + + + + + + +
+
+
+ + Processes Logs + +
+
+
+ + + Processes Logs + + + + + + + + + + + +
+
+
+ Kubernetes
- -
-
-
- - - Instrumentation - - - - - - - -
-
-
- - Collection + System +
+
+
+ + + Kubernetes... + + + + + + + + + + + +
+
+
+ + 5 + +
+
+
+ + + 5 + + + + + + + + + + + +
+
+
+ Log Agent +
+ (Fluent Bit) +
+
+
+ + + Log Agent... + + + + + + + + + + + + + + + +
+
+
+ stdout/stderr +
+
+
+ + + stdout/stderr + + + + + + + + + + + +
+
+
+ + 2 + +
+
+
+ + + 2 + + + + + + + + + + + + +
+
+
+ + tails + +
+ + logs + +
+
+
+ + + tails... + + + + + + + + + + + +
+
+
+ Log Agent
- -
-
-
- - - Collection - - - - - - - -
-
-
- Telemetry -
- Manager -
-
-
- - - Telemetry... - - - - - - - -
-
-
- - - Reference key: - - -
-
-
- - - Reference key: - - - - - - - -
-
-
- - Processes Logs - -
-
-
- - - Processes Logs - - - - - - - -
-
-
- Kubernetes -
- System -
-
-
- - - Kubernetes... - - - - - - - -
-
-
- - 5 - -
-
-
- - - 5 - - - - - - - -
-
-
- Log Agent -
- (Fluent Bit) -
-
-
- - - Log Agent... - - - - - - - - -
-
-
- stdout/stderr -
-
-
- - - stdout/stderr - - - - - - - -
-
-
- - 2 - -
-
-
- - - 2 - - - - - - - - -
-
-
- - tails - -
- - logs - -
-
-
- - - tails... - - - - - - - -
-
-
- Log Agent -
- (Fluent Bit) -
-
-
- - - Log Agent... - - - - - - - - -
-
-
- - discovers + (Fluent Bit) +
+
+
+ + + Log Agent... + + + + + + + + + + + + +
+
+
+ + discovers +
+ Pod metadata + +
+
+
+ + + discovers... + + + + + + + + + + + +
+
+
+ + 3 + +
+
+
+ + + 3 + + + + + + + + + + + +
+
+
+ + 4 + +
+
+
+ + + 4 + + + + + + + + + + + + +
+
+
+ sends logs
- Pod metadata - -
-
-
- - - discovers... - - - - - - - -
-
-
- - 3 - -
-
-
- - - 3 - - - - - - - -
-
-
- - 4 - -
-
-
- - - 4 - - - - - - - - -
-
-
- sends logs -
- using HTTP -
-
-
- - - sends logs... - - - - - - - - - -
-
-
- - Kubernetes Node -
- -
-
-
- - - Kubernetes Node - - - - - - - -
-
-
- var/log -
-
-
- - - var/log - - - - - - - -
-
-
- - 1 - -
-
-
- - - 1 - - - - - - - - -
-
-
- - manages + using HTTP +
+
+
+ + + sends logs... + + + + + + + + + + + + + + + +
+
+
+ + Kubernetes Node +
+ +
+
+
+ + + Kubernetes Node + + + + + + + + + + + +
+
+
+ var/log +
+
+
+ + + var/log + + + + + + + + + + + +
+
+
+ + 1 + +
+
+
+ + + 1 + + + + + + + + + + + + +
+
+
+ + manages +
+ and observes + +
+
+
+ + + manages... + + + + + + + + + + + + + + +
+
+
+ + Application + +
+
+
+ + + Application + + + + + + + + + + + +
+
+
+ + Istio +
+ Proxy + +
+
+
+ + + Istio... + + + + + + + + + + + + + + + + + + + +
+
+
+ + User-Managed Resource + +
+
+
+ + + User-Managed Resource + + + + + + + + + + + +
+
+
+ + Kyma-Managed Resource + +
+
+
+ + + Kyma-Managed Resource + + + + + + + + + + + + + + + + +
+
+
+ analyzes
- and observes - -
-
-
- - - manages... - - - - - - - - -
-
-
- - Application - -
-
-
- - - Application - - - - - - - -
-
-
- - Istio -
- Proxy - -
-
-
- - - Istio... - - - - - - - - - - - -
-
-
- - User-Managed Resource - -
-
-
- - - User-Managed Resource - - - - - - - -
-
-
- - Kyma-Managed Resource - -
-
-
- - - Kyma-Managed Resource - - - - - - - - - - -
-
-
- analyzes -
- logs -
-
-
- - - analyzes... - - + logs +
+
+
+ + + External +
+ Environment +
+ - -
-
-
+
+
+ + Storage and Analysis +
- -
-
-
+
+
+ + + analyzes... + + + - + Text is not SVG - cannot display diff --git a/docs/user/assets/logs-fluentbit-flow.drawio.svg b/docs/user/assets/logs-fluentbit-flow.drawio.svg index 10be0dc6c0..f89e3f6653 100644 --- a/docs/user/assets/logs-fluentbit-flow.drawio.svg +++ b/docs/user/assets/logs-fluentbit-flow.drawio.svg @@ -1,4 +1,160 @@ - - - -
Kubernetes Filter

Name                           kubernetes
Match                           pipeline-1.*
Merge_Log                   On
Kubernetes Filter...
Tail Input

Name                  tail
Path                    /var/log/containers/*.log
Tag                      pipeline-1.*
Multiline.parser   cri

Tail Input...
Container Log Message


/var/log/containers/*.log



Container Log Message...Text is not SVG - cannot display \ No newline at end of file + + + + + + + + + + + + + +
+
+
+ + + + Kubernetes Filter + + +
+
+
+ + Name                           kubernetes + +
+
+ + Match                           pipeline-1.* + +
+
+ + Merge_Log                   On + +
+ +
+
+
+ + + Kubernetes Filter... + + + + + + + + + + + +
+
+
+ + + + Tail Input +
+ + +
+
+ + Name                  tail + +
+
+ + Path + + + /var/log/containers/*.log + +
+
+ + Tag                      pipeline-1.* + +
+
+ + Multiline.parser   cri + +
+
+ +
+ +
+ +
+
+
+ + + Tail Input... + + + + + + + + + + + + + + +
+
+
+ + + + Container Log Message +
+ + +
+
+ + /var/log/containers/*.log +
+ +
+
+
+ +
+
+
+ + + Container Log Message... + + + + + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/docs/user/assets/logs-fluentbit-pipelines.drawio.svg b/docs/user/assets/logs-fluentbit-pipelines.drawio.svg index 8b5fd66970..b60da440f6 100644 --- a/docs/user/assets/logs-fluentbit-pipelines.drawio.svg +++ b/docs/user/assets/logs-fluentbit-pipelines.drawio.svg @@ -1,4 +1,474 @@ - - - -
Filter A
Filter A
Filter B
Filter B
Output A
Output A
Output B
Output B
Log Pipeline "B"
Log Pipeline "B"
Log Pipeline "A"
Log Pipeline "A"
Name     tail
Path       /var/log/[...]/*.log
Tag         pipeline-a.[...]
Parser    json 
Name     tail...
Name           kubernetes
Match           pipeline-a.*
Merge_Log  On
Name           kuber...
Input
Input
Outputs
Outputs
Filters
Filters
Filter A
Filter A
Filter B
Filter B
Name     tail
Path       /var/log/[...]/*.log
Tag         pipeline-b.[...]
Parser    json 
Name     tail...
Name           kubernetes
Match           pipeline-b.*
Merge_Log  On
Name           kuber...Text is not SVG - cannot display \ No newline at end of file + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+
+ + Filter A + +
+
+
+
+ + + Filter A + + + + + + + + + + + + + + + +
+
+
+
+ + Filter B + +
+
+
+
+ + + Filter B + + + + + + + + + + + +
+
+
+
+ + Output A + +
+
+
+
+ + + Output A + + + + + + + + + + + +
+
+
+
+ + Output B + +
+
+
+
+ + + Output B + + + + + + + + + + + +
+
+
+ Log Pipeline "B" +
+
+
+ + + Log Pipeline "B" + + + + + + + + + + + +
+
+
+ Log Pipeline "A" +
+
+
+ + + Log Pipeline "A" + + + + + + + + + + + + + + + +
+
+
+
+ + Name     tail + +
+
+ + Path       /var/log/[...]/*.log + +
+
+ + Tag         pipeline-a.[...] + +
+
+ + Parser    json + +
+
+
+
+ + + Name     tail... + + + + + + + + + + + +
+
+
+
+ + Name           kubernetes + +
+
+ + Match           pipeline-a.* + +
+
+ + Merge_Log  On + +
+
+
+
+ + + Name           kuber... + + + + + + + + + + + +
+
+
+ + + Input + + +
+
+
+ + + Input + + + + + + + + + + + +
+
+
+ + + Outputs + + +
+
+
+ + + Outputs + + + + + + + + + + + +
+
+
+ + + Filters + + +
+
+
+ + + Filters + + + + + + + + + + + +
+
+
+
+ + Filter A + +
+
+
+
+ + + Filter A + + + + + + + + + + + +
+
+
+
+ + Filter B + +
+
+
+
+ + + Filter B + + + + + + + + + + + +
+
+
+
+ + Name     tail + +
+
+ + Path       /var/log/[...]/*.log + +
+
+ + Tag         pipeline-b.[...] + +
+
+ + Parser    json + +
+
+
+
+ + + Name     tail... + + + + + + + + + + + +
+
+
+
+ + Name           kubernetes + +
+
+ + Match           pipeline-b.* + +
+
+ + Merge_Log  On + +
+
+
+
+ + + Name           kuber... + + + + + + + + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/docs/user/assets/logs-fluentbit-resources.drawio.svg b/docs/user/assets/logs-fluentbit-resources.drawio.svg index 2040116308..c6917f8463 100644 --- a/docs/user/assets/logs-fluentbit-resources.drawio.svg +++ b/docs/user/assets/logs-fluentbit-resources.drawio.svg @@ -1,4 +1,661 @@ - - - -
Kyma Cluster
Kyma Cluster
Reference key:
Reference key:
Processes Logs
Processes Logs
User-Managed Resource
User-Managed Resource
Kyma-System Namespace
Kyma-System Namespace
<<Deployment>>
telemetry-manager
<<Deployment>>...
watches
watches
<<LogPipeline>>
custom
<<LogPipeline>>...
User Namespace
User Namespace
<<Secret>>
custom
<<Secret>>...
<<Secret>>
custom
<<Secret>>...
references
references
<<DaemonSet>>
telemetry-fluent-bit
<<DaemonSet>>...
<<ConfigMap>>
  - telemetry-fluent-bit
  - telemetry-fluent-bit-sections

<<ConfigMap>>...
<<Secret>>
- telemetry-fluent-bit-env
- telemetry-fluent-bit-tls-config

<<Secret>>...
mounts
as volume
mounts...
references
as env
references...
manages
lifecycle
manages...
3
3
2
2
generates
configuration
generates...
copies user Secrets
into one gateway Secret
copies user Secrets...
4
4
1
1
Kyma-Managed Resource
Kyma-Managed ResourceText is not SVG - cannot display \ No newline at end of file + + + + + + + + + + + + + +
+
+
+ + + Kyma Cluster +
+ + +
+
+
+ + + Kyma Cluster + + + + + + + + + + + +
+
+
+ + + Reference key: + + +
+
+
+ + + Reference key: + + + + + + + + + + + +
+
+
+ + Processes Logs + +
+
+
+ + + Processes Logs + + + + + + + + + + + +
+
+
+ + User-Managed Resource + +
+
+
+ + + User-Managed Resource + + + + + + + + + + + +
+
+
+ + Kyma-System Namespace +
+ +
+
+
+ + + Kyma-System Namespace + + + + + + + + + + + +
+
+
+ <<Deployment>> +
+ telemetry-manager +
+
+
+ + + <<Deployment>>... + + + + + + + + + + + + + + + +
+
+
+ watches +
+
+
+ + + watches + + + + + + + + + + + +
+
+
+ <<LogPipeline>> +
+ custom +
+
+
+ + + <<LogPipeline>>... + + + + + + + + + + + +
+
+
+ + User Namespace +
+ +
+
+
+ + + User Namespace + + + + + + + + + + + +
+
+
+ <<Secret>> +
+ custom +
+
+
+ + + <<Secret>>... + + + + + + + + + + + +
+
+
+ <<Secret>> +
+ custom +
+
+
+ + + <<Secret>>... + + + + + + + + + + + + + + + + +
+
+
+ + references + +
+
+
+ + + references + + + + + + + + + + + +
+
+
+ <<DaemonSet>> +
+ telemetry-fluent-bit +
+
+
+ + + <<DaemonSet>>... + + + + + + + + + + + +
+
+
+
+ +
+ + <<ConfigMap>> + +
+
+
+ + - telemetry-fluent-bit + +
+ +
+ + - telemetry-fluent-bit-sections +
+
+ +
+ +
+ +
+
+
+ + + <<ConfigMap>>... + + + + + + + + + + + +
+
+
+ +
+ + + <<Secret>> +
+
+ + - telemetry-fluent-bit-env + +
+ + + +
+ + - telemetry-fluent-bit-tls-config + +
+ +
+
+ +
+
+
+ + + <<Secret>>... + + + + + + + + + + + + +
+
+
+ + mounts +
+ as volume + +
+
+
+ + + mounts... + + + + + + + + + + + + +
+
+
+ + references +
+ + as env +
+
+
+ + + references... + + + + + + + + + + + + +
+
+
+ manages +
+ lifecycle +
+
+
+ + + manages... + + + + + + + + + + + +
+
+
+ + 3 + +
+
+
+ + + 3 + + + + + + + + + + + +
+
+
+ + 2 + +
+
+
+ + + 2 + + + + + + + + + + + + +
+
+
+ + generates + +
+ + configuration + +
+
+
+ + + generates... + + + + + + + + + + + + +
+
+
+ + copies user Secrets +
+ into one gateway Secret + +
+
+
+ + + copies user Secrets... + + + + + + + + + + + +
+
+
+ + 4 + +
+
+
+ + + 4 + + + + + + + + + + + +
+
+
+ + 1 + +
+
+
+ + + 1 + + + + + + + + + + + +
+
+
+ + Kyma-Managed Resource + +
+
+
+ + + Kyma-Managed Resource + + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/docs/user/assets/logs-resources.drawio.svg b/docs/user/assets/logs-resources.drawio.svg index d1dfac46cf..696ef52288 100644 --- a/docs/user/assets/logs-resources.drawio.svg +++ b/docs/user/assets/logs-resources.drawio.svg @@ -1,4 +1,4 @@ - + @@ -12,9 +12,9 @@
-
-
- +
+
+ Kyma Cluster
@@ -24,14 +24,14 @@
- + Kyma Cluster - + @@ -80,7 +80,7 @@ - + @@ -112,7 +112,7 @@
-
+
manages
lifecycle @@ -120,7 +120,7 @@
- + manages... @@ -139,13 +139,13 @@
-
+
watches
- + watches @@ -199,7 +199,7 @@ - + @@ -222,7 +222,7 @@ - + @@ -258,7 +258,7 @@
-
+
references @@ -266,7 +266,7 @@
- + references @@ -296,7 +296,7 @@ - + @@ -328,7 +328,7 @@ - + @@ -373,7 +373,7 @@
-
+
mounts
@@ -383,7 +383,7 @@
- + mounts... @@ -399,7 +399,7 @@
-
+
mounts as volume @@ -407,7 +407,7 @@
- + mounts as volume @@ -446,13 +446,13 @@
-
+
references as env
- + references as env @@ -482,7 +482,7 @@ - + @@ -577,7 +577,7 @@
-
+
generates @@ -589,7 +589,7 @@
- + generates... @@ -605,7 +605,7 @@
-
+
copies user Secrets
@@ -615,7 +615,7 @@
- + copies user Secrets... diff --git a/docs/user/assets/manager-arch.drawio.svg b/docs/user/assets/manager-arch.drawio.svg index 0662157f79..cda33905d7 100644 --- a/docs/user/assets/manager-arch.drawio.svg +++ b/docs/user/assets/manager-arch.drawio.svg @@ -1,4 +1,424 @@ - - - -
Kyma Cluster
Kyma Cluster

Kyma-System Namespace
Kyma-System Namespace...
Telemetry
Manager
Telemetry...
Log Agent
(Fluent Bit)
Log Agent...
Trace Gateway
(Otel Collector)
Trace Gateway...
Metric Gateway/Agent
(Otel Collector)
Metric Gateway/Age...
Self Monitor
Self Monitor
scrapes
health metrics
scrapes...
gets
health
status
gets...
Reference key:
Reference key:
Trace Gateway
(OTel Collector)
Trace Gateway...
Log Agent
(Fluent Bit)
Log Agent...
Metric Gateway/Agent
(OTel Collector)
Metric Gateway/Ag...
Processes Logs
Processes Logs
Processes Traces
Processes Traces
Processes Metrics
Processes Metrics
Kyma-Managed Resource
Kyma-Managed ResourceText is not SVG - cannot display \ No newline at end of file + + + + + + + + + + + + + +
+
+
+ + + Kyma Cluster +
+ + +
+
+
+ + + Kyma Cluster + + + + + + + + + + + +
+
+
+ +
+ Kyma-System Namespace +
+ +
+
+
+ + + Kyma-System Namespace... + + + + + + + + + + + +
+
+
+ Telemetry +
+ Manager +
+
+
+ + + Telemetry... + + + + + + + + + + + +
+
+
+ Log Agent +
+ (Fluent Bit) +
+
+
+ + + Log Agent... + + + + + + + + + + + +
+
+
+ Trace Gateway +
+ (Otel Collector) +
+
+
+ + + Trace Gateway... + + + + + + + + + + + +
+
+
+ Metric Gateway/Agent +
+ (Otel Collector) +
+
+
+ + + Metric Gateway/Age... + + + + + + + + + + + +
+
+
+ Self Monitor +
+
+
+ + + Self Monitor + + + + + + + + + + + + +
+
+
+ scrapes +
+ health metrics +
+
+
+ + + scrapes... + + + + + + + + + + + + + + + + + + + + +
+
+
+ gets +
+ health +
+ status +
+
+
+ + + gets... + + + + + + + + + + + +
+
+
+ + + Reference key: + + +
+
+
+ + + Reference key: + + + + + + + + + + + +
+
+
+ Trace Gateway +
+ (OTel Collector) +
+
+
+ + + Trace Gateway... + + + + + + + + + + + +
+
+
+ Log Gateway/Agent +
+ (OTel Collector) +
+
+
+ + + Log Gateway/Agent... + + + + + + + + + + + +
+
+
+ Metric Gateway/Agent +
+ (OTel Collector) +
+
+
+ + + Metric Gateway/Ag... + + + + + + + + + + + +
+
+
+ + Processes Logs + +
+
+
+ + + Processes Logs + + + + + + + + + + + +
+
+
+ + Processes Traces + +
+
+
+ + + Processes Traces + + + + + + + + + + + +
+
+
+ + Processes Metrics + +
+
+
+ + + Processes Metrics + + + + + + + + + + + +
+
+
+ + Kyma-Managed Resource + +
+
+
+ + + Kyma-Managed Resource + + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/docs/user/assets/manager-resources.drawio.svg b/docs/user/assets/manager-resources.drawio.svg index ac28bf5c9e..0501fe9a51 100644 --- a/docs/user/assets/manager-resources.drawio.svg +++ b/docs/user/assets/manager-resources.drawio.svg @@ -1,4 +1,685 @@ - - - -
Kyma Cluster
Kyma Cluster
Reference key:
Reference key:
Processes Logs
Processes Logs
Processes Traces
Processes Traces
Processes Metrics
Processes Metrics
Kyma-System Namespace
Kyma-System Namespace
manages lifecycle
manages lifecycle
<<Deployment>>
telemetry-manager
<<Deployment>>...
<<MetricPipeline>>
custom
<<MetricPipeline>>...
<<TracePipeline>>
custom
<<TracePipeline>>...
<<Telemetry>>
default
<<Telemetry>>...
reads module
configuration 
and syncs status
reads module...
<<LogPipeline>>
custom
<<LogPipeline>>...
watches
watches
<<Deployment>>
telemetry-trace-gateway
<<Deployment>>...
<<DaemonSet>>
telemetry-fluent-bit
<<DaemonSet>>...
<<Service>>
telemetry-otlp-traces
<<Service>>...
<<Deployment>>
telemetry-self-monitor
<<Deployment>>...
<<Deployment>>
telemetry-metric-gateway
<<Deployment>>...
<<DaemonSet>>
telemetry-metric-agent
<<DaemonSet>>...
<<Service>>
telemetry-otlp-metrics
<<Service>>...
2
2
1
1
3
3
User-Managed Resource
User-Managed Resource
Kyma-Managed Resource
Kyma-Managed ResourceText is not SVG - cannot display \ No newline at end of file + + + + + + + + + + + + + +
+
+
+ + + Kyma Cluster +
+ + +
+
+
+ + + Kyma Cluster + + + + + + + + + + + +
+
+
+ + + Reference key: + + +
+
+
+ + + Reference key: + + + + + + + + + + + +
+
+
+ + Processes Logs + +
+
+
+ + + Processes Logs + + + + + + + + + + + +
+
+
+ + Processes Traces + +
+
+
+ + + Processes Traces + + + + + + + + + + + +
+
+
+ + Processes Metrics + +
+
+
+ + + Processes Metrics + + + + + + + + + + + +
+
+
+ + Kyma-System Namespace +
+ +
+
+
+ + + Kyma-System Namespace + + + + + + + + + + + + +
+
+
+ + manages + + + lifecycle + +
+
+
+ + + manages lifecycle + + + + + + + + + + + + + + + + + + + +
+
+
+ <<Deployment>> +
+ telemetry-manager +
+
+
+ + + <<Deployment>>... + + + + + + + + + + + +
+
+
+ <<MetricPipeline>> +
+ custom +
+
+
+ + + <<MetricPipeline>>... + + + + + + + + + + + +
+
+
+ <<TracePipeline>> +
+ custom +
+
+
+ + + <<TracePipeline>>... + + + + + + + + + + + +
+
+
+ <<Telemetry>> +
+ default +
+
+
+ + + <<Telemetry>>... + + + + + + + + + + + + +
+
+
+ reads module +
+ configuration +
+ and syncs status +
+
+
+ + + reads module... + + + + + + + + + + + +
+
+
+ <<LogPipeline>> +
+ custom +
+
+
+ + + <<LogPipeline>>... + + + + + + + + + + + + +
+
+
+ watches +
+
+
+ + + watches + + + + + + + + + + + + + + +
+
+
+ + <<Deployment>> +
+ telemetry-trace-gateway + +
+
+
+ + + <<Deployment>>... + + + + + + + + + + + +
+
+
+ <<Service>> +
+ telemetry-otlp-traces +
+
+
+ + + <<Service>>... + + + + + + + + + + + +
+
+
+ <<Deployment>> +
+ telemetry-self-monitor +
+
+
+ + + <<Deployment>>... + + + + + + + + + + + +
+
+
+ <<Deployment>> +
+ telemetry-metric-gateway +
+
+
+ + + <<Deployment>>... + + + + + + + + + + + +
+
+
+ <<DaemonSet>> +
+ telemetry-metric-agent +
+
+
+ + + <<DaemonSet>>... + + + + + + + + + + + +
+
+
+ <<Service>> +
+ telemetry-otlp-metrics +
+
+
+ + + <<Service>>... + + + + + + + + + + + +
+
+
+ + 2 + +
+
+
+ + + 2 + + + + + + + + + + + +
+
+
+ + 1 + +
+
+
+ + + 1 + + + + + + + + + + + +
+
+
+ + 3 + +
+
+
+ + + 3 + + + + + + + + + + + +
+
+
+ + User-Managed Resource + +
+
+
+ + + User-Managed Resource + + + + + + + + + + + +
+
+
+ + Kyma-Managed Resource + +
+
+
+ + + Kyma-Managed Resource + + + + + + + + + + + +
+
+
+ <<Deployment>> +
+ telemetry-log-gateway +
+
+
+ + + <<Deployment>>... + + + + + + + + + + + +
+
+
+ <<DaemonSet>> +
+ telemetry-log-agent +
+
+
+ + + <<DaemonSet>>... + + + + + + + + + + + +
+
+
+ <<Service>> +
+ telemetry-otlp-log +
+
+
+ + + <<Service>>... + + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/docs/user/assets/metrics-arch.drawio.svg b/docs/user/assets/metrics-arch.drawio.svg index 1683b44a83..03fb6f83aa 100644 --- a/docs/user/assets/metrics-arch.drawio.svg +++ b/docs/user/assets/metrics-arch.drawio.svg @@ -1,4 +1,4 @@ -
Kyma Cluster
Kyma Cluster
Kyma-System Namespace
Kyma-System Namespace
User Namespace
User Namespace
Storage and Analysis
Storage and Analysis
External
Environment
External...
Observability System
(like Datadog/Splunk/...)
Observability System...
Observability System
(like Loki/Jaeger/...)
Observability System...
Instrumentation
Instrumentation
Kubernetes
System
Kubernetes...
Collection
Collection
Reference key:
Reference key:
Processes Metrics
Processes Metrics
Kyma User
Kyma User
analyzes
metrics
analyzes...
9
9
8
8
Telemetry
Manager
Telemetry...
7
7
1
1
sends metrics
using OTLP
sends metrics...
5
5
2
2

discovers
Pod metadata
discovers...

collects metrics
collects metrics
3
3
4
4
6
6
manages
and observes
manages...
Application
Application
 Istio
 Proxy
Istio...
Application
Application
 Istio
 Proxy
Istio...sends metricsusing OTLP
collects metrics
in Prometheus format
collects metrics...

User-Managed Resource
User-Managed Resource
Kyma-Managed Resource
Kyma-Managed Resource
Metric Agent
Metric Agent
Metric Gateway
(OTel Collector)
Metric Gateway...
sends metrics
using OTLP
sends metrics...Text is not SVG - cannot display \ No newline at end of file +
Kyma Cluster
Kyma Cluster
Kyma-System Namespace
Kyma-System Namespace
User Namespace
User Namespace
Storage and Analysis
Storage and Analysis
External
Environment
External...
Observability 
Backend
Observability...
Observability
Backend
Observability...
Instrumentation
Instrumentation
Kubernetes
System
Kubernetes...
Collection and Processing
Collection and Processing
Reference key:
Reference key:
Processes Metrics
Processes Metrics
Kyma User
Kyma User
analyzes
metrics
analyzes...
8
8
7
7
Telemetry
Manager
Telemetry...
6
6
1
1
2
2
3
3
4
4
6
6
Application
Application
 Istio
 Proxy
Istio...
Application
Application
 Istio
 Proxy
Istio...
collects metrics
in Prometheus format
collects metrics...
User-Managed Resource
User-Managed Resource
Kyma-Managed Resource
Kyma-Managed Resource

discovers
Pod metadata
discovers...

collects metrics
collects metrics
manages
and observes
manages...

Metric Agent
Metric Agent
Metric Gateway
(OTel Collector)
Metric Gateway...

sends metrics
using OTLP
sends metrics...sends metricsusing OTLP
5
5Text is not SVG - cannot display \ No newline at end of file diff --git a/docs/user/assets/metrics-resources.drawio.svg b/docs/user/assets/metrics-resources.drawio.svg index fd424c28dc..95274afebe 100644 --- a/docs/user/assets/metrics-resources.drawio.svg +++ b/docs/user/assets/metrics-resources.drawio.svg @@ -1,4 +1,4 @@ - + @@ -12,9 +12,9 @@
-
-
- +
+
+ Kyma Cluster
@@ -24,14 +24,14 @@
- + Kyma Cluster - + @@ -39,7 +39,7 @@
-
+
Reference key: @@ -49,7 +49,7 @@
- + Reference key: @@ -80,7 +80,7 @@ - + @@ -112,7 +112,7 @@
-
+
manages
lifecycle @@ -120,7 +120,7 @@
- + manages... @@ -139,13 +139,13 @@
-
+
watches
- + watches @@ -199,7 +199,7 @@ - + @@ -222,7 +222,7 @@ - + @@ -258,7 +258,7 @@
-
+
references @@ -266,7 +266,7 @@
- + references @@ -319,7 +319,7 @@ - + @@ -351,7 +351,7 @@ - + @@ -396,7 +396,7 @@
-
+
mounts
@@ -406,7 +406,7 @@
- + mounts... @@ -422,7 +422,7 @@
-
+
mounts as volume @@ -430,7 +430,7 @@
- + mounts as volume @@ -469,13 +469,13 @@
-
+
references as env
- + references as env @@ -505,7 +505,7 @@ - + @@ -600,7 +600,7 @@
-
+
generates @@ -612,7 +612,7 @@
- + generates... @@ -628,7 +628,7 @@
-
+
copies user Secrets
@@ -638,7 +638,7 @@
- + copies user Secrets... diff --git a/docs/user/assets/otlp-input.drawio.svg b/docs/user/assets/otlp-input.drawio.svg new file mode 100644 index 0000000000..4f1e0f9c2f --- /dev/null +++ b/docs/user/assets/otlp-input.drawio.svg @@ -0,0 +1,227 @@ + + + + + + + + + + + + + +
+
+
+ + + Kyma Cluster +
+ + +
+
+
+ + + Kyma Cluster + + + + + + + + + + + +
+
+
+ + + Reference key: + + +
+
+
+ + + Reference key: + + + + + + + + + + + +
+
+
+ + User-Managed Resource + +
+
+
+ + + User-Managed Resource + + + + + + + + + + + +
+
+
+ + Kyma-Managed Resource + +
+
+
+ + + Kyma-Managed Resource + + + + + + + + + + + +
+
+
+ + Kyma-System Namespace +
+ +
+
+
+ + + Kyma-System Namespace + + + + + + + + + + + +
+
+
+ Gateway +
+
+
+ + + Gateway + + + + + + + + + + + +
+
+
+ + User Namespace +
+ +
+
+
+ + + User Namespace + + + + + + + + + + + +
+
+
+ + Application + +
+
+
+ + + Application + + + + + + + + + + + + +
+
+
+ HTTP / gRPC +
+
+
+ + + HTTP / gRPC + + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/docs/user/assets/otlp-output.drawio.svg b/docs/user/assets/otlp-output.drawio.svg new file mode 100644 index 0000000000..826caefd80 --- /dev/null +++ b/docs/user/assets/otlp-output.drawio.svg @@ -0,0 +1,301 @@ + + + + + + + + + + + + + +
+
+
+ + + Kyma Cluster +
+ + +
+
+
+ + + Kyma Cluster + + + + + + + + + + + +
+
+
+ + Kyma-System Namespace +
+ +
+
+
+ + + Kyma-System Namespace + + + + + + + + + + + +
+
+
+ + User Namespace + +
+
+
+ + + User Namespace + + + + + + + + + + + +
+
+
+ Gateway +
+
+
+ + + Gateway + + + + + + + + + + + +
+
+
+ + + Reference key: + + +
+
+
+ + + Reference key: + + + + + + + + + + + +
+
+
+ + User-Managed Resource + +
+
+
+ + + User-Managed Resource + + + + + + + + + + + +
+
+
+ + Observability Backend + +
+
+
+ + + Observability Back... + + + + + + + + + + + + +
+
+
+ HTTP / gRPC +
+
+
+ + + HTTP / gRPC + + + + + + + + + + + +
+
+
+ + + External +
+ Environment +
+
+
+
+
+ + +
+
+
+ + + External... + + + + + + + + + + + +
+
+
+ Observability Backend +
+
+
+ + + Observability Back... + + + + + + + + + + + +
+
+
+ + Kyma-Managed Resource + +
+
+
+ + + Kyma-Managed Resource + + + + + + + + + + + + +
+
+
+ HTTP / gRPC +
+
+
+ + + HTTP / gRPC + + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/docs/user/assets/gateways-istio.drawio.svg b/docs/user/assets/pipeline-structure.drawio.svg similarity index 53% rename from docs/user/assets/gateways-istio.drawio.svg rename to docs/user/assets/pipeline-structure.drawio.svg index b46b42972f..f68eff6cb6 100644 --- a/docs/user/assets/gateways-istio.drawio.svg +++ b/docs/user/assets/pipeline-structure.drawio.svg @@ -1,20 +1,20 @@ - + - + - + -
-
-
- +
+
+
+ Kyma Cluster
@@ -24,512 +24,541 @@
- + Kyma Cluster - + + + + + + +
+
+
+ + Backends +
+ +
+
+
+ + + Backends + + + + + + -
+
- Kyma-System Namespace + Telemetry Pipeline
- - Kyma-System Namespace + + Telemetry Pipeline - + -
+
-
- - User Namespace +
+ + + External +
+ Environment +
+
- - User Namespace + + External... - + -
+
-
- - User Namespace -
- +
+ Observability System
- - User Namespace + + Observability Syst... - + -
+
- Service Mesh / Istio -
+ Data Sources
- - Service Mesh / Istio + + Data Sources - - + + -
+
-
- HTTP/GRPC +
+ various +
+ protocols
- - HTTP/GRPC + + various... - + -
+
- - Application - + Observability System
- - Application + + Observability Syst... - + + -
+
-
- - Istio -
- Proxy - +
+ OTLP
- - Istio... + + OTLP - + + + + + + -
+
-
- Gateway +
+ Kubernetes Runtime
- - Gateway + + Kubernetes Runtime - + -
+
-
- - - Reference key: - - +
+ Application
- - Reference key: + + Application - + -
+
-
- - User-Managed Resource - +
+ Istio Sidecar
- - User-Managed Resource + + Istio Sidecar - + -
+
- - Application - + Nodes
- - Application + + Nodes - + -
+
-
- - Backend +
+ + Output +
- - Backend + + Output - + -
+
-
- - Backend +
+ + Input +
- - Backend + + Input - - + -
+
-
- HTTP/GRPC +
+ + Processing +
+
- - HTTP/GRPC + + Processing - - + -
+
-
- mTLS +
+ OTLP
- - mTLS + + OTLP - - + + + + + + + + + -
+
-
- mTLS +
+ Service Name Determination
- - mTLS + + Service Name Deter... - + -
+
- - Istio -
- Proxy - + Kubernetes Metadata Enrichment
- - Istio... + + Kubernetes Metadat... - - + - - + + + +
+
+
+ Custom Transform +
+
+
+ + + Custom Transform + + + - + -
-
-
- - - External -
- Environment -
-
-
-
-
- - +
+
+
+ Custom Filter
- - External... + + Custom Filter - + -
+
- Observability System -
- (like Datadog/Splunk/...) + OTLP
- - Observability System... + + OTLP - - + -
+
-
- - HTTP/GRPC +
+ Metrics-specific +
+ ( + + Prometheus, Istio, Runtime)
- - HTTP/GRPC + + Metrics-specific... - + -
+
-
- - Kyma-Managed Resource - +
+ Log-specific +
+
+ (Application) +
- - Kyma-Managed Resource + + Log-specific... diff --git a/docs/user/assets/telemetry-arch.drawio.svg b/docs/user/assets/telemetry-arch.drawio.svg index 231fc9e285..f77849e5a5 100644 --- a/docs/user/assets/telemetry-arch.drawio.svg +++ b/docs/user/assets/telemetry-arch.drawio.svg @@ -1,4 +1,868 @@ - - - -
Kyma Cluster
Kyma Cluster
Kyma-System Namespace
Kyma-System Namespace
User Namespace
User Namespace
Storage and Analysis
Storage and Analysis
External
Environment
External...
Logs 
Traces
Metrics
Logs...
Logs 
Traces
Metrics
Logs...
Observability System
(like Datadog/Splunk/...)
Observability System...
Observability System
(like Loki/Jaeger/...)
Observability System...
Logs
Traces
Metrics
Logs...
Instrumentation
Instrumentation
Kubernetes
System
Kubernetes...
Kyma
System
Kyma...
Collection
Collection
Telemetry
Manager
Telemetry...
Log Agent
(Fluent Bit)
Log Agent...
Trace Gateway
(Otel Collector)
Trace Gateway...
Reference key:
Reference key:
Processes Logs
Processes Logs
Processes Traces
Processes Traces
Processes Metrics
Processes Metrics
User-Managed Resource
User-Managed Resource
Stages of Observability
Stages of Observability
Metric Gateway/Agent
(Otel Collector)
Metric Gateway/Age...
Kyma User
Kyma User
analyzes
data
analyzes...
Metric Gateway/Agent
(OTel Collector)
Metric Gateway/Ag...
Trace Gateway
(OTel Collector)
Trace Gateway...
Log Gateway/Agent
(OTel Collector)
Log Gateway/Agent...
Application
Application
Application
Application
 Istio
 Proxy
Istio...
 Istio
 Proxy
Istio...
Kyma-Managed Resource
Kyma-Managed ResourceText is not SVG - cannot display \ No newline at end of file + + + + + + + + + + + + + +
+
+
+ + + Kyma Cluster +
+ + +
+
+
+ + + Kyma Cluster + + + + + + + + + + + +
+
+
+ + Kyma-System Namespace +
+ +
+
+
+ + + Kyma-System Namespace + + + + + + + + + + + + + + +
+
+
+ + User Namespace +
+ +
+
+
+ + + User Namespace + + + + + + + + + + + +
+
+
+ + Storage and Analysis +
+ +
+
+
+ + + Storage and Analysis + + + + + + + + + + + +
+
+
+ + + External +
+ Environment +
+ + +
+
+
+ + + External... + + + + + + + + + + + + +
+
+
+ Logs +
+ Traces +
+ Metrics +
+
+
+ + + Logs... + + + + + + + + + + + + +
+
+
+ Logs +
+ Traces +
+ Metrics +
+
+
+ + + Logs... + + + + + + + + + + + +
+
+
+ Observability +
+ System +
+
+
+
+
+ + + Observability... + + + + + + + + + + + +
+
+
+ Observability +
+ Backend +
+
+
+
+
+ + + Observability... + + + + + + + + + + + + +
+
+
+ + Logs +
+ Traces +
+ Metrics +
+ +
+
+
+ + + Logs... + + + + + + + + + + + +
+
+
+ + Instrumentation +
+ +
+
+
+ + + Instrumentation + + + + + + + + + + + +
+
+
+ Kubernetes +
+ System +
+
+
+ + + Kubernetes... + + + + + + + + + + + +
+
+
+ Kyma +
+ System +
+
+
+ + + Kyma... + + + + + + + + + + + +
+
+
+ + Collection + +
+ + and Processing +
+ +
+
+
+
+ + + Collection... + + + + + + + + + + + +
+
+
+ Telemetry +
+ Manager +
+
+
+ + + Telemetry... + + + + + + + + + + + +
+
+
+ Log Agent +
+ (Fluent Bit) +
+
+
+ + + Log Agent... + + + + + + + + + + + +
+
+
+ Trace Gateway +
+ (Otel Collector) +
+
+
+ + + Trace Gateway... + + + + + + + + + + + +
+
+
+ + + Reference key: + + +
+
+
+ + + Reference key: + + + + + + + + + + + +
+
+
+ + Processes Logs + +
+
+
+ + + Processes Logs + + + + + + + + + + + +
+
+
+ + Processes Traces + +
+
+
+ + + Processes Traces + + + + + + + + + + + +
+
+
+ + Processes Metrics + +
+
+
+ + + Processes Metrics + + + + + + + + + + + +
+
+
+ + User-Managed Resource + +
+
+
+ + + User-Managed Resource + + + + + + + + + + + +
+
+
+ + + Stages of Observability + +
+ +
+
+
+ + + Stages of Observability + + + + + + + + + + + +
+
+
+ Metric Gateway/Agent +
+ (Otel Collector) +
+
+
+ + + Metric Gateway/Age... + + + + + + + + + + + + + + + +
+
+
+ + + Kyma User + + +
+
+
+ + + Kyma User + + + + + + + + + + + + +
+
+
+ + analyzes +
+ data +
+ +
+
+
+ + + analyzes... + + + + + + + + + + + + + + + +
+
+
+ Metric Gateway/Agent +
+ (OTel Collector) +
+
+
+ + + Metric Gateway/Ag... + + + + + + + + + + + +
+
+
+ Trace Gateway +
+ (OTel Collector) +
+
+
+ + + Trace Gateway... + + + + + + + + + + + +
+
+
+ Log Gateway/Agent +
+ (OTel Collector) +
+
+
+ + + Log Gateway/Agent... + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + Application + +
+
+
+ + + Application + + + + + + + + + + + +
+
+
+ + Application + +
+
+
+ + + Application + + + + + + + + + + + +
+
+
+ + Istio +
+ Proxy + +
+
+
+ + + Istio... + + + + + + + + + + + +
+
+
+ + Istio +
+ Proxy + +
+
+
+ + + Istio... + + + + + + + + + + + +
+
+
+ + Kyma-Managed Resource + +
+
+
+ + + Kyma-Managed Resource + + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/docs/user/assets/telemetry-stages.drawio.svg b/docs/user/assets/telemetry-stages.drawio.svg index 16b9507200..c9252d0d6c 100644 --- a/docs/user/assets/telemetry-stages.drawio.svg +++ b/docs/user/assets/telemetry-stages.drawio.svg @@ -1,4 +1,260 @@ - - - -
Observability Backend








Observability Backend...
Shipment
Shipment
Workload Runtime








Workload Runtime...
Instrumentation
Instrumentation
Collection and Infrastructure Enrichment
Collection and Inf...
Storage
Storage
Analysis,
Query API,
Reporting,
Alerting
Analysis,...
1
1
2
2
3
3
4
4Text is not SVG - cannot display \ No newline at end of file + + + + + + + + + + + + + +
+
+
+ + + Telemetry Module + + +
+
+
+ + + Telemetry Module + + + + + + + + + + + +
+
+
+ + + Backend +
+ + +
+
+
+ + + Backend + + + + + + + + + + + + + + + +
+
+
+ + Application +
+ +
+
+
+ + + Application + + + + + + + + + + + +
+
+
+ Instrument +
+
+
+ + + Instrument + + + + + + + + + + + +
+
+
+ Collect and Process +
+
+
+ + + Collect and Pr... + + + + + + + + + + + +
+
+
+ + Analyze and Visualize + +
+
+
+ + + Analyze and Vi... + + + + + + + + + + + + + + + +
+
+
+ Ship +
+
+
+ + + Ship + + + + + + + + + + + + + + + +
+
+
+ + + Reference key: + + +
+
+
+ + + Reference key: + + + + + + + + + + + +
+
+
+ + User-Managed Resource + +
+
+
+ + + User-Managed Resource + + + + + + + + + + + +
+
+
+ + Kyma-Managed Resource + +
+
+
+ + + Kyma-Managed Resource + + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/docs/user/assets/traces-arch.drawio.svg b/docs/user/assets/traces-arch.drawio.svg index 7ad85b444f..cc0eb4527d 100644 --- a/docs/user/assets/traces-arch.drawio.svg +++ b/docs/user/assets/traces-arch.drawio.svg @@ -1,4 +1,947 @@ - - - -
Kyma Cluster
Kyma Cluster
Kyma-System Namespace
Kyma-System Namespace
User Namespace
User Namespace
Storage and Analysis
Storage and Analysis
External
Environment
External...
sends traces
using OTLP
sends traces...
Observability System
(like Datadog/Splunk/...)
Observability System...
Observability System
(like Loki/Jaeger/...)
Observability System...
Instrumentation
Instrumentation
propagates
propagates
Kubernetes
System
Kubernetes...
Collection
Collection
Reference key:
Reference key:
analyzes
traces
analyzes...
7
7
6
6
Telemetry
Manager
Telemetry...
5
5
2
2

discovers Pod metadata
discovers Pod metadata
4
4
Processes Traces
Processes Traces
Trace Gateway
Trace Gateway
Trace Gateway
Trace Gateway

manages
and observes
manages...
Application
Application
 Istio
 Proxy
Istio...
3
3
Application
Application
 Istio
 Proxy
Istio...
1
1sends tracesusing OTLP

sends traces
using OTLP
sends traces...
User-Managed Resource
User-Managed Resource
Kyma-Managed Resource
Kyma-Managed Resource
Kyma User
Kyma UserText is not SVG - cannot display \ No newline at end of file + + + + + + + + + + + + + +
+
+
+ + + Kyma Cluster +
+ + +
+
+
+ + + Kyma Cluster + + + + + + + + + + + + + + +
+
+
+ + Kyma-System Namespace +
+ +
+
+
+ + + Kyma-System Namespace + + + + + + + + + + + +
+
+
+ + User Namespace + +
+
+
+ + + User Namespace + + + + + + + + + + + +
+
+
+ + Storage and Analysis +
+ +
+
+
+ + + Storage and Analysis + + + + + + + + + + + +
+
+
+ + + External +
+ Environment +
+ + +
+
+
+ + + External... + + + + + + + + + + + + + + + + +
+
+
+ sends traces +
+ using OTLP +
+
+
+ + + sends traces... + + + + + + + + + + + +
+
+
+ Observability +
+ Backend +
+
+
+
+ + + Observability... + + + + + + + + + + + +
+
+
+ Observability +
+ Backend +
+
+
+
+ + + Observability... + + + + + + + + + + + +
+
+
+ + Instrumentation +
+ +
+
+
+ + + Instrumentation + + + + + + + + + + + + +
+
+
+ propagates +
+
+
+ + + propagates + + + + + + + + + + + +
+
+
+ Kubernetes +
+ System +
+
+
+ + + Kubernetes... + + + + + + + + + + + +
+
+
+ + Collection and Processing +
+ +
+
+
+ + + Collection and Processing + + + + + + + + + + + +
+
+
+ + + Reference key: + + +
+
+
+ + + Reference key: + + + + + + + + + + + + +
+
+
+ + analyzes +
+ traces + +
+
+
+ + + analyzes... + + + + + + + + + + + + + + + +
+
+
+ + 7 + +
+
+
+ + + 7 + + + + + + + + + + + +
+
+
+ + 6 + +
+
+
+ + + 6 + + + + + + + + + + + +
+
+
+ Telemetry +
+ Manager +
+
+
+ + + Telemetry... + + + + + + + + + + + +
+
+
+ + 5 + +
+
+
+ + + 5 + + + + + + + + + + + +
+
+
+ + 2 + +
+
+
+ + + 2 + + + + + + + + + + + + +
+
+
+ +
+ +
+
+
+ + + + + + + + + +
+
+
+ + discovers + + + Pod metadata + +
+
+
+ + + discovers Pod metadata + + + + + + + + + + + +
+
+
+ + 4 + +
+
+
+ + + 4 + + + + + + + + + + + +
+
+
+ + Processes Traces + +
+
+
+ + + Processes Traces + + + + + + + + + + + +
+
+
+ Trace Gateway +
+
+
+ + + Trace Gateway + + + + + + + + + + + +
+
+
+ Trace Gateway +
+
+
+ + + Trace Gateway + + + + + + + + + + + + +
+
+
+ +
+ +
+
+
+ + + + + + + + + +
+
+
+ + manages +
+ and observes + +
+
+
+ + + manages... + + + + + + + + + + + + + + +
+
+
+ + Application + +
+
+
+ + + Application + + + + + + + + + + + +
+
+
+ + Istio +
+ Proxy + +
+
+
+ + + Istio... + + + + + + + + + + + +
+
+
+ + 3 + +
+
+
+ + + 3 + + + + + + + + + + + + + + +
+
+
+ + Application + +
+
+
+ + + Application + + + + + + + + + + + +
+
+
+ + Istio +
+ Proxy + +
+
+
+ + + Istio... + + + + + + + + + + + +
+
+
+ + 1 + +
+
+
+ + + 1 + + + + + + + + + + + + sends traces + + + using OTLP + + + + + + + + + + + + + + + +
+
+
+ +
+ +
+
+
+ + + + + + + + + +
+
+
+ + sends traces + +
+ + using OTLP + +
+
+
+ + + sends traces... + + + + + + + + + + + +
+
+
+ + User-Managed Resource + +
+
+
+ + + User-Managed Resource + + + + + + + + + + + +
+
+
+ + Kyma-Managed Resource + +
+
+
+ + + Kyma-Managed Resource + + + + + + + + + + + + + + + +
+
+
+ + + Kyma User + + +
+
+
+ + + Kyma User + + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/docs/user/assets/traces-intro.drawio.svg b/docs/user/assets/traces-intro.drawio.svg deleted file mode 100644 index 93ad5ab6d9..0000000000 --- a/docs/user/assets/traces-intro.drawio.svg +++ /dev/null @@ -1,275 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
-
-
- - - No Distributed Tracing - - -
-
-
- - - No Distributed Tracing - - - - - - -
-
-
- - - Distributed Tracing - - -
-
-
- - - Distributed Tracing - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
-
-
- service-to-service connection -
-
-
- - - service-to-service connection - - - - - - -
-
-
- individual request path -
-
-
-
-
- - - individual request path... - - - - - - - - - Text is not SVG - cannot display - - - - \ No newline at end of file diff --git a/docs/user/assets/traces-resources.drawio.svg b/docs/user/assets/traces-resources.drawio.svg index 057a14440a..5952e5a7c1 100644 --- a/docs/user/assets/traces-resources.drawio.svg +++ b/docs/user/assets/traces-resources.drawio.svg @@ -1,4 +1,628 @@ - - - -
Kyma Cluster
Kyma Cluster
Reference key:
Reference key:
Kyma-System Namespace
Kyma-System Namespace
<<Deployment>>
telemetry-manager
<<Deployment>>...
watches
watches
<<TracePipeline>>
custom
<<TracePipeline>>...
User Namespace
User Namespace
<<Secret>>
custom
<<Secret>>...
references
references
Processes Traces
Processes Traces
<<Deployment>>
telemetry-trace-gateway
<<Deployment>>...
<<ConfigMap>>
 telemetry-trace-gateway

<<ConfigMap>>...
<<Secret>>
telemetry-trace-gateway

<<Secret>>...
mounts
as volume
mounts...
references
as env
references...
manages
lifecycle
manages...
3
3
2
2
generates
configuration
generates...
copies user Secrets
into one gateway Secret
copies user Secrets...
4
4
1
1
User-Managed Resource
User-Managed Resource
Kyma-Managed Resource
Kyma-Managed ResourceText is not SVG - cannot display \ No newline at end of file + + + + + + + + + + + + + +
+
+
+ + + Kyma Cluster +
+ + +
+
+
+ + + Kyma Cluster + + + + + + + + + + + +
+
+
+ + + Reference key: + + +
+
+
+ + + Reference key: + + + + + + + + + + + +
+
+
+ + Kyma-System Namespace +
+ +
+
+
+ + + Kyma-System Namespace + + + + + + + + + + + +
+
+
+ <<Deployment>> +
+ telemetry-manager +
+
+
+ + + <<Deployment>>... + + + + + + + + + + + + + + + +
+
+
+ watches +
+
+
+ + + watches + + + + + + + + + + + +
+
+
+ <<TracePipeline>> +
+ custom +
+
+
+ + + <<TracePipeline>>... + + + + + + + + + + + +
+
+
+ + User Namespace +
+ +
+
+
+ + + User Namespace + + + + + + + + + + + + + + +
+
+
+ <<Secret>> +
+ custom +
+
+
+ + + <<Secret>>... + + + + + + + + + + + + + + + + +
+
+
+ + references + +
+
+
+ + + references + + + + + + + + + + + +
+
+
+ + Processes Traces + +
+
+
+ + + Processes Traces + + + + + + + + + + + +
+
+
+ <<Deployment>> +
+ telemetry-trace-gateway +
+
+
+ + + <<Deployment>>... + + + + + + + + + + + +
+
+
+
+ + <<ConfigMap>> + +
+
+ + + + telemetry-trace-gateway + + + +
+ +
+ +
+
+
+ + + <<ConfigMap>>... + + + + + + + + + + + +
+
+
+ +
+ + + <<Secret>> + + +
+
+ + + telemetry-trace-gateway + + +
+
+ +
+
+
+ + + <<Secret>>... + + + + + + + + + + + + +
+
+
+ + mounts +
+ as volume + +
+
+
+ + + mounts... + + + + + + + + + + + + +
+
+
+ + references +
+ + as env +
+
+
+ + + references... + + + + + + + + + + + + +
+
+
+ manages +
+ lifecycle +
+
+
+ + + manages... + + + + + + + + + + + +
+
+
+ + 3 + +
+
+
+ + + 3 + + + + + + + + + + + +
+
+
+ + 2 + +
+
+
+ + + 2 + + + + + + + + + + + + +
+
+
+ + generates + +
+ + configuration + +
+
+
+ + + generates... + + + + + + + + + + + + +
+
+
+ + copies user Secrets +
+ into one gateway Secret + +
+
+
+ + + copies user Secrets... + + + + + + + + + + + +
+
+
+ + 4 + +
+
+
+ + + 4 + + + + + + + + + + + +
+
+
+ + 1 + +
+
+
+ + + 1 + + + + + + + + + + + +
+
+
+ + User-Managed Resource + +
+
+
+ + + User-Managed Resource + + + + + + + + + + + +
+
+
+ + Kyma-Managed Resource + +
+
+
+ + + Kyma-Managed Resource + + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/docs/user/collecting-logs/README.md b/docs/user/collecting-logs/README.md new file mode 100644 index 0000000000..738e7bf57a --- /dev/null +++ b/docs/user/collecting-logs/README.md @@ -0,0 +1,64 @@ +# Collecting Logs + +With the Telemetry module, you can observe and debug your applications by collecting, processing, and exporting logs. To begin collecting logs, you create a LogPipeline resource. It automatically collects OTLP logs and application logs from the `stdout`/`stderr` channel. You can also activate Istio log collection. + +## Overview + +A LogPipeline is a Kubernetes custom resource (CR) that configures log collection for your cluster. When you create a LogPipeline, the Telemetry Manager automatically deploys the necessary components (for details, see [Logs Architecture](./../architecture/README.md)): + +- A **log gateway** that provides a central OTLP endpoint for receiving logs pushed from your applications. +- A **log agent** that runs on each cluster node to collect logs written to `stdout` and `stderr` by your application containers. + +The pipeline enriches all collected logs with Kubernetes metadata and transforms them into the OTLP format before sending them to your chosen backend. + +Log collection is optional. If you don't create a LogPipeline, the log collection components are not deployed. + +## Prerequisites + +- Before you can collect logs from a component, it must emit the logs. Typically, it uses a logger framework for the used language runtime (like Node.js) and prints them to the `stdout` or `stderr` channel (see [Kubernetes: How nodes handle container logs](https://kubernetes.io/docs/concepts/cluster-administration/logging/#how-nodes-handle-container-logs)). Alternatively, you can use the [OTel SDK](https://opentelemetry.io/docs/languages/) to use the [push-based OTLP format](https://opentelemetry.io/docs/specs/otlp/). + +- If you want to emit the logs to the `stdout/stderr` channel, use structured logs in a JSON format with a logger library like log4J. With that, the log agent can parse your log and enrich all JSON attributes as log attributes, and a backend can use that. + +- If you prefer the push-based alternative with OTLP, also use a logger library like log4J. However, you must additionally instrument that logger and bridge it to the OTel SDK. For details, see [OpenTelemetry: New First-Party Application Logs](https://opentelemetry.io/docs/specs/otel/logs/#new-first-party-application-logs). + +## Minimal LogPipeline + +For a minimal setup, you only need to create a LogPipeline that specifies your backend destination (see [Integrate With Your OTLP Backend](./../integrate-otlp-backend/README.md)): + +```yaml +apiVersion: telemetry.kyma-project.io/v1alpha1 +kind: LogPipeline +metadata: + name: backend +output: + otlp: + endpoint: + value: http://myEndpoint:4317 +``` + +By default, this minimal pipeline enables the following types of log collection: + +- **Application logs**: Collects `stdout` and `stderr` logs from all containers running in non-system namespaces (such as `kyma-system` and `kube-system`). +- **OTLP logs**: Activates cluster-internal endpoints to receive logs in the OTLP format. Your applications can push logs directly to these URLs: + - gRPC: `http://telemetry-otlp-logs.kyma-system:4317` + - HTTP: `http://telemetry-otlp-logs.kyma-system:4318` + +## Configure Log Collection + +You can customize your LogPipeline using the available parameters and attributes (see [LogPipeline: Custom Resource Parameters](https://kyma-project.io/#/telemetry-manager/user/resources/02-logpipeline?id=custom-resource-parameters)): + +- Configure or disable the collection of application logs from the `stdout`/`stderr` channel (see [Configure Application Logs](../collecting-logs/application-input.md)). +- Set up the collection of Istio access logs (see [Configure Istio Access Logs](../collecting-logs/istio-support.md)). +- Choose from which specific namespaces you want to include or exclude logs (see [Filter Logs](../filter-and-process/filter-logs.md)). +- If you have more than one backend, specify which **input** source sends logs to which backend (see [Route Specific Inputs to Different Backends](../otlp-input.md#route-specific-inputs-to-different-backends)). + +## Limitations + +- **Throughput**: + - When pushing OTLP logs of an average size of 2KB to the log gateway, using its default configuration (two instances), the Telemetry module can process approximately 12,000 logs per second (LPS). To ensure availability, the log gateway runs with multiple instances. For higher throughput, manually scale out the gateway by increasing the number of replicas (see [Module Configuration and Status](https://kyma-project.io/#/telemetry-manager/user/01-manager?id=module-configuration)). Ensure that the chosen scaling factor does not exceed the maximum throughput of the backend, as it may refuse logs if the rate is too high. + - For example, to scale out the gateway for scenarios like a `Large` instance of SAP Cloud Logging (up to 30,000 LPS), you can raise the throughput to about 20,000 LPS by increasing the number of replicas to 4 instances. + - The log agent, running one instance per node, handles tailing logs from stdout using the **runtime** input. When writing logs of an average size of 2KB to stdout, a single log agent instance can process approximately 9,000 LPS. +- **Load Balancing With Istio**: By design, the connections to the gateway are long-living connections (because OTLP is based on gRPC and HTTP/2). For optimal scaling of the gateway, the clients or applications must balance the connections across the available instances, which is automatically achieved if you use an Istio sidecar. If your application has no Istio sidecar, the data is always sent to one instance of the gateway. +- **Unavailability of Output**: For up to 5 minutes, a retry for data is attempted when the destination is unavailable. After that, data is dropped. +- **No Guaranteed Delivery**: The used buffers are volatile. If the gateway or agent instances crash, logs data can be lost. +- **Multiple LogPipeline Support**: The maximum amount of LogPipeline resources is 5. diff --git a/docs/user/collecting-logs/application-input.md b/docs/user/collecting-logs/application-input.md new file mode 100644 index 0000000000..5d05fd1753 --- /dev/null +++ b/docs/user/collecting-logs/application-input.md @@ -0,0 +1,43 @@ +# Configure Application Logs + +To collect logs that your applications write to `stdout` and `stderr`, create a LogPipeline. The **application** input is enabled by default and uses an agent on each node to tail container log files. You can control which namespaces and containers to include or exclude. + +## Prerequisites + +- You have the Telemetry module in your cluster. +- You have access to Kyma dashboard. Alternatively, if you prefer CLI, you need [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl). + +## Context + +Use the **application** input section to restrict or specify which resources you want to include. You can define the namespaces to include in the input collection, exclude namespaces from the input collection, or choose that only system namespaces are included. For details, see [LogPipeline: Custom Resource Parameters](https://kyma-project.io/#/telemetry-manager/user/resources/02-logpipeline?id=custom-resource-parameters). + +When you apply the LogPipeline resource to your Kubernetes cluster, a log agent is deployed and starts collecting the log data, transforms them to OTLP, and sends them to your backend. For details, see [Transformation to OTLP Logs](../filter-and-process/transformation-to-otlp-logs.md). + +## Enable or Disable Log Collection + +The **application** input is enabled by default. To create a pipeline that only accepts logs pushed with OTLP, you can disable it. + +```yaml + ... + input: + application: + enabled: false # Default is true +``` + +By default, input is collected from all namespaces, except the system namespaces `kube-system`, `istio-system`, `kyma-system`, which are excluded by default. + +> [!TIP] +> To select logs from specific namespaces and containers, or to include system namespaces, see [Filter Logs](../filter-and-process/filter-logs.md). + +## Discard the Original Log Body + +By default, the log agent preserves the original JSON log message by moving it to the **attributes."log.original"** field after parsing. For details, see [Transformation to OTLP Logs](../filter-and-process/transformation-to-otlp-logs.md). + +To reduce data volume, you can disable this behavior. Set the parameter to `false` to discard the original JSON string after its contents are parsed into attributes. + +```yaml + ... + input: + application: + keepOriginalBody: false # Default is true +``` \ No newline at end of file diff --git a/docs/user/collecting-logs/istio-support.md b/docs/user/collecting-logs/istio-support.md new file mode 100644 index 0000000000..3fbe07b654 --- /dev/null +++ b/docs/user/collecting-logs/istio-support.md @@ -0,0 +1,77 @@ +# Configure Istio Access Logs + +To monitor traffic in your service mesh, configure Istio to send access logs. The LogPipeline automatically receives these logs through its default OTLP input. + +## Prerequisites + +- You have the Istio module in your cluster. See [Quick Install](https://kyma-project.io/#/02-get-started/01-quick-install). +- You have access to Kyma dashboard. Alternatively, if you prefer CLI, you need [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl). + +## Context + +Istio access logs help you monitor the "four golden signals" (latency, traffic, errors, and saturation) and troubleshoot anomalies. + +By default, these logs are disabled because they can generate a high volume of data. To collect them, you apply an [Istio](https://istio.io/latest/docs/reference/config/telemetry/) `Telemetry` resource to a specific namespace, for the Istio Ingress Gateway, or for the entire mesh. + +> [!WARNING] +> Enabling access logs, especially for the entire mesh, can significantly increase log volume and may lead to higher storage costs. Enable this feature only for the resources or components that you want to monitor. + +After enabling Istio access logs, reduce data volume and costs by filtering them (see [Filter Logs](../filter-and-process/filter-logs.md)). + +The Istio module provides a preconfigured [extension provider](https://istio.io/latest/docs/reference/config/istio.mesh.v1alpha1/#MeshConfig-ExtensionProvider) called `kyma-logs`, which tells Istio to send access logs to the Telemetry module's OTLP endpoint. If your LogPipeline uses the legacy **http** output, you must use the `stdout-json` provider instead. + +> [!NOTE] +> You can only have one mesh-wide Istio `Telemetry` resource. If you also plan to enable Istio tracing (see [Configure Istio Tracing](./../collecting-traces/istio-support.md)), configure both access logging and tracing in this single resource. + +## Enable Istio Logs for a Namespace + +Apply the Istio `Telemetry` resource to a specifc namespace: + +```yaml +apiVersion: telemetry.istio.io/v1 +kind: Telemetry +metadata: + name: access-config + namespace: $YOUR_NAMESPACE +spec: + accessLogging: + - providers: + - name: kyma-logs +``` + +## Enable Istio Logs for the Ingress Gateway + +To monitor all traffic entering your mesh, enable access logs on the Istio Ingress Gateway (instead of the individual proxies of your workloads). + +Apply the Istio `Telemetry` resource to the `istio-system` namespace, selecting the gateway Pods: + +```yaml +apiVersion: telemetry.istio.io/v1 +kind: Telemetry +metadata: + name: mesh-default + namespace: istio-system +spec: + selector: + matchLabels: + istio: ingressgateway + accessLogging: + - providers: + - name: kyma-logs +``` + +## Enable Istio Logs for the Entire Mesh + +To enable access logs globally for all proxies in the mesh, apply the Istio `Telemetry` resource to the `istio-system` namespace. Use this option with caution due to the high data volume. + +```yaml +apiVersion: telemetry.istio.io/v1 +kind: Telemetry +metadata: + name: mesh-default + namespace: istio-system +spec: + accessLogging: + - providers: + - name: kyma-logs +``` \ No newline at end of file diff --git a/docs/user/collecting-metrics/README.md b/docs/user/collecting-metrics/README.md new file mode 100644 index 0000000000..b1e5b6475f --- /dev/null +++ b/docs/user/collecting-metrics/README.md @@ -0,0 +1,69 @@ +# Collecting Metrics + +With the Telemetry module, you can collect metrics from your workloads and Kubernetes resources to monitor their health, performance, and behavior. To begin collecting metrics, you create a MetricPipeline resource. You can collect Prometheus, Istio, and runtime metrics. + +## Overview + +A MetricPipeline is a Kubernetes custom resource (CR) that configures metric collection for your cluster. When you create a MetricPipeline, it automatically provisions the necessary components (for details, see [Metrics Architecture](../architecture/metrics-architecture.md)): + +- A metric gateway that provides a central OTLP endpoint for receiving metrics pushed from applications. +- A metric agent that runs on each cluster node to pull (scrape) metrics from applications and Kubernetes resources. + +The pipeline enriches all collected metrics with Kubernetes metadata. It also transforms non-OTLP formats (like Prometheus) into the OTLP standard before sending them to your chosen backend. + +Metrics collection is optional. If you don't create a MetricPipeline, the metric collection components are not deployed. + +## Prerequisites + +Before you can collect metrics data from a component, it must expose (or instrument) the metrics. Typically, it instruments specific metrics for the used language runtime (like Node.js) and custom metrics specific to the business logic. Also, the exposure can be in different formats, like the pull-based Prometheus format or the [push-based OTLP format](https://opentelemetry.io/docs/specs/otlp/). + +If you use Prometheus-based metrics, instrument your application with a library like the Prometheus client library. Expose a port in your workload to serve as a Prometheus metrics endpoint. + +If you scrape the metric endpoint with Istio, define the app protocol in your Service port definition. See [Collect Prometheus Metrics](prometheus-input.md). + +For instrumentation, use an SDK, such as the Prometheus client libraries or the OpenTelemetry SDKs. Both libraries provide extensions to activate language-specific auto-instrumentation (for example, for Node.js) and an API to implement custom instrumentation. + +## Minimal MetricPipeline + +For a minimal setup, you only need to create a MetricPipeline that specifies your backend destination (see [Integrate With Your OTLP Backend](./../integrate-otlp-backend/README.md)): + +```yaml +apiVersion: telemetry.kyma-project.io/v1alpha1 +kind: MetricPipeline +metadata: + name: backend +output: + otlp: + endpoint: + value: http://myEndpoint:4317 +``` + +By default, this minimal pipeline collects the following types of metrics: + +- OTLP Metrics: Activates cluster-internal endpoints to receive metrics in the OTLP format. Your applications can push metrics directly to these URLs: + - gRPC: `http://telemetry-otlp-metrics.kyma-system:4317` + - HTTP: `http://telemetry-otlp-metrics.kyma-system:4318` +- Health Metrics: Collects health and performance metrics about the Telemetry module's components. This input is always active and cannot be disabled. For details, see [Monitor Pipeline Health](../monitor-pipeline-health.md). + +To collect metrics from Kyma modules like Istio, Eventing, or Serverless, enable additional inputs. + +## Configure Metrics Collection + +You can adjust the MetricPipeline using runtime configuration with the available parameters (see [MetricPipeline: Custom Resource Parameters](https://kyma-project.io/#/telemetry-manager/user/resources/05-metricpipeline?id=custom-resource-parameters)). + +- Scrape **prometheus** metrics from applications that expose a Prometheus-compatible endpoint (see [Collect Prometheus Metrics](prometheus-input.md)). +- Collect **istio** service mesh metrics from Istio proxies and control plane components (see [Collect Istio Metrics](istio-input.md)). +- Collect **runtime** resource usage and status metrics from Kubernetes components like Pods, Nodes, and Deployments (see [Collect Runtime Metrics](runtime-input.md)). +- Use diagnostic metrics to debug your **prometheus** and **istio** configuration (see [Collect Diagnostic Metrics](./prometheus-input.md#collect-diagnostic-metrics)). +- Choose from which specific namespaces you want to include or exclude metrics (see [Filter Metrics](../filter-and-process/filter-metrics.md)). +- Avoid redundancy by dropping push-based OTLP metrics that are sent directly to the metric gateway (see [Route Specific Inputs to Different Backends](./../otlp-input.md#route-specific-inputs-to-different-backends)). + +## Limitations + +- **Throughput**: Assuming an average metric with 20 metric data points and 10 labels, the default metric **gateway** setup has a maximum throughput of 34K metric data points/sec. If more data is sent to the gateway, it is refused. To increase the maximum throughput, manually scale out the gateway by increasing the number of replicas for the metric gateway (see [Module Configuration and Status](https://kyma-project.io/#/telemetry-manager/user/01-manager?id=module-configuration)). + The metric **agent** setup has a maximum throughput of 14K metric data points/sec per instance. If more data must be ingested, it is refused. If a metric data endpoint emits more than 50.000 metric data points per scrape loop, the metric agent refuses all the data. +- **Load Balancing With Istio**: To ensure availability, the metric gateway runs with multiple instances. If you want to increase the maximum throughput, use manual scaling and enter a higher number of instances. + By design, the connections to the gateway are long-living connections (because OTLP is based on gRPC and HTTP/2). For optimal scaling of the gateway, the clients or applications must balance the connections across the available instances, which is automatically achieved if you use an Istio sidecar. If your application has no Istio sidecar, the data is always sent to one instance of the gateway. +- **Unavailability of Output**: For up to 5 minutes, a retry for data is attempted when the destination is unavailable. After that, data is dropped. +- **No Guaranteed Delivery**: The used buffers are volatile. If the gateway or agent instances crash, metric data can be lost. +- **Multiple MetricPipeline Support**: The maximum amount of MetricPipeline resources is 5. diff --git a/docs/user/collecting-metrics/istio-input.md b/docs/user/collecting-metrics/istio-input.md new file mode 100644 index 0000000000..f0547af872 --- /dev/null +++ b/docs/user/collecting-metrics/istio-input.md @@ -0,0 +1,71 @@ +# Collect Istio Metrics + +To monitor the health and performance of your service mesh, enable the **istio** input in your MetricPipeline. This scrapes metrics directly from Istio proxies (sidecars) and the control plane. You can enable Envoy and diagnostic metrics, and control from which namespaces metrics are collected. + +## Prerequisites + +- The Istio module is added in your cluster. See [Quick Install](https://kyma-project.io/#/02-get-started/01-quick-install). +- You have access to Kyma dashboard. Alternatively, if you prefer CLI, you need [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl). + +> [!NOTE] +> This configuration collects metrics generated by the Istio proxies themselves. To collect custom metrics exposed by your own applications running in the service mesh, you must configure your application accordingly. For details, see [Scrape Metrics from Istio-enabled Workloads](prometheus-input.md#scrape-metrics-from-istio-enabled-workloads). + +## Activate Istio Metrics + +By default, the **istio** input is disabled. If you are using Istio, enable the collection of Istio metrics: + +```yaml + ... + input: + istio: + enabled: true +``` + +With this, the metric agent starts collecting all Istio metrics from Istio sidecars from all namespaces (including system namespaces). + +> [!TIP] +> To validate or debug your configuration, use diagnostic metrics (see [Collect Diagnostic Metrics](#collect-diagnostic-metrics)). +> To select metrics from specific namespaces, see [Filter Metrics](../filter-and-process/filter-metrics.md). + +## Collect Envoy Metrics + +By default, the metric agent collects only Istio metrics (prefixed with `istio_`) and ignores Envoy metrics (prefixed with `envoy_`). + +Envoy metrics help you understand the performance and behavior of your Envoy proxy, providing details like request rates, latencies, and error counts. For details, see [Envoy metrics](https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cluster_stats) and [server metrics](https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/statistics). + +To use Envoy metrics to observe and troubleshoot service mesh traffic, enable the **envoyMetrics** section under the **istio** input: + +```yaml + ... + input: + istio: + enabled: true + envoyMetrics: + enabled: true +``` + +## Collect Diagnostic Metrics + +To validate or debug your scraping configuration for the **prometheus** and **istio** input, you can use diagnostic metrics. By default, they are disabled. + +> [!NOTE] +> Unlike the **prometheus** and **istio** inputs, the **runtime** input gathers data directly from Kubernetes APIs instead of using a scraping process, so it does not generate scrape-specific diagnostic metrics. + +To use diagnostic metrics, enable the **diagnosticMetrics** for the input in your MetricPipeline: + +```yaml + ... + input: + : + enabled: true + diagnosticMetrics: + enabled: true +``` + +When enabled, the metric agent generates metrics about its own scrape jobs, such as the following: + +- `up`: The scraping was successful +- `scrape_duration_seconds`: Duration of the scrape +- `scrape_samples_scraped`: The number of samples the target exposed +- `scrape_samples_post_metric_relabeling`: The number of samples remaining after metric relabeling was applied +- `scrape_series_added`: The approximate number of new series in this scrape diff --git a/docs/user/collecting-metrics/prometheus-input.md b/docs/user/collecting-metrics/prometheus-input.md new file mode 100644 index 0000000000..42bbd31b23 --- /dev/null +++ b/docs/user/collecting-metrics/prometheus-input.md @@ -0,0 +1,97 @@ +# Collect Prometheus Metrics + +To collect metrics from applications that expose a Prometheus-compatible endpoint, enable the **prometheus** input in your MetricPipeline and annotate your Pods or Services for discovery. You can enable diagnostic metrics and control from which namespaces metrics are collected. + +## Prerequisites + +Instrument your application using a library like the [Prometheus client library](https://prometheus.io/docs/instrumenting/clientlibs/) or the OTel SDK (with [Prometheus exporter](https://opentelemetry.io/docs/specs/otel/metrics/sdk_exporters/prometheus/)). Expose a port in your workload as a Prometheus metrics endpoint. + +## Activate Prometheus Metrics + +By default, the **prometheus** input is disabled. If your applications emit Prometheus metrics, enable the collection of Prometheus-based metrics: + +```yaml + ... + input: + prometheus: + enabled: true +``` + +> [!TIP] +> To validate or debug your configuration, use diagnostic metrics (see [Collect Diagnostic Metrics](#collect-diagnostic-metrics)). +> To select metrics from specific namespaces or to include system namespaces, see [Filter Metrics](../filter-and-process/filter-metrics.md). + +## Enable Metrics Collection With Annotations + +The metric agent automatically discovers Prometheus endpoints in your cluster by looking for specific annotations on your Kubernetes Services or Pods. + +To enable automatic metrics collection, apply the following annotations. If your Pod has an Istio sidecar, annotate the Service. Otherwise, annotate the Pod directly. + +> [!NOTE] +> If your service mesh enforces `STRICT` mTLS, the agent scrapes the endpoint over HTTPS automatically. If you don't use `STRICT` mTLS, add the annotation `prometheus.io/scheme: http` to force scraping over plain HTTP. + +| Annotation Key | Values | Description | +|----------------------------------|------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------| +| `prometheus.io/scrape` (mandatory) | true, false (no default value) | Set to true to enable scraping for this target. | +| `prometheus.io/port` (mandatory) | 8080, 9100 (no default value) | Specify the port on the Pod where your application exposes metrics. | +| `prometheus.io/path` | /metrics (default), /custom_metrics | Set the HTTP path for the metrics endpoint. | +| `prometheus.io/scheme` | https-metrics (default with Istio), http (default without Istio) | Define the protocol for scraping: Either HTTPS with mTLS, or plain HTTP. | +| `prometheus.io/param_` | Example: format: prometheus (no default) | Add a URL parameter to the scrape request. For example, prometheus.io/param_format: prometheus adds ?format=prometheus to the URL. | + +For example, see the following `Service` configuration: + +```yaml +apiVersion: v1 +kind: Service +metadata: + annotations: + prometheus.io/port: "8080" + prometheus.io/scrape: "true" + name: sample +spec: + ports: + - name: http-metrics + appProtocol: http + port: 8080 + protocol: TCP + targetPort: 8080 + selector: + app: sample + type: ClusterIP +``` + +## Scrape Metrics from Istio-enabled Workloads + +If your application is part of an Istio service mesh, you must consider service port naming and mutual TLS (mTLS) configuration: + +- Istio must be able to identify the **appProtocol** from the Service port definition. Otherwise, Istio may block the scrape request. + You must either prefix the port name with the protocol like in `http-metrics`, or explicitly define the **appProtocol** attribute. + +- The metric agent can scrape endpoints from workloads that enforce mutual TLS (mTLS). For scraping through HTTPS, Istio must configure the workload using STRICT mTLS mode. + If you can't use STRICT mTLS mode, you can set up scraping through plain HTTP by adding the following annotation to your Service: `prometheus.io/scheme: http`. For related troubleshooting, see [MetricPipeline: Failed to Scrape Prometheus Endpoint](../troubleshooting.md#metricpipeline-failed-to-scrape-prometheus-endpoint). + +## Collect Diagnostic Metrics + +To validate or debug your scraping configuration for the **prometheus** and **istio** input, you can use diagnostic metrics. By default, they are disabled. + +> [!NOTE] +> Unlike the **prometheus** and **istio** inputs, the **runtime** input gathers data directly from Kubernetes APIs instead of using a scraping process, so it does not generate scrape-specific diagnostic metrics. + +To use diagnostic metrics, enable the **diagnosticMetrics** for the input in your MetricPipeline: + +```yaml + ... + input: + : + enabled: true + diagnosticMetrics: + enabled: true +``` + +When enabled, the metric agent generates metrics about its own scrape jobs, such as the following: + +- `up`: The scraping was successful +- `scrape_duration_seconds`: Duration of the scrape +- `scrape_samples_scraped`: The number of samples the target exposed +- `scrape_samples_post_metric_relabeling`: The number of samples remaining after metric relabeling was applied +- `scrape_series_added`: The approximate number of new series in this scrape diff --git a/docs/user/collecting-metrics/runtime-input.md b/docs/user/collecting-metrics/runtime-input.md new file mode 100644 index 0000000000..1e5e810119 --- /dev/null +++ b/docs/user/collecting-metrics/runtime-input.md @@ -0,0 +1,64 @@ +# Collect Runtime Metrics + +To monitor the health and resource usage of your Kubernetes cluster, enable the **runtime** input in your MetricPipeline. This uses an agent on each node to gather metrics for resources like Pods, Nodes, and Deployments. You can choose the specific resources to monitor and control from which namespaces metrics are collected. + +## Activate Runtime Metrics + +By default, the **runtime** input is disabled. If you want to monitor your Kubernetes resources, enable the collection of runtime metrics: + +```yaml +... + input: + runtime: + enabled: true +``` + +With this, the metric agent starts collecting all runtime metrics from all resources (Pod, container, Node, Volume, DaemonSet, Deployment, StatefulSet, and Job). + +> [!TIP] +> To select metrics from specific namespaces or to include system namespaces, see [Filter Metrics](../filter-and-process/filter-metrics.md). + +## Select Resource Types + +By default, metrics for all supported resource types are collected. To enable or disable the collection of metrics for a specific resource, use the **resources** section in the **runtime** input. + +The following example collects only DaemonSet, Deployment, StatefulSet, and Job metrics: + + ```yaml + ... + input: + runtime: + enabled: true + resources: + pod: + enabled: false + container: + enabled: false + node: + enabled: false + volume: + enabled: false + daemonset: + enabled: true + deployment: + enabled: true + statefulset: + enabled: true + job: + enabled: true + ``` + +See a summary of the types of information you can gather for each resource: + +| Resource | Metrics Collected | +|-------------|-------------------------------------------------------------------------| +| pod | CPU, memory, filesystem, and network usage; current Pod phase | +| container | CPU/memory requests, limits, and usage; container restart count | +| node | Aggregated CPU, memory, filesystem, and network usage for the Node | +| volume | Filesystem capacity, usage, and inode statistics for persistent volumes | +| deployment | Number of desired versus available replicas | +| daemonset | Number of desired, current, and ready Nodes | +| statefulset | Number of desired, current, and ready Pods | +| job | Counts of active, successful, and failed Pods | + +To learn which specific metrics are collected from which source (`kubletstatsreceiver` or `k8sclusterreceiver`), see [Runtime Metrics](runtime-metrics.md). diff --git a/docs/user/collecting-metrics/runtime-metrics.md b/docs/user/collecting-metrics/runtime-metrics.md new file mode 100644 index 0000000000..4190b6a23b --- /dev/null +++ b/docs/user/collecting-metrics/runtime-metrics.md @@ -0,0 +1,98 @@ +# Runtime Metrics + +## Pod Metrics + +If `pod` metrics are enabled, the following metrics are collected: + +- From the [kubletstatsreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/kubeletstatsreceiver): + - `k8s.pod.cpu.capacity` + - `k8s.pod.cpu.usage` + - `k8s.pod.filesystem.available` + - `k8s.pod.filesystem.capacity` + - `k8s.pod.filesystem.usage` + - `k8s.pod.memory.available` + - `k8s.pod.memory.major_page_faults` + - `k8s.pod.memory.page_faults` + - `k8s.pod.memory.rss` + - `k8s.pod.memory.usage` + - `k8s.pod.memory.working_set` + - `k8s.pod.network.errors` + - `k8s.pod.network.io` +- From the [k8sclusterreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sclusterreceiver): + - `k8s.pod.phase` + +If `container` metrics are enabled, the following metrics are collected: + +- From the [kubletstatsreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/kubeletstatsreceiver): + - `container.cpu.time` + - `container.cpu.usage` + - `container.filesystem.available` + - `container.filesystem.capacity` + - `container.filesystem.usage` + - `container.memory.available` + - `container.memory.major_page_faults` + - `container.memory.page_faults` + - `container.memory.rss` + - `container.memory.usage` + - `container.memory.working_set` +- From the [k8sclusterreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sclusterreceiver): + - `k8s.container.cpu_request` + - `k8s.container.cpu_limit` + - `k8s.container.memory_request` + - `k8s.container.memory_limit` + - `k8s.container.restarts` + +If `node` metrics are enabled, the following metrics are collected: + +- From the [kubletstatsreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/kubeletstatsreceiver): + - `k8s.node.cpu.usage` + - `k8s.node.filesystem.available` + - `k8s.node.filesystem.capacity` + - `k8s.node.filesystem.usage` + - `k8s.node.memory.available` + - `k8s.node.memory.usage` + - `k8s.node.memory.rss` + - `k8s.node.memory.working_set` + - `k8s.node.network.errors`, + - `k8s.node.network.io`, + +If `volume` metrics are enabled, the following metrics are collected: + +- From the [kubletstatsreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/kubeletstatsreceiver): + - `k8s.volume.available` + - `k8s.volume.capacity` + - `k8s.volume.inodes` + - `k8s.volume.inodes.free` + - `k8s.volume.inodes.used` + +If `deployment` metrics are enabled, the following metrics are collected: + +- From the [k8sclusterreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sclusterreceiver): + - `k8s.deployment.available` + - `k8s.deployment.desired` + +If `daemonset` metrics are enabled, the following metrics are collected: + +- From the [k8sclusterreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sclusterreceiver): + - `k8s.daemonset.current_scheduled_nodes` + - `k8s.daemonset.desired_scheduled_nodes` + - `k8s.daemonset.misscheduled_nodes` + - `k8s.daemonset.ready_nodes` + +If `statefulset` metrics are enabled, the following metrics are collected: + +- From the [k8sclusterreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sclusterreceiver): + - `k8s.statefulset.current_pods` + - `k8s.statefulset.desired_pods` + - `k8s.statefulset.ready_pods` + - `k8s.statefulset.updated_pods` + +If `job` metrics are enabled, the following metrics are collected: + +- From the [k8sclusterreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sclusterreceiver): + - `k8s.job.active_pods` + - `k8s.job.desired_successful_pods` + - `k8s.job.failed_pods` + - `k8s.job.max_parallel_pods` + - `k8s.job.successful_pods` + \ No newline at end of file diff --git a/docs/user/collecting-traces/README.md b/docs/user/collecting-traces/README.md new file mode 100644 index 0000000000..18e1c64ea1 --- /dev/null +++ b/docs/user/collecting-traces/README.md @@ -0,0 +1,61 @@ +# Collecting Traces + +With the Telemetry module, you can collect distributed traces to understand the flow of requests through your applications and infrastructure. To begin collecting traces, you create a TracePipeline resource. It automatically collects OTLP traces and can be configured to collect traces from the Istio service mesh. + +## Overview + +A TracePipeline is a Kubernetes custom resource (CR) that configures trace collection for your cluster. When you create a TracePipeline, it automatically provisions a trace gateway that provides a central OTLP endpoint receiving traces pushed from applications (for details, see [Traces Architecture](./../architecture/traces-architecture.md)). + +The pipeline enriches all collected traces with Kubernetes metadata before sending them to your chosen backend. + +Trace collection is optional. If you don't create a TracePipeline, the trace gateway is not deployed. + +## Prerequisites + +For the recording of a distributed trace, every involved component must propagate at least the trace context. For details, see [Trace Context](https://www.w3.org/TR/trace-context/#problem-statement). + +- In Kyma, all modules involved in users’ requests support the [W3C Trace Context](https://www.w3.org/TR/trace-context) protocol. The involved Kyma modules are, for example, Istio, Serverless, and Eventing. +- Your application also must propagate the W3C Trace Context for any user-related activity. This can be achieved easily using the [Open Telemetry SDKs](https://opentelemetry.io/docs/instrumentation/) available for all common programming languages. If your application propagates the W3C Trace Context and is part of the Istio service mesh, Istio automatically generates its own spans for the traffic entering and leaving your application. +- Furthermore, your application must enrich a trace with additional span data and send this data to the cluster-central telemetry services. You can achieve this with [Open Telemetry SDKs](https://opentelemetry.io/docs/instrumentation/). + +With the default configuration, the trace gateway collects push-based OTLP traces of any container running in Kyma runtime, and the data is shipped to your backend. + +## Minimal TracePipeline + +For a minimal setup, you only need to create a TracePipeline that specifies your backend destination (see [Integrate With Your OTLP Backend](./../integrate-otlp-backend/README.md)): + +```yaml +apiVersion: telemetry.kyma-project.io/v1alpha1 +kind: TracePipeline +metadata: + name: backend +output: + otlp: + endpoint: + value: http://myEndpoint:4317 +``` + +By default, this minimal pipeline collects push-based OTLP traces of any container running in Kyma runtime. + +It activates cluster-internal endpoints to receive traces in the OTLP format. Applications can push traces directly to these URLs: + +- gRPC: `http://telemetry-otlp-traces.kyma-system:4317` +- HTTP: `http://telemetry-otlp-traces.kyma-system:4318` + +## Configure Trace Collection + +You can adjust the TracePipeline using runtime configuration with the available parameters and attributes (see [TracePipeline: Custom Resource Parameters](https://kyma-project.io/#/telemetry-manager/user/resources/04-tracepipeline?id=custom-resource-parameters)). + +- If you use Istio, activate Istio tracing. For details, see [Configure Istio Tracing](istio-support.md). You can adjust which percentage of the trace data is collected. +- The Serverless module integrates the [OpenTelemetry SDK](https://opentelemetry.io/docs/specs/otel/metrics/sdk/) by default. It automatically propagates the trace context for chained calls and reports custom spans for incoming and outgoing requests. You can add more spans within your Function's source code. For details, see [Customize Function Traces](https://kyma-project.io/#/serverless-manager/user/tutorials/01-100-customize-function-traces). +- The Eventing module uses the CloudEvents protocol, which natively supports [W3C Trace Context](https://www.w3.org/TR/trace-context/) propagation. It ensures that the trace context is passed along but doesn't enrich a trace with more advanced span data. + +## Limitations + +- **Throughput**: Assuming an average span with 40 attributes with 64 characters, the maximum throughput is 4200 span/sec ~= 15.000.000 spans/hour. If this limit is exceeded, spans are refused. To increase the maximum throughput, manually scale out the gateway by increasing the number of replicas for the trace gateway (see [Module Configuration and Status](https://kyma-project.io/#/telemetry-manager/user/01-manager?id=module-configuration)). +- **Unavailability of Output**: For up to 5 minutes, a retry for data is attempted when the destination is unavailable. After that, data is dropped. +- **No Guaranteed Delivery**: The used buffers are volatile. If the OTel Collector instance crashes, trace data can be lost. +- **Multiple TracePipeline Support**: The maximum amount of TracePipeline resources is 5. +- **System Span Filtering**: System-related spans reported by Istio are filtered out without the opt-out option, for example: + - Any communication of applications to the Telemetry gateways + - Any communication from the gateways to backends diff --git a/docs/user/collecting-traces/istio-support.md b/docs/user/collecting-traces/istio-support.md new file mode 100644 index 0000000000..6e8e869c7d --- /dev/null +++ b/docs/user/collecting-traces/istio-support.md @@ -0,0 +1,88 @@ +# Configure Istio Tracing + +To get an end-to-end view of requests in your service mesh, configure Istio to send trace data. The TracePipeline receives these traces through its default OTLP input. You can adjust the sampling rate and apply settings to specific namespaces or workloads. + +## Prerequisites + +- You have the Istio module in your cluster. See [Quick Install](https://kyma-project.io/#/02-get-started/01-quick-install). +- You have access to Kyma dashboard. Alternatively, if you prefer CLI, you need [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl). + +## Context + +By default, Istio traces are disabled because they can generate a high volume of data. To collect them, you create an [Istio](https://istio.io/latest/docs/reference/config/telemetry/) `Telemetry` resource in the `istio-system` namespace. When you enable this feature, the Istio proxy sidecars automatically propagate the trace context and report spans for traffic between your services. + +The Istio module provides a preconfigured [extension provider](https://istio.io/latest/docs/tasks/observability/telemetry/) called `kyma-traces` to send this data to the Telemetry module's trace gateway. + +Istio plays a key role in distributed tracing. Its [Ingress Gateway](https://istio.io/latest/docs/tasks/traffic-management/ingress/ingress-control/) is typically where external requests enter your cluster. If a request doesn't have a trace context, Istio adds it. Furthermore, every component within the Istio service mesh runs an Istio proxy, which propagates the trace context and creates span data. When you enable Istio tracing, and it manages trace propagation in your application, you get a complete picture of a trace, because every component automatically contributes span data. Also, Istio tracing is preconfigured to use the vendor-neutral [W3C Trace Context](https://www.w3.org/TR/trace-context/) protocol. + +> [!WARNING] +> Enabling Istio traces can significantly increase data volume and might quickly consume your trace storage. Start with a low sampling rate in production environments. + +## Enable Istio Tracing for the Entire Mesh + +To enable tracing for all workloads in the service mesh, apply an Istio `Telemetry` resource to the `istio-system` namespace. Use this option to establish a baseline configuration for your mesh. + +> [!NOTE] +> You can only have one mesh-wide [Istio](https://istio.io/latest/docs/reference/config/telemetry/) Telemetry resource in the `istio-system` namespace. If you also want to configure Istio access logs, combine both configurations into a single resource (see [Configure Istio Access Logs](./../collecting-logs/istio-support.md)). + +The following example enables tracing with a default sampling rate of 1%: + + ```yaml + apiVersion: telemetry.istio.io/v1 + kind: Telemetry + metadata: + name: mesh-default + namespace: istio-system + spec: + tracing: + - providers: + - name: "kyma-traces" + randomSamplingPercentage: 1.00 + ``` + + +> [!TIP] +> After setting a mesh-wide default in the `istio-system` namespace, you can apply more specific tracing configurations for an entire namespace or for individual workloads within a namespace. This is useful for debugging a particular service by increasing its sampling rate without affecting the entire mesh. For details, see [Filter Traces](../filter-and-process/filter-traces.md). + +## Configure the Sampling Rate + +By default, Istio samples 1% of traces to reduce data volume. To change it, set the `randomSamplingPercentage` value. The sampling decision is propagated within the [trace context](https://www.w3.org/TR/trace-context/#sampled-flag) to ensure that either all or no spans for a given trace are reported. + +> [!NOTE] +> For production environments, a low sampling rate (1–5%) is recommended to manage costs and performance. For development or debugging, you can set it to 100.00 to capture every trace. + +```yaml +apiVersion: telemetry.istio.io/v1 +kind: Telemetry +metadata: + name: mesh-default + namespace: istio-system +spec: + tracing: + - providers: + - name: "kyma-traces" + randomSamplingPercentage: 5.00 # Samples 5% of all traces +``` + +### Propagate Trace Context Without Reporting Spans + +In some cases, you may want Istio to propagate the W3C Trace Context for context-aware logging but not report any trace spans. This approach enriches your access logs with **traceId** and **spanId** without the overhead of full distributed tracing. + +To achieve this, set **randomSamplingPercentage** to `0.00` in your mesh-wide configuration. + +> [!TIP] +> For guidance on Istio access logs, see [Configure Istio Access Logs](../collecting-logs/istio-support.md). + + +```yaml +apiVersion: telemetry.istio.io/v1 +kind: Telemetry +metadata: + name: mesh-default + namespace: istio-system +spec: + tracing: + - providers: + - name: "kyma-traces" + randomSamplingPercentage: 0 +``` diff --git a/docs/user/filter-and-process/README.md b/docs/user/filter-and-process/README.md new file mode 100644 index 0000000000..f2cb337d47 --- /dev/null +++ b/docs/user/filter-and-process/README.md @@ -0,0 +1,16 @@ +# Filtering and Processing Data + +When you configure the inputs for your logs, traces, and metrics, you can choose from which specific Kubernetes resources you want to include or exclude data. Your data is automatically transformed and enriched so you can analyze it in your OTLP backend. + +## Filtering Mechanisms + +The Telemetry module supports the following mechanisms to filter the data, which apply at different stages of the collection process: + +- Filtering within a pipeline (LogPipeline, MetricPipeline): You can configure the `input` section of a pipeline to select or reject data before it is processed by the agent. This is the most common method for filtering application logs, runtime metrics, and Prometheus metrics. +- Configuring the data source (Istio `Telemetry` CRD): For Istio-generated data (access logs and traces), you configure the Istio `Telemetry` resource itself. This controls which workloads generate data and at what volume (sampling rate), before it is even sent to a pipeline. + +## Processing Data + +Application logs from containers are automatically transformed into structured OpenTelemetry (OTLP) log records. + +All pipelines automatically enrich telemetry data with Kubernetes resource attributes, such as Pod name, namespace, and labels. With this context information, you can easily identify the source of telemetry data in your backend. diff --git a/docs/user/filter-and-process/automatic-data-enrichment.md b/docs/user/filter-and-process/automatic-data-enrichment.md new file mode 100644 index 0000000000..d7d2822c0d --- /dev/null +++ b/docs/user/filter-and-process/automatic-data-enrichment.md @@ -0,0 +1,69 @@ +# Automatic Data Enrichment + +The Telemetry gateways automatically enrich your data with OTel resource attributes, so you can easily identify the source of the data in your backend. + +## Service Name + +The service name is the logical name of the service that emits the telemetry data. The gateway ensures that this attribute always has a valid value. + +If you don't provide a service name, or if its value follows the pattern `unknown_service:` as described in the [specification](https://opentelemetry.io/docs/specs/semconv/resource/#service), the gateway generates it from Kubernetes metadata. + +The gateway determines the service name based on the following hierarchy of labels and names: + +1. `app.kubernetes.io/name`: Pod label value +2. `app`: Pod label value +3. Deployment/DaemonSet/StatefulSet/Job name +4. Pod name +5. If none of the above is available, the value is `unknown_service` + +## Kubernetes Metadata + +`k8s.*` attributes encapsulate various pieces of Kubernetes metadata associated with the Pod, such as: + +- `k8s.pod.name`: The Kubernetes Pod name of the Pod that emitted the data. +- `k8s.pod.uid`: The Kubernetes Pod ID of the Pod that emitted the data. +- `k8s..name`: The Kubernetes workload name to which the emitting Pod belongs. Workload is either Deployment, DaemonSet, StatefulSet, Job, or CronJob. +- `k8s.namespace.name`: The Kubernetes namespace name with which the emitting Pod is associated. +- `k8s.cluster.name`: A logical identifier of the cluster, which, by default, is the API Server URL. To set a custom name, configure the `enrichments.cluster.name` field in the Telemetry CRD. +- `k8s.cluster.uid`: A unique identifier of the cluster, realized by the UID of the `kube-system` namespace. +- `k8s.node.name`: The Kubernetes node name to which the emitting Pod is scheduled. +- `k8s.node.uid`: The Kubernetes Node ID to which the emitting Pod belongs. + +## Pod Label Attributes + +In the [Telemetry CRD](https://kyma-project.io/#/telemetry-manager/user/resources/01-telemetry), you can also specify your own enrichments of telemetry data based on Pod labels. + +To capture custom application metadata (for example, for filtering, grouping, or correlation), configure specific label keys or label key prefixes to include in the enrichment process. The gateway adds all matching Pod labels to the telemetry data as resource attributes, using the label key format `k8s.pod.label.`. + +The following example configuration enriches the telemetry data with Pod labels that match the specified keys or key prefixes: + +- `k8s.pod.label.app.kubernetes.io/name`: The value of the exact label key `app.kubernetes.io/name` from the Pod. +- `k8s.pod.label.app.kubernetes.io.*`: All labels that start with the prefix `app.kubernetes.io` from the Pod, where `*` is replaced by the actual label key. + +```yaml +apiVersion: operator.kyma-project.io/v1alpha1 +kind: Telemetry +metadata: + name: default + namespace: kyma-system +spec: + enrichments: + extractPodLabels: + - key: "" # for example, "app.kubernetes.io/name" + - keyPrefix: "" # for example, "app.kubernetes.io" +``` + +## Cloud Provider Attributes + +If data is available, the gateway automatically adds [cloud provider](https://opentelemetry.io/docs/specs/semconv/resource/cloud/) attributes to the telemetry data: + +- `cloud.provider`: Cloud provider name +- `cloud.region`: Region where the Node runs (from Node label `topology.kubernetes.io/region`) +- `cloud.availability_zone`: Zone where the Node runs (from Node label `topology.kubernetes.io/zone`) + +## Host Attributes + +If data is available, the gateway automatically adds [host](https://opentelemetry.io/docs/specs/semconv/resource/host/) attributes to the telemetry data: + +- `host.type`: Machine type of the Node (from Node label `node.kubernetes.io/instance-type`) +- `host.arch`: CPU architecture of the system the Node runs on (from Node label `kubernetes.io/arch`) diff --git a/docs/user/filter-and-process/filter-logs.md b/docs/user/filter-and-process/filter-logs.md new file mode 100644 index 0000000000..1d6ede0352 --- /dev/null +++ b/docs/user/filter-and-process/filter-logs.md @@ -0,0 +1,144 @@ +# Filter Logs + +Filter logs from the OTLP, application, and Istio input to control which data your pipeline processes. You can define filters to include or exclude logs based on their source namespace, container, and other attributes. + +## Overview + +| Source | Granularity | Behavior without 'namespaces' Block | Collect from All Namespaces | Collect from Specific Namespaces | +| :---------- | :------------------------------------------------ | :---------------------------------- | :------------------------------------ | :------------------------------------- | +| OTLP (default) | Namespace | **includes** system namespaces | This is the default, no action needed. | Use the `include` or `exclude` filter | +| Application | Namespace, Container\* | **excludes** system namespaces | Set the **system** attribute to `true` | Use the `include` or `exclude` filter | +| Istio | Namespace, Workload (`selector`), Log content (`filter.expression`) | n/a | Apply the Istio `Telemetry` resource mesh-wide | Apply the Istio `Telemetry` resource to specific namespaces | + +\* The **application** input provides an additional **containers** selector that behaves the same way as the **namespaces** selector. + +## Filter OTLP Logs by Namespaces + +You can filter incoming OTLP logs by namespace. By default, all system namespaces are included. The `include` and `exclude` filters are mutually exclusive. + +- To collect metrics from specific namespaces, use the `include` filter: + + ```yaml + spec: + input: + otlp: + namespaces: + include: + - namespaceA + - namespaceB + ``` + +- To collect OTLP logs from all namespaces **except** specific ones, use the `exclude` filter: + + ```yaml + spec: + input: + otlp: + namespaces: + exclude: + - namespaceA + - namespaceB + ``` + +## Filter Application Logs by Namespace + +You can control which namespaces to collect logs from using `include`, `exclude`, and `system` filters. The `include` and `exclude` filters are mutually exclusive. + +- To collect logs from specific namespaces, use the `include` filter: + + ```yaml + ... + input: + application: + namespaces: + include: + - namespaceA + - namespaceB + ``` + +- To collect logs from all namespaces except specific ones, use the `exclude` filter: + + ```yaml + ... + input: + application: + namespaces: + exclude: + - namespaceC + ``` + +## Collect Application Logs from System Namespaces + +By default, application logs from `kube-system`, `istio-system`, and `kyma-system` are excluded. To override this and collect logs from them, set the **system** attribute to true: + +```yaml + ... + input: + application: + enabled: true + namespaces: + system: true +``` + +## Filter Application Logs by Container + +You can also filter logs based on the container name with `include` and `exclude` filters. These filters apply in addition to any namespace filters. + +The following pipeline collects input from all namespaces excluding `kyma-system` and only from the `istio-proxy` containers: + +```yaml +... + input: + application: + enabled: true + namespaces: + exclude: + - myNamespace + containers: + exclude: + - myContainer + otlp: + ... +``` + + +## Select Istio Logs from a Specific Application + +To limit logging to a single application within a namespace, configure label-based selection for this workload with a [selector](https://istio.io/latest/docs/reference/config/type/workload-selector/#WorkloadSelector) in the Istio `Telemetry` resource. + +```yaml +apiVersion: telemetry.istio.io/v1 +kind: Telemetry +metadata: + name: access-config + namespace: $YOUR_NAMESPACE +spec: + selector: + matchLabels: + service.istio.io/canonical-name: $YOUR_LABEL + accessLogging: + - providers: + - name: kyma-logs +``` + +## Filter Istio Logs by Content + +To reduce noise and cost, filter your Istio access logs. This is especially useful for filtering out traffic that doesn't use an HTTP-based protocol, as those log entries often lack useful details. + +Add a `filter` expression to the same `accessLogging` block that you used to enable the logs. The expression uses [Envoy attributes](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/advanced/attributes) to define which log entries to keep. + +The following example enables mesh-wide logging but only keeps logs that have a defined request protocol, effectively filtering out most non-HTTP traffic. + +```yaml +apiVersion: telemetry.istio.io/v1 +kind: Telemetry +metadata: + name: mesh-default + namespace: istio-system +spec: + accessLogging: + - filter: + expression: 'has(request.protocol)' + providers: + - name: kyma-logs +``` diff --git a/docs/user/filter-and-process/filter-metrics.md b/docs/user/filter-and-process/filter-metrics.md new file mode 100644 index 0000000000..07e6c85a30 --- /dev/null +++ b/docs/user/filter-and-process/filter-metrics.md @@ -0,0 +1,59 @@ +# Filter Metrics + +Filter metrics from the OTLP, Istio, Prometheus, and runtime input to control which data your pipeline processes. You can define filters to include or exclude metrics based on their source namespace and resource type. + +## Overview + +| Source | Granularity | Behavior without 'namespaces' Block | Collect from All Namespaces | Collect from Specific Namespaces | +| :---------- | :------------------------------------------------ | :---------------------------------- | :------------------------------------ | :------------------------------------- | +| OTLP (default) | Namespace | includes system namespaces | This is the default, no action needed. | Use the `include` or `exclude` filter | +| Istio | Namespace | excludes system namespaces | Add `namespaces: {}` to the input's configuration | Use the `include` or `exclude` filter | +| Prometheus | Namespace | excludes system namespaces | Add `namespaces: {}` to the input's configuration | Use the `include` or `exclude` filter | +| Runtime | Namespace, Resource Type\* | excludes system namespaces | Add `namespaces: {}` to the input's configuration | Use the `include` or `exclude` filter | + +\* The **runtime** input provides additional filters for Kubernetes resources such as Pods or Nodes. For details, see [Select Resource Types](../collecting-metrics/runtime-input.md#select-resource-types). + + +## Filter Metrics by Namespace + +For the all inputs (`otlp`, `prometheus`, `istio`, and `runtime`), you can filter incoming metrics by namespace. The `include` and `exclude` filters are mutually exclusive. + +- To collect metrics from specific namespaces, use the `include` filter: + + ```yaml + ... + input: + : + enabled: true + namespaces: + include: + - namespaceA + - namespaceB + ``` + +- To collect metrics from all namespaces **except** specific ones, use the `exclude` filter: + + ```yaml + ... + input: + : + enabled: true + namespaces: + exclude: + - namespaceA + - namespaceB + ``` + +## Collect Metrics From System Namespaces + +For **otlp** metrics, system namespaces are included by default. + +To include system namespaces for **prometheus**, **istio**, and **runtime** metrics without specifying any other namespaces, explicitly configure an empty namespace object: `namespaces: {}`: + +```yaml +... + input: + : + enabled: true + namespaces: {} +``` \ No newline at end of file diff --git a/docs/user/filter-and-process/filter-traces.md b/docs/user/filter-and-process/filter-traces.md new file mode 100644 index 0000000000..7f48827dbf --- /dev/null +++ b/docs/user/filter-and-process/filter-traces.md @@ -0,0 +1,47 @@ +# Filter Traces + +TracePipeline resources have no `input` specification. You can configure Istio trace collection by applying the Istio `Telemetry` resource to specific namespaces. + +## Override Tracing for a Namespace or Workload + +After setting a mesh-wide default, apply more specific tracing configurations for an entire namespace or for individual workloads within a namespace. + +To do this, create an Istio `Telemetry` resource in the workload's namespace. To apply a tracing configuration to a specific workload within the namespace, add a `selector` that matches the workload's labels. + +For example, increase the sampling rate (in this example, to `100.00`) to debug a specific application without affecting the entire mesh: + +```yaml +apiVersion: telemetry.istio.io/v1 +kind: Telemetry +metadata: + name: tracing + namespace: $YOUR_NAMESPACE +spec: + selector: + matchLabels: + app.kubernetes.io/name: "my-app" + tracing: + - providers: + - name: "kyma-traces" + randomSamplingPercentage: 100.00 +``` + +## Disable Tracing for a Specific Workload + +To completely disable Istio span reporting for a specific workload while keeping it enabled for the rest of the mesh, create a `Telemetry` resource that targets the workload and set `disableSpanReporting` to `true`. + +```yaml +apiVersion: telemetry.istio.io/v1 +kind: Telemetry +metadata: + name: $YOUR_APP_NAME-tracing-disable + namespace: $YOUR_NAMESPACE +spec: + selector: + matchLabels: + app.kubernetes.io/name: "$YOUR_APP_NAME" + tracing: + - providers: + - name: "kyma-traces" + disableSpanReporting: true +``` diff --git a/docs/user/filter-and-process/transformation-to-otlp-logs.md b/docs/user/filter-and-process/transformation-to-otlp-logs.md new file mode 100644 index 0000000000..c41d38d8ab --- /dev/null +++ b/docs/user/filter-and-process/transformation-to-otlp-logs.md @@ -0,0 +1,124 @@ +# Transformation to OTLP Logs + +Learn how the log agent processes the original container logs and transforms them into structured OpenTelemetry (OTLP) log records. + +## Original Log Message + +The following example shows a container `myContainer` in Pod `myPod`, running in namespace `myNamespace`, logging to `stdout` with the following JSON message: + +```json +{ + "level": "warn", + "message": "This is the original message", + "tenant": "myTenant", + "traceID": "123" +} +``` + +## Log Tailing + +The log agent reads the log message from a log file managed by the container runtime. The file name contains namespace, Pod, and container information that becomes available later as log attributes. The raw log record looks like the following example: + +```json +{ + "time": "2022-05-23T15:04:52.193317532Z", + "stream": "stdout", + "_p": "F", + "log": "{\"level\": \"warn\",\"message\": \"This is the original message\",\"tenant\": \"myTenant\",\"trace_id\": \"123\"}" +} +``` + +After the tailing, the created OTLP record looks like the following example: + +```json +{ + "time": "2022-05-23T15:04:52.100000000Z", + "observedTime": "2022-05-23T15:04:52.200000000", + "attributes": { + "log.file.path": "/var/log/pods/myNamespace_myPod-/myContainer/.log", + "log.iostream": "stdout" + }, + "resourceAttributes": { + "k8s.container.name": "myContainer", + "k8s.container.restart_count": "", + "k8s.pod.name": "myPod", + "k8s.namespace.name": "myNamespace" + }, + "body": "{\"level\": \"warn\",\"message\": \"This is the original message\",\"tenant\": \"myTenant\",\"trace_id\": \"123\"}" +} +``` + +The agent enriches all information identifying the log source (such as container, Pod, and namespace name) as resource attributes, following [Kubernetes conventions](https://opentelemetry.io/docs/specs/semconv/resource/k8s/). It enriches further metadata, like the original file name and channel, as log attributes, following [log attribute conventions](https://opentelemetry.io/docs/specs/semconv/general/logs/). The agent uses the `time` value from the container runtime's log entry as the **time** attribute in the new OTel record, as it closely matches the actual log event time. Additionally, the agent sets `observedTime` with the time it actually reads the log record, as the OTel log specification recommends. The agent moves the log payload to the OTLP `body` field. + +## JSON Parsing + +If the `body` value is a JSON document, the agent parses the value and enriches all JSON root attributes as additional log attributes. The agent moves the original body into the **log.original** attribute (managed with the LogPipeline attribute `input.application.keepOriginalBody: true`). + +After JSON parsing, the OTLP record looks like the following example: + +```json +{ + "time": "2022-05-23T15:04:52.100000000Z", + "observedTime": "2022-05-23T15:04:52.200000000", + "attributes": { + "log.file.path": "/var/log/pods/myNamespace_myPod-/myContainer/.log", + "log.iostream": "stdout", + "log.original": "{\"level\": \"warn\",\"message\": \"This is the original message\",\"tenant\": \"myTenant\",\"trace_id\": \"123\"}", + "level": "warn", + "tenant": "myTenant", + "trace_id": "123", + "message": "This is the original message" + }, + "resourceAttributes": { + "k8s.container.name": "myContainer", + "k8s.container.restart_count": "", + "k8s.pod.name": "myPod", + "k8s.namespace.name": "myNamespace" + }, + "body": "" +} +``` + +## Severity Parsing + +Typically, a log message includes a log level in the `level` field. Based on this, the agent parses the `level` log attribute with a severity parser. If parsing succeeds, the agent transforms the log attribute into the OTel attributes `severityText` and `severityNumber`. + +## Trace Parsing + +OTLP natively attaches trace context to log records. If possible, the log agent parses the following log attributes according to the [W3C-Tracecontext specification](https://www.w3.org/TR/trace-context/#traceparent-header): + +* `trace_id` +* `span_id` +* `trace_flags` +* `traceparent` + +## Log Body Determination + +Because the original log message typically resides in the **body** attribute, the agent moves a log attribute called **message** (or **msg**) into the body. + +At this point, before further enrichment, the resulting overall log record looks like the following example: + +```json +{ + "time": "2022-05-23T15:04:52.100000000Z", + "observedTime": "2022-05-23T15:04:52.200000000", + "attributes": { + "log.file.path": "/var/log/pods/myNamespace_myPod-/myContainer/.log", + "log.iostream": "stdout", + "log.original": "{\"level\": \"warn\",\"message\": \"This is the original message\",\"tenant\": \"myTenant\",\"trace_id\": \"123\"}", + "tenant": "myTenant", + }, + "resourceAttributes": { + "k8s.container.name": "myContainer", + "k8s.container.restart_count": "", + "k8s.pod.name": "myPod", + "k8s.namespace.name": "myNamespace" + }, + "body": "This is the original message", + "severityNumber": 13, + "severityText": "warn", + "trace_id": 123 +} +``` + +This structured record is now ready to be shipped to your observability backend. For details on further enrichment, see [Automatic Data Enrichment](automatic-data-enrichment.md). diff --git a/docs/user/gateways.md b/docs/user/gateways.md index 819f5220ec..81b1d6a189 100644 --- a/docs/user/gateways.md +++ b/docs/user/gateways.md @@ -1,107 +1,3 @@ # Telemetry Gateways -The Telemetry gateways in Kyma take care of data enrichment, filtering, and dispatching, as well as native support for Istio communication. - -## Features - -The traces, metrics, and (OTLP-based) logs features are based on a gateway, which is provisioned as soon as you define any pipeline resource. All telemetry data of the related domain passes the gateway, so it acts as a central point and provides the following benefits: - -- [Data Enrichment](#data-enrichment) to achieve a certain data quality -- Filtering to apply namespace filtering and remove noisy system data (individually for logs, traces, and metrics) -- Dispatching to the configured backends (individually for logs, traces, and metrics) - -When the Istio module is added to your Kyma cluster, the gateways support mTLS for the communication from the workload to the gateway, as well as for communication to backends running in the cluster. For details, see [Istio Support](#istio-support). - -The gateways are based on the [OTel Collector](https://opentelemetry.io/docs/collector/) and come with a concept of pipelines consisting of receivers, processors, and exporters, with which you can flexibly plug pipelines together (see [Configuration](https://opentelemetry.io/docs/collector/configuration/)). Kyma's MetricPipeline provides a hardened setup of an OTel Collector and also abstracts the underlying pipeline concept. Such abstraction has the following benefits: - -- Compatibility: An abstraction layer supports compatibility when underlying features change. -- Migratability: Smooth migration experiences when switching underlying technologies or architectures. -- Native Kubernetes support: API provided by Kyma supports an easy integration with Secrets, for example, served by the [SAP BTP Service Operator](https://github.com/SAP/sap-btp-service-operator#readme). Telemetry Manager takes care of the full lifecycle. -- Focus: The user doesn't need to understand the underlying concepts. - -## Usage - -You can set up a pipeline with a backend that subsequently instantiates a gateway. For details, see [Logs](logs.md) [Traces](03-traces.md) and [Metrics](04-metrics.md). -To see whether you've set up your gateways and their push endpoints successfully, check the status of the default Telemetry resource: - -```sh -kubectl -n kyma-system get telemetries.operator.kyma-project.io default -oyaml -``` - -In the status of the returned resource, you see the pipeline health as well as the available push endpoints: - -```yaml - endpoints: - logs: - grpc: http://telemetry-otlp-logs.kyma-system:4317 - http: http://telemetry-otlp-logs.kyma-system:4318 - metrics: - grpc: http://telemetry-otlp-metrics.kyma-system:4317 - http: http://telemetry-otlp-metrics.kyma-system:4318 - traces: - grpc: http://telemetry-otlp-traces.kyma-system:4317 - http: http://telemetry-otlp-traces.kyma-system:4318 -``` - -For every signal type, there's a dedicated endpoint to which you can push data using [OTLP](https://opentelemetry.io/docs/specs/otel/protocol/). OTLP supports GRPC and HTTP-based communication, each having its individual port on every endpoint. Use port `4317` for GRPC and `4318` for HTTP. - -![Gateways-Plain](assets/gateways-plain.drawio.svg) - -Applications that support OTLP typically use the [OTel SDK](https://opentelemetry.io/docs/languages/) for instrumentation of the data. You can either configure the endpoints hardcoded in the SDK setup, or you use standard [environment variables](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/#otel_exporter_otlp_traces_endpoint) configuring the OTel exporter, for example: - -- Traces GRPC: `export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://telemetry-otlp-traces.kyma-system:4317"` -- Traces HTTP: `export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://telemetry-otlp-traces.kyma-system:4318/v1/traces"` -- Metrics GRPC: `export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT="http://telemetry-otlp-metrics.kyma-system:4317"` -- Metrics HTTP: `export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT="http://telemetry-otlp-metrics.kyma-system:4318/v1/metrics"` - -## Data Enrichment - -The Telemetry gateways automatically enrich your data by adding the following attributes: - -- `service.name`: The logical name of the service that emits the telemetry data. The gateway ensures that this attribute always has a valid value. - If not provided by the user, or if its value follows the pattern `unknown_service:` as described in the [specification](https://opentelemetry.io/docs/specs/semconv/resource/#service), then it is generated from Kubernetes metadata. The gateway determines the service name based on the following hierarchy of labels and names: - 1. `app.kubernetes.io/name`: Pod label value - 2. `app`: Pod label value - 3. Deployment/DaemonSet/StatefulSet/Job name - 4. Pod name - 5. If none of the above is available, the value is `unknown_service` -- `k8s.pod.name`: The Kubernetes Pod name of the Pod that emitted the data -- `k8s.pod.uid`: The Kubernetes Pod id of the Pod that emitted the data -- `k8s..name`: The Kubernetes workload name to which the emitting Pod belongs. Workload is either Deployment, DaemonSet, StatefulSet, Job or CronJob -- `k8s.namespace.name`: The Kubernetes namespace name with which the emitting Pod is associated -- `k8s.cluster.name`: A logical identifier of the cluster, which, by default, is the API Server URL. Users can set a custom name by configuring the `enrichments.cluster.name` field in the [Telemetry CRD](resources/01-telemetry.md) -- `k8s.cluster.uid`: A unique identifier of the cluster, realized by the UID of the "kube-system" namespace -- `k8s.node.name`: The Kubernetes node name to which the emitting Pod is scheduled. -- `k8s.node.uid`: The Kubernetes Node id to which the emitting Pod belongs -- `k8s.pod.label.` attributes: In addition to the predefined enrichments, the Telemetry gateways support user-defined enrichments of telemetry data based on Pod labels (see [Telemetry CRD](resources/01-telemetry.md)). By configuring specific label keys or label key prefixes to include in the enrichment process, you can capture custom application metadata that may be relevant for filtering, grouping, or correlation purposes. All matching Pod labels are added to the telemetry data as resource attributes, using the label key format `k8s.pod.label.`. - - The following example configuration enriches the telemetry data with Pod labels that match the specified keys or key prefixes: - - `k8s.pod.label.app.kubernetes.io/name`: The value of the exact label key `app.kubernetes.io/name` from the Pod. - - `k8s.pod.label.app.kubernetes.io.*`: All labels that start with the prefix `app.kubernetes.io` from the Pod, where `*` is replaced by the actual label key. - ```yaml - apiVersion: operator.kyma-project.io/v1alpha1 - kind: Telemetry - metadata: - name: default - namespace: kyma-system - spec: - enrichments: - extractPodLabels: - - key: "" # for example, "app.kubernetes.io/name" - - keyPrefix: "" # for example, "app.kubernetes.io" - ``` -- Cloud provider attributes: If data is available, the gateway automatically adds [cloud provider](https://opentelemetry.io/docs/specs/semconv/resource/cloud/) attributes to the telemetry data. - - `cloud.provider`: Cloud provider name - - `cloud.region`: Region where the Node runs (from Node label `topology.kubernetes.io/region`) - - `cloud.availability_zone`: Zone where the Node runs (from Node label `topology.kubernetes.io/zone`) -- Host attributes: If data is available, the gateway automatically adds [host](https://opentelemetry.io/docs/specs/semconv/resource/host/) attributes to the telemetry data: - - `host.type`: Machine type of the Node (from Node label `node.kubernetes.io/instance-type`) - - `host.arch`: CPU architecture of the system the Node is running on (from Node label `kubernetes.io/arch`) - -## Istio Support - -The Telemetry module automatically detects whether the Istio module is added to your cluster, and injects Istio sidecars to the Telemetry components. Additionally, the ingestion endpoints of gateways are configured to allow traffic in the permissive mode, so they accept mTLS-based communication as well as plain text. - -![Gateways-Istio](assets/gateways-istio.drawio.svg) - -Clients in the Istio service mesh transparently communicate to the gateway with mTLS. Clients that don't use Istio can communicate with the gateway in plain text mode. The same pattern applies for the communication to the backends running in the cluster. External clusters use the configuration as specified in the pipelines output section. +This content moved to [Telemetry Pipeline API](./pipelines.md). diff --git a/docs/user/integrate-otlp-backend/README.md b/docs/user/integrate-otlp-backend/README.md new file mode 100644 index 0000000000..21d71f667f --- /dev/null +++ b/docs/user/integrate-otlp-backend/README.md @@ -0,0 +1,139 @@ +# Integrate With Your OTLP Backend + +Define the backend destination for your telemetry data by configuring the OTLP output, including the protocol and authentication method. + +## Overview + +In every pipeline, you must configure an `output` section, which defines the destination for your telemetry data using OTLP. This is where you specify your chosen observability backend (see [OpenTelemetry: Vendors](https://opentelemetry.io/ecosystem/vendors/)). + +![OTLP-Output](./../assets/otlp-output.drawio.svg) + +> [!NOTE] +> Each pipeline resource supports exactly one backend. However, you can send specific inputs to different backends by setting up designated pipelines. For details, see [Route Specific Inputs to Different Backends](../otlp-input.md#route-specific-inputs-to-different-backends). + +## Specify the OTLP Endpoint + +For the minimal pipeline configuration, you only specify an OTLP endpoint; though it's recommended that you use authentication. + +```yaml +... + output: + otlp: + endpoint: + value: https://backend.example.com:4317 +``` + +## Choose a Protocol + +The default protocol for shipping the data to a backend is gRPC. If your backend requires the HTTP protocol instead, set the **protocol** attribute to `http`. Based on this setting, the gateway chooses the exporter: `otlp` for gRPC (the default) or `otlphttp` for HTTP. + +Ensure the port in your endpoint URL is correct for the chosen protocol. + +```yaml + ... + output: + otlp: + protocol: http + endpoint: + value: https://backend.example.com:4318 +``` + +## Set Up Authentication + +For each pipeline, add authentication details (like user names, passwords, certificates, or tokens) to connect securely to your observability backend. You can use mutual TLS (mTLS), custom headers, or Basic Authentication. + +While you can choose to add your authentication details from plain text, it’s recommended to store these sensitive details in a Kubernetes `Secret` and reference the Secret's keys in your pipeline configuration. When you rotate the `Secret` and update its values, Telemetry Manager detects the changes and applies the new `Secret` to your setup. + +> [!TIP] +> If you use a Secret owned by the [SAP BTP Service Operator](https://github.com/SAP/sap-btp-service-operator), you can configure an automated rotation policy with a specific rotation frequency and don’t have to intervene manually. + +- To use client certificates for mTLS, configure the `tls` section with your public certificate and private key. + + ```yaml + ... + output: + otlp: + endpoint: + value: https://backend.example.com/otlp:4317 + tls: + cert: + valueFrom: + secretKeyRef: + name: backend + namespace: default + key: cert + key: + valueFrom: + secretKeyRef: + name: backend + namespace: default + key: key + ``` + +- To send an authentication token (such as a bearer token) in an HTTP header, configure the `headers` section. + + ```yaml + ... + output: + otlp: + endpoint: + value: https://backend.example.com:4317 + headers: + - name: Authorization + prefix: Bearer + valueFrom: + secretKeyRef: + name: backend + namespace: default + key: token + ``` + +- To use a username and password for authentication, configure the `authentication.basic` section. + + ```yaml + ... + output: + otlp: + endpoint: + valueFrom: + secretKeyRef: + name: backend + namespace: default + key: endpoint + authentication: + basic: + user: + valueFrom: + secretKeyRef: + name: backend + namespace: default + key: user + password: + valueFrom: + secretKeyRef: + name: backend + namespace: default + key: password + ``` + +- If you want to configure authentication details from plain text, use the following pattern. The example shows mTLS, but you can also use Basic Authentication or custom headers: + + ```yaml + ... + output: + otlp: + endpoint: + value: https://backend.example.com/otlp:4317 + tls: + cert: + value: | + -----BEGIN CERTIFICATE----- + ... + key: + value: | + -----BEGIN RSA PRIVATE KEY----- + ... + ``` + +> [!NOTE] +> If your backend is running inside the cluster and is part of the Istio service mesh, the gateways automatically secure the connection and you don't have to configure the `authentication` block. For details, see [Sending Data to In-Cluster Backends](../architecture/istio-integration.md#sending-data-to-in-cluster-backends). diff --git a/docs/user/integrate-otlp-backend/migration-to-otlp-logs.md b/docs/user/integrate-otlp-backend/migration-to-otlp-logs.md new file mode 100644 index 0000000000..23a890e201 --- /dev/null +++ b/docs/user/integrate-otlp-backend/migration-to-otlp-logs.md @@ -0,0 +1,112 @@ +# Migrate Your LogPipeline From HTTP to OTLP + +To use the OpenTelemetry Protocol (OTLP) for sending logs, you must migrate your LogPipeline from the `http` or `custom` output to the `otlp` output. With OTLP, you can correlate logs with traces and metrics, collect logs pushed directly from applications, and use features available only for the OTLP-based stack. + +## Prerequisites + +* You have an active Kyma cluster with the Telemetry module added. +* You have one or more LogPipeline resources that use the `http` or `custom` output. +* Your observability backend has an OTLP ingestion endpoint. + If your backend doesn't support OTLP natively, you must run a custom OTel Collector as gateway between the Telemetry module and the target backend. + +## Context + +When you want to migrate to the `otlp` output, create a new LogPipeline. To prevent data loss, run it in parallel with your existing pipeline. After verifying that the new pipeline works correctly, you can delete the old one. + +You can't modify an existing LogPipeline to change its output type. You must create a new resource. + +## Procedure + +1. Create a new LogPipeline that uses the `otlp` output. + + Pay special attention to the following settings (for details, see [Integrate With Your OTLP Backend](migration-to-otlp-logs.md)): + + * Endpoint URL: Use the OTLP-specific ingestion endpoint from your observability backend. This URL is different from the one used for the legacy `http` output. + * Protocol: The `otlp` output defaults to the gRPC protocol. If your backend uses HTTP, you must include the protocol in the endpoint URL (for example, https://my-otlp-http-endpoint:4318). + * Authentication: The OTLP endpoint often uses different credentials or API permissions than your previous log ingestion endpoint. Verify that your credentials have the necessary permissions for OTLP log ingestion. + + ```yaml + apiVersion: telemetry.kyma-project.io/v1alpha1 + kind: LogPipeline + metadata: + name: my-otlp-pipeline + spec: + output: + otlp: + endpoint: + value: "my-backend:4317" + ``` + +2. (Optional) If your old pipeline uses `custom` filters, rewrite them using the OpenTelemetry Transformation Language ([OTTL](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/README.md)) and add them to your new LogPipeline. + + Example: You want to replace a legacy Fluent Bit filter that dropped health checks and added a **tenant** attribute: + + ```yaml + apiVersion: telemetry.kyma-project.io/v1alpha1 + kind: LogPipeline + metadata: + name: my-http-pipeline + spec: + filter: + - custom: | + Name grep + Exclude path /healthz/ready + - custom: | + Name record_modifier + Record tenant myTenant + output: + http: + ... + ``` + + In your new OTLP pipeline, use the `filter` and `transform` sections with OTTL expressions: + + ```yaml + apiVersion: telemetry.kyma-project.io/v1alpha1 + kind: LogPipeline + metadata: + name: my-http-pipeline + spec: + transform: + - conditions: + - log.attributes["tenant"] == "" + statements: + - set(log.attributes["tenant"], "myTenant") + filter: + conditions: + - log.attributes["path"] == "/healthz/ready" + output: + otlp: + ... + ``` + +3. (Optional) To enrich logs with Pod labels, configure the central Telemetry resource ([Telemetry CRD](https://kyma-project.io/#/telemetry-manager/user/resources/01-telemetry)). + + In contrast to a Fluent Bit LogPipeline, the `otlp` output doesn't automatically add all Pod labels. To continue enriching logs with specific labels, you must explicitly enable it in the spec.enrichments.extractPodLabels field. + + > [!NOTE] + > Enrichment with Pod annotations is no longer supported. + +4. Deploy the new LogPipeline: + + ```shell + kubectl apply -f logpipeline.yaml + ``` + +5. Check that the new LogPipeline is healthy: + + ```shell + kubectl get logpipeline my-otlp-pipeline + ``` + +6. Check your observability backend to confirm that log data is arriving. + +7. Delete the old LogPipeline: + + ```shell + kubectl delete logpipeline my-old-pipeline + ``` + +## Result + +Your cluster now sends logs exclusively through your new OTLP-based LogPipeline. Your filter and enrichment logic is preserved. diff --git a/docs/user/integration/assets/cloudwatch.drawio.svg b/docs/user/integration/assets/cloudwatch.drawio.svg index 4d5f6712fe..72b949d620 100644 --- a/docs/user/integration/assets/cloudwatch.drawio.svg +++ b/docs/user/integration/assets/cloudwatch.drawio.svg @@ -1,4 +1,4 @@ - + @@ -139,13 +139,13 @@
-
+
OTLP Metrics
- + OTLP Metrics @@ -161,20 +161,20 @@
-
+
OTLP Traces
- + OTLP Traces - + @@ -182,7 +182,7 @@
-
+
OTLP Metrics @@ -190,7 +190,7 @@
- + OTLP Metrics @@ -207,20 +207,20 @@
-
+
Tail Log Files
- + Tail Log Files - + @@ -252,7 +252,7 @@
-
+
Pull prometheus
metrics @@ -260,7 +260,7 @@
- + Pull prometheus... @@ -354,13 +354,13 @@
-
+
OTLP Traces
- + OTLP Traces @@ -376,13 +376,13 @@
-
+
OTLP Logs
- + OTLP Logs @@ -419,13 +419,13 @@
-
+
OTLP Logs
- + OTLP Logs @@ -483,13 +483,13 @@
-
+
OTLP segv4
- + OTLP segv4 @@ -505,13 +505,13 @@
-
+
OTLP segv4
- + OTLP segv4 @@ -527,13 +527,13 @@
-
+
EMF
- + EMF diff --git a/docs/user/integration/assets/dynatrace.drawio.svg b/docs/user/integration/assets/dynatrace.drawio.svg index 5976b387ac..0ceb6f92eb 100644 --- a/docs/user/integration/assets/dynatrace.drawio.svg +++ b/docs/user/integration/assets/dynatrace.drawio.svg @@ -1,4 +1,4 @@ - + @@ -38,36 +38,23 @@ -
+
- - Custom Namespace -
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+ + + + Custom Namespace + +
- - Custom Namespace... + + Custom Namespace @@ -79,35 +66,23 @@ -
+
- - Custom Namespace -
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+ + + + Custom Namespace + +
- - Custom Namespace... + + Custom Namespace @@ -173,17 +148,17 @@ -
+
-
- +
+ OTLP Traces
- + OTLP Traces @@ -197,15 +172,15 @@ -
+
-
+
OTLP Traces
- + OTLP Traces @@ -221,20 +196,20 @@
-
+
OTLP Metrics
- + OTLP Metrics - + @@ -273,15 +248,15 @@
-
- +
+ OTLP Metrics delta
- + OTLP Metrics delta @@ -298,13 +273,13 @@
-
+
Auto-instrumented data
- + Auto-instrumented data @@ -317,7 +292,7 @@ -
+
auto-Instrumentation @@ -325,14 +300,14 @@
- + auto-Instrumentation - + @@ -383,15 +358,15 @@ -
+
-
+
OTLP
- + OTLP @@ -428,15 +403,15 @@ -
+
-
+
manages
- + manages @@ -452,13 +427,13 @@
-
+
manages
- + manages @@ -518,29 +493,29 @@
-
+
Custom Metrics & Traces
- + Custom Metrics & Traces - + -
+
-
+
Istio & Prometheus
Metrics @@ -548,15 +523,15 @@
- + Istio & Prometheus... - - + + @@ -564,8 +539,8 @@
-
- +
+ OTLP Metrics
cumulative @@ -574,7 +549,7 @@
- + OTLP Metrics... diff --git a/docs/user/integration/assets/jaeger.drawio.svg b/docs/user/integration/assets/jaeger.drawio.svg index 91b4539469..d949ded550 100644 --- a/docs/user/integration/assets/jaeger.drawio.svg +++ b/docs/user/integration/assets/jaeger.drawio.svg @@ -1,4 +1,4 @@ - + @@ -14,7 +14,7 @@
-
+
Kyma Cluster @@ -25,7 +25,7 @@
- + Kyma Cluster @@ -39,17 +39,21 @@
-
-
- - Custom Namespace +
+
+ + + + Custom Namespace + +
- + Custom Namespace @@ -92,20 +96,20 @@
-
+
OTLP Traces
- + OTLP Traces - + @@ -113,7 +117,7 @@
-
+
OTLP Traces @@ -121,7 +125,7 @@
- + OTLP Traces @@ -136,17 +140,21 @@
-
-
- - Custom Namespace +
+
+ + + + Custom Namespace + +
- + Custom Namespace @@ -174,7 +182,7 @@ - + diff --git a/docs/user/integration/assets/loki.drawio.svg b/docs/user/integration/assets/loki.drawio.svg index 86ca09ad67..9f2365b2bf 100644 --- a/docs/user/integration/assets/loki.drawio.svg +++ b/docs/user/integration/assets/loki.drawio.svg @@ -1,4 +1,4 @@ - + @@ -41,8 +41,12 @@
- - Custom Namespace + + + + Custom Namespace + +
@@ -92,20 +96,20 @@
-
+
Tail Logs from stdout
- + Tail Logs from stdout - + @@ -113,7 +117,7 @@
-
+
OTLP Logs @@ -121,7 +125,7 @@
- + OTLP Logs @@ -138,8 +142,12 @@
- - Custom Namespace + + + + Custom Namespace + +
@@ -174,7 +182,7 @@ - + @@ -210,13 +218,13 @@
-
+
Push OTLP Logs
- + Push OTLP Logs diff --git a/docs/user/integration/assets/otel-demo.drawio.svg b/docs/user/integration/assets/otel-demo.drawio.svg index 2ec10a3bf5..f4425d1291 100644 --- a/docs/user/integration/assets/otel-demo.drawio.svg +++ b/docs/user/integration/assets/otel-demo.drawio.svg @@ -1,4 +1,4 @@ - + @@ -41,8 +41,12 @@
- - Custom Namespace + + + + Custom Namespace + +
@@ -83,11 +87,11 @@ - + - + @@ -117,13 +121,13 @@
-
+
OTLP Metrics
- + OTLP Metrics @@ -139,13 +143,13 @@
-
+
OTLP Traces
- + OTLP Traces diff --git a/docs/user/integration/assets/prometheus.drawio.svg b/docs/user/integration/assets/prometheus.drawio.svg index d751a9a957..caadf2d7a8 100644 --- a/docs/user/integration/assets/prometheus.drawio.svg +++ b/docs/user/integration/assets/prometheus.drawio.svg @@ -1,4 +1,4 @@ - + @@ -39,17 +39,21 @@
-
-
- - Custom Namespace +
+
+ + + + Custom Namespace + +
- + Custom Namespace @@ -63,17 +67,21 @@
-
-
- - Custom Namespace +
+
+ + + + Custom Namespace + +
- + Custom Namespace @@ -150,20 +158,20 @@
-
+
Custom OTLP Metrics
- + Custom OTLP Metrics - + @@ -176,20 +184,20 @@
-
+
OTLP Metrics
- + OTLP Metrics - + @@ -221,13 +229,13 @@
-
+
Istio OTLP Metrics
- + Istio OTLP Metrics diff --git a/docs/user/integration/assets/sample-app.drawio.svg b/docs/user/integration/assets/sample-app.drawio.svg index e3f6d90a5c..797ceb7af0 100644 --- a/docs/user/integration/assets/sample-app.drawio.svg +++ b/docs/user/integration/assets/sample-app.drawio.svg @@ -1,4 +1,4 @@ - + @@ -41,8 +41,12 @@
- - Custom Namespace + + + + Custom Namespace + +
@@ -83,11 +87,11 @@ - + - + @@ -117,13 +121,13 @@
-
+
OTLP Metrics
- + OTLP Metrics @@ -139,13 +143,13 @@
-
+
OTLP Traces
- + OTLP Traces diff --git a/docs/user/integration/assets/sap-cloud-logging.drawio.svg b/docs/user/integration/assets/sap-cloud-logging.drawio.svg index def4c62b99..178a0dd265 100644 --- a/docs/user/integration/assets/sap-cloud-logging.drawio.svg +++ b/docs/user/integration/assets/sap-cloud-logging.drawio.svg @@ -1,20 +1,20 @@ - + - + - + -
+
-
+
Kyma Cluster @@ -25,293 +25,267 @@
- + Kyma Cluster - + -
-
-
- - Custom Namespace +
+
+
+ + + + User Namespace + +
- - Custom Namespace + + User Namespace + - + -
+
-
- - Kyma -
- Telemetry Module -
- (kyma-system) +
+ + + SAP Cloud Logging +
+
- - Kyma... + + SAP Cloud Logging - - + -
+
-
- OTLP Traces +
+ + Alerts and Dashboards +
- - OTLP Traces + + Alerts and Dashboards - - + + -
+
-
- OTLP Metrics +
+ push OTLP data +
+
+ (logs, traces, metrics) +
- - OTLP Metrics + + push OTLP data... - + -
+
-
- - OTLP Traces - +
+ OTLP data +
+ (logs, metrics, traces) +
- - OTLP Traces + + OTLP data... - - + + -
+
-
- Tail Log Files +
+ pull Prometheus +
+ metrics +
- - Tail Log Files + + pull Prometheus... - + -
+
- Custom application + Application
- (with Istio Sidecar) + with +
+ Istio sidecar +
- - Custom application... + + Application... - - - - - - - -
-
-
- Pull prometheus -
- metrics -
-
-
- - - Pull prometheus... - - - + + - + + -
+
-
- - - SAP Cloud Logging -
- - +
+ HTTP logs
- - SAP Cloud Logging + + HTTP logs - + + -
+
-
- - OTLP - +
+ tail log files
- - OTLP + + tail log files - - - - - - + -
-
-
- OTLP Metrics -
-
-
- - - OTLP Metrics - - - - - - - - - - - - -
-
-
- HTTP Logs +
+
+
+ + Kyma Telemetry + +
+ + Module +
+ (kyma-system) +
+ +
- - HTTP Logs + + Kyma Telemetry... diff --git a/docs/user/integration/aws-cloudwatch/README.md b/docs/user/integration/aws-cloudwatch/README.md index 338debd4c4..85cfa441f0 100644 --- a/docs/user/integration/aws-cloudwatch/README.md +++ b/docs/user/integration/aws-cloudwatch/README.md @@ -99,7 +99,7 @@ The previous command uses the [values.yaml](https://raw.githubusercontent.com/ky Use the Kyma Telemetry module to enable ingestion of the signals from your workloads: -1. Deploy a [LogPipeline](./../../02-logs.md): +1. Deploy a [LogPipeline](./../../collecting-logs/README.md): ```bash kubectl apply -f - < -- If you want to use Istio access logs, make sure that the [Istio module](https://kyma-project.io/#/istio/user/README) is added. -- An instance of [SAP Cloud Logging](https://help.sap.com/docs/cloud-logging?locale=en-US&version=Cloud) with OpenTelemetry enabled to ingest distributed traces. +- Kyma as the target deployment environment, with the following modules added (see [Quick Install](https://kyma-project.io/#/02-get-started/01-quick-install)): + - Telemetry module + - To collect data from your Istio service mesh: Istio module (default module) + - SAP BTP Operator module (default module) +- An instance of [SAP Cloud Logging](https://help.sap.com/docs/cloud-logging?locale=en-US&version=Cloud) with OpenTelemetry ingestion enabled. For details, see [Ingest via OpenTelemetry API Endpoint](https://help.sap.com/docs/SAP_CLOUD_LOGGING/d82d23dc499c44079e1e779c1d3a5191/fdc78af7c69246bc87315d90a061b321.html?locale=en-US). > [!TIP] - > Create the instance with the SAP BTP service operator (see [Create an SAP Cloud Logging Instance through SAP BTP Service Operator](https://help.sap.com/docs/cloud-logging/cloud-logging/create-sap-cloud-logging-instance-through-sap-btp-service-operator?locale=en-US&version=Cloud)), because it takes care of creation and rotation of the required Secret. However, you can choose any other method of creating the instance and the Secret, as long as the parameter for OTLP ingestion is enabled in the instance. For details, see [Configuration Parameters](https://help.sap.com/docs/cloud-logging/cloud-logging/configuration-parameters?locale=en-US&version=Cloud). -- A Secret in the respective namespace in the Kyma cluster, holding the credentials and endpoints for the instance. In the following example, the Secret is named `sap-cloud-logging` and the namespace `sap-cloud-logging-integration`, as illustrated in the [secret-example.yaml](https://github.com/kyma-project/telemetry-manager/blob/main/docs/user/integration/sap-cloud-logging/secret-example.yaml). + > Create the SAP Cloud Logging instance with the SAP BTP service operator (see [Create an SAP Cloud Logging Instance through SAP BTP Service Operator](https://help.sap.com/docs/cloud-logging/cloud-logging/create-sap-cloud-logging-instance-through-sap-btp-service-operator?locale=en-US&version=Cloud)), because it takes care of creation and rotation of the required Secret. However, you can choose any other method of creating the instance and the Secret, as long as the parameter for OTLP ingestion is enabled in the instance. For details, see [Configuration Parameters](https://help.sap.com/docs/cloud-logging/cloud-logging/configuration-parameters?locale=en-US&version=Cloud). +- A Secret in the respective namespace in your Kyma cluster, holding the credentials and endpoints for the instance. It’s recommended that you rotate your Secret (see [SAP BTP Security Recommendation BTP-CLS-0003](https://help.sap.com/docs/btp/sap-btp-security-recommendations-c8a9bb59fe624f0981efa0eff2497d7d/sap-btp-security-recommendations?seclist-index=BTP-CLS-0003&version=Cloud&locale=en-US)). In the following example, the Secret is named `sap-cloud-logging` and the namespace `sap-cloud-logging-integration`, as illustrated in the [secret-example.yaml](https://github.com/kyma-project/telemetry-manager/blob/main/docs/user/integration/sap-cloud-logging/secret-example.yaml). - Kubernetes CLI (kubectl) (see [Install the Kubernetes Command Line Tool](https://developers.sap.com/tutorials/cp-kyma-download-cli.html)). @@ -39,7 +37,9 @@ Learn how to configure the Telemetry module to ingest application and access log ## Context -The Telemetry module supports shipping logs and ingesting distributed traces as well as metrics from applications and the Istio service mesh to SAP Cloud Logging. Furthermore, you can set up Kyma dashboard integration and use SAP Cloud Logging alerts and dashboards. +The Telemetry module supports shipping logs and ingesting distributed traces as well as metrics from applications and the Istio service mesh to SAP Cloud Logging. + +First, set up the Telemetry module to ship the logs, traces, and metrics to your backend by deploying the pipelines and other required resources. Then, you configure Kyma dashboard integration. Finally, set up SAP Cloud Logging alerts and dashboards. SAP Cloud Logging is an instance-based and environment-agnostic observability service to store, visualize, and analyze logs, metrics, and traces. @@ -47,149 +47,191 @@ SAP Cloud Logging is an instance-based and environment-agnostic observability se ## Ship Logs to SAP Cloud Logging -You can set up shipment of applications and access logs to SAP Cloud Logging. The following instructions distinguish application logs and access logs, which can be configured independently. +The Telemetry module supports two protocols for shipping logs to your backend. The method you choose determines how you configure your LogPipeline and which SAP Cloud Logging dashboards you can use. -### Set Up Application Logs - -1. Deploy the LogPipeline for application logs with the following script: +### Context -
-
Script: Application Logs +Choose one of the following methods to configure shipping for application and access logs - ```bash - kubectl apply -n sap-cloud-logging-integration -f - < +1. Deploy a [LogPipeline](./../../collecting-logs/README.md) for application logs: + + - For OTLP, run: +
+
Script: Application Logs + + ```bash + kubectl apply -f - <
-2. Wait for the LogPipeline to be in the `Running` state. To check the state, run: `kubectl get logpipelines`. - -### Set Up Access Logs - -By default, Istio sidecar injection and Istio access logs are disabled in Kyma. To analyze access logs of your workload in the default SAP Cloud Logging dashboards shipped for SAP BTP, Kyma runtime, you must enable them: - -1. Enable Istio sidecar injection for your workload. See [Enabling Istio Sidecar Injection](https://kyma-project.io/#/istio/user/tutorials/01-40-enable-sidecar-injection). - -2. Enable Istio access logs for your workload. See [Configure Istio Access Logs](https://kyma-project.io/#/istio/user/tutorials/01-45-enable-istio-access-logs). - -3. Deploy the LogPipeline for Istio access logs and enable access logs in Kyma with the following script: - -
-
Script: Access Logs - - ```bash - kubectl apply -n sap-cloud-logging-integration -f - < +
Script: Application Logs + + ```bash + kubectl apply -f - <
-4. Wait for the LogPipeline to be in the `Running` state. To check the state, run: `kubectl get logpipelines`. - -## Ship Distributed Traces to SAP Cloud Logging - -You can set up ingestion of distributed traces from applications and the Istio service mesh to the OTLP endpoint of the SAP Cloud Logging service instance. +1. Verify that the LogPipeline is running: -1. Deploy the Istio Telemetry resource with the following script: + ```bash + kubectl get logpipelines + ``` -
-
Script: Istio Traces +#### Set Up Istio Access Logs + +By default, Istio sidecar injection and Istio access logs are disabled in Kyma. To analyze them, you must enable them: + +1. Enable Istio sidecar injection for your workload (see [Enabling Istio Sidecar Proxy Injection](https://kyma-project.io/#/istio/user/tutorials/01-40-enable-sidecar-injection)). + +1. Depending on your log shipment protocol, configure the [Istio](https://istio.io/latest/docs/reference/config/telemetry/) Telemetry resource (see [Configure Istio Access Logs](./../../collecting-logs/istio-support.md)): + + - For OTLP, set up the Istio Telemetry resource with the OTLP-based kyma-logs extension provider. + - For HTTP: + 1. Set up the Istio Telemetry resource with the `stdout-json` extension provider. + 1. Deploy a LogPipeline for Istio access logs: + +
+
Script: Access Logs + + ```bash + kubectl apply -f - < +
+ + 1. Verify that the LogPipeline for Istio access logs is running: + + ```bash + kubectl get logpipelines + ``` + +## Ship Traces to SAP Cloud Logging - ```bash - kubectl apply -n istio-system -f - < -
+### Procedure - The default configuration has the **randomSamplingPercentage** property set to `1.0`, meaning it samples 1% of all requests. To change the sampling rate, adjust the property to the desired value, up to 100 percent. +#### Set Up Traces -2. Deploy the TracePipeline with the following script: +1. Deploy a [TracePipeline](./../../collecting-traces/README.md):
Script: Distributed Traces ```bash - kubectl apply -n sap-cloud-logging-integration -f - <
-3. Wait for the TracePipeline to be in the `Running` state. To check the state, run: `kubectl get tracepipelines`. +1. Verify that the TracePipeline is running: + + ```bash + kubectl get tracepipelines + ``` + +#### Set Up Istio Tracing + +By default, Istio sidecar injection and Istio tracing are disabled in Kyma. To analyze them, you must enable them: + +1. Enable Istio sidecar injection for your workload (see [Enabling Istio Sidecar Proxy Injection](https://kyma-project.io/#/istio/user/tutorials/01-40-enable-sidecar-injection)). +1. Configure the [Istio](https://istio.io/latest/docs/reference/config/telemetry/) Telemetry resource to use the kyma-traces extension provider based on OTLP (see [Configure Istio Tracing](./../../collecting-traces/istio-support.md)). ## Ship Metrics to SAP Cloud Logging You can set up ingestion of metrics from applications and the Istio service mesh to the OTLP endpoint of the SAP Cloud Logging service instance. -1. Deploy the MetricPipeline with the following script: +### Procedure + +1. Deploy a [MetricPipeline](./../../collecting-metrics/README.md):
Script: SAP Cloud Logging ```bash - kubectl apply -n sap-cloud-logging-integration -f - <
- By default, the MetricPipeline assures that a gateway is running in the cluster to push OTLP metrics. + The default configuration creates a gateway to receive OTLP metrics from your applications. +1. Optional: To collect additional metrics, such as those from the runtime or Istio, configure the presets in the input section of the MetricPipeline. +For the available options, see [Configure Metrics Collection](./../../collecting-metrics/README.md). +1. Verify that the MetricPipeline is running: -2. If you want to use additional metric collection, configure the presets under `input`. + ```bash + kubectl get metricpipelines + ``` - For the available options, see [Metrics](./../../04-metrics.md). +## Set Up Kyma Dashboard Integration -3. Wait for the MetricPipeline to be in the `Running` state. To check the state, run: `kubectl get metricpipelines`. +To add direct links from Kyma dashboard to SAP Cloud Logging, apply the ConfigMap that corresponds to your chosen log shipping method. -## Set Up Kyma Dashboard Integration +### Context -For easier access from the Kyma dashboard, add links to the navigation under **SAP Cloud Logging**, and add deep links to the **Pod**, **Deployment**, and **Namespace** views. +Depending on the output you use in your LogPipeline, apply the ConfigMap. If your Secret has a different name or namespace, then download the file first and adjust the namespace and name accordingly in the dataSources section of the file. -Depending on the output you use in your LogPipeline, apply the ConfigMap. If your Secret has a different name or namespace, then download the file first and adjust the namespace and name accordingly in the 'dataSources' section of the file. +### Procedure -- For OTLP, run: +1. If your Secret has a name or namespace different from the example, download the file and edit the dataSources section before you apply it. +1. Apply the ConfigMap: + - For OTLP, run: - ```bash - kubectl apply -f https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/kyma-dashboard-configmap.yaml - ``` + ```bash + kubectl apply -f https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/kyma-dashboard-configmap.yaml + ``` -- For HTTP, run: + - For HTTP, run: - ```bash - kubectl apply -f https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/kyma-dashboard-http-configmap.yaml - ``` + ```bash + kubectl apply -f https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/kyma-dashboard-http-configmap.yaml + ``` ## Use SAP Cloud Logging Alerts -Learn how to define and import recommended alerts for SAP Cloud Logging. The following alerts are based on JSON documents defining a `Monitor` for the alerting plugin. +You can import predefined alerts for SAP Cloud Logging to monitor the health of your telemetry integration. + +### Procedure + +1. In the SAP Cloud Logging dashboard, define a “notification channel” to receive alert notifications. +1. To import a monitor, use the development tools of the SAP Cloud Logging dashboard. +1. Execute `POST _plugins/_alerting/monitors`, followed by the contents of the respective JSON file. +1. Depending on the pipelines you are using, enable some or all of the following alerts: + The alerts are based on JSON documents defining a Monitor for the alerting plugin. -1. Define a `destination`, which will be used by all your alerts. -2. To import a monitor, use the development tools of the SAP Cloud Logging dashboard. -3. Execute `POST _plugins/_alerting/monitors`, followed by the contents of the respective JSON file. -4. Depending on the pipelines you are using, enable the some or all of the following alerts: - | Category | File | Description | - | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | - | SAP Cloud Logging | [OpenSearch cluster health](https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/alert-health.json) | The OpenSearch cluster might become unhealthy, which is indicated by a "red" status using the [cluster health API](https://opensearch.org/docs/1.3/api-reference/cluster-api/cluster-health). | - | Kyma Telemetry Integration | [Application log ingestion](https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/alert-app-log-ingestion.json) | The LogPipeline for shipping [application logs](#ship-logs-to-sap-cloud-logging) might lose connectivity to SAP Cloud Logging, with the effect that no application logs are ingested anymore. | - | Kyma Telemetry Integration | [Access log ingestion](https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/alert-access-log-ingestion.json) | The LogPipeline for shipping [access logs](#ship-logs-to-sap-cloud-logging) might lose connectivity to SAP Cloud Logging, with the effect that no access logs are ingested anymore. | - | Kyma Telemetry Integration | [Trace ingestion](https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/alert-trace-ingestion.json) | The TracePipeline for shipping [traces](#ship-distributed-traces-to-sap-cloud-logging) might lose connectivity to SAP Cloud Logging, with the effect that no traces are ingested anymore. | - | Kyma Telemetry Integration | [Metric ingestion](https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/alert-metric-ingestion.json) | The MetricPipeline for shipping [metrics](#ship-metrics-to-sap-cloud-logging) might lose connectivity to SAP Cloud Logging, with the effect that no metrics are ingested anymore. | - | Kyma Telemetry Integration | [Kyma Telemetry Status](https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/alert-telemetry-status.json) | The Telemetry module might report a non-ready state indicating a configuration or data flow problem. | + | Monitored Component | File | Description | + | -- | -- | -- | + | SAP Cloud Logging | [alert-health.json](https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/alert-health.json) | Monitors the health of the underlying OpenSearch cluster in SAP Cloud Logging using the [cluster health API](https://opensearch.org/docs/1.3/api-reference/cluster-api/cluster-health). Triggers if the cluster status becomes `red`. | + | SAP Cloud Logging | [alert-rejection-in-progress.json](https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/alert-rejection-in-progress.json) | Monitors the `cls-rejected-*` index for new data. Triggers if new rejected data is observed. | + | Kyma Telemetry Integration | [alert-telemetry-status.json](https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/alert-app-log-ingestion.json) | Monitors the status of the Telemetry module. Triggers if the module reports a non-ready state. | + | Kyma Telemetry Integration | [alert-log-ingestion.json](https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/alert-log-ingestion.json) | For OTLP: Monitors the single LogPipeline used in the OTLP method. Triggers if log data stops flowing. | + | Kyma Telemetry Integration | [alert-app-log-ingestion.json](https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/alert-access-log-ingestion.json) | For HTTP (legacy): Monitors the application log LogPipeline. Triggers if log data stops flowing. | + | Kyma Telemetry Integration | [alert-access-log-ingestion.json](https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/alert-access-log-ingestion.json) | For HTTP (legacy): Monitors the Istio access log LogPipeline. Triggers if log data stops flowing. | + | Kyma Telemetry Integration | [alert-trace-ingestion.json](https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/alert-trace-ingestion.json) | Monitors the TracePipeline. Triggers if trace data stops flowing to SAP Cloud Logging. | + | Kyma Telemetry Integration | [alert-metric-ingestion.json](https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/alert-metric-ingestion.json) | Monitors the MetricPipeline. Triggers if metric data stops flowing to SAP Cloud Logging. | + -5. Edit notification action: Add the `destination` and adjust the intervals and thresholds to your needs. +5. After importing, edit the monitor to attach your notification channel or destination and adjust thresholds as needed. 6. Verify that the new monitor definition is listed among the SAP Cloud Logging alerts. ## Use SAP Cloud Logging Dashboards -You can view logs, traces, and metrics in SAP Cloud Logging dashboards: +You can view logs, traces, and metrics in SAP Cloud Logging dashboards. Several dashboards come with SAP Cloud Logging, and you can import additional dashboards as needed. + +### Context + +The preconfigured `Kyma_*` dashboards in SAP Cloud Logging are compatible only with the legacy (HTTP) logging method. + +### Procedure -- To view the traffic and application logs, use the SAP Cloud Logging dashboards prefixed with `Kyma_`, which are based on both kinds of log ingestion: application and access logs. -- To view distributed traces, use the OpenSearch plugin **Observability**. -- To view the container- and Pod-related metrics collected by the MetricPipeline `runtime` input, use the dashboard **[OTel] K8s Container Metrics**. -- To view the Kubernetes Node-related metrics collected by the MetricPipeline `runtime` input, manually import the file [K8s Nodes](https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/dashboard-nodes.ndjson). -- To view the Kubernetes Volume-related metrics collected by the MetricPipeline `runtime` input, manually import the file [K8s Volumes](https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/dashboard-volumes.ndjson). -- To view the Kubernetes Workload-related metrics collected by the MetricPipeline `runtime` input, manually import the file [K8s Workloads](https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/dashboard-workloads.ndjson). -- To view the status of the SAP Cloud Logging integration with the Kyma Telemetry module, manually import the file [Kyma Telemetry Status](https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/dashboard-status.ndjson). -- To use the dashboard for Istio metrics of Pods that have an active Istio sidecar injection (collected by the MetricPipeline `istio` input), manually import the file [Kyma Istio Service Metrics](https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/dashboard-istio.ndjson). +- For the status of the SAP Cloud Logging integration with the Telemetry module, import the file [dashboard-status.ndjson](https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/dashboard-status.ndjson). +- For application logs and Istio access logs using the http output: Use the preconfigured dashboards prefixed with `Kyma_*`. +- For traces, use the OpenSearch plugin “Observability”. +- For runtime metrics, import the file [dashboard-runtime.ndjson](https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/dashboard-runtime.ndjson). +- For Istio Pod metrics, import the file [dashboard-istio.ndjson](https://raw.githubusercontent.com/kyma-project/telemetry-manager/main/docs/user/integration/sap-cloud-logging/dashboard-istio.ndjson). diff --git a/docs/user/integration/sap-cloud-logging/alert-access-log-ingestion.json b/docs/user/integration/sap-cloud-logging/alert-access-log-ingestion.json index 9e5ab5d6cf..1abf83a65b 100644 --- a/docs/user/integration/sap-cloud-logging/alert-access-log-ingestion.json +++ b/docs/user/integration/sap-cloud-logging/alert-access-log-ingestion.json @@ -40,7 +40,6 @@ "triggers": [ { "query_level_trigger": { - "id": "7o90X4UBxib8bUp3mEDF", "name": "No access log ingestion", "severity": "2", "condition": { @@ -51,7 +50,6 @@ }, "actions": [ { - "id": "7490X4UBxib8bUp3mEDF", "name": "Notify", "destination_id": "", "message_template": { @@ -71,46 +69,5 @@ ] } } - ], - "ui_metadata": { - "schedule": { - "timezone": null, - "frequency": "interval", - "period": { - "unit": "MINUTES", - "interval": 1 - }, - "daily": 0, - "weekly": { - "tue": false, - "wed": false, - "thur": false, - "sat": false, - "fri": false, - "mon": false, - "sun": false - }, - "monthly": { - "type": "day", - "day": 1 - }, - "cronExpression": "0 */1 * * *" - }, - "search": { - "searchType": "graph", - "timeField": "@timestamp", - "aggregations": [], - "groupBy": [], - "bucketValue": 5, - "bucketUnitOfTime": "m", - "where": { - "fieldName": [], - "fieldRangeEnd": 0, - "fieldRangeStart": 0, - "fieldValue": "", - "operator": "is" - } - }, - "monitor_type": "query_level_monitor" - } + ] } \ No newline at end of file diff --git a/docs/user/integration/sap-cloud-logging/alert-app-log-ingestion.json b/docs/user/integration/sap-cloud-logging/alert-app-log-ingestion.json index 7851497984..04f9b5411d 100644 --- a/docs/user/integration/sap-cloud-logging/alert-app-log-ingestion.json +++ b/docs/user/integration/sap-cloud-logging/alert-app-log-ingestion.json @@ -40,7 +40,6 @@ "triggers": [ { "query_level_trigger": { - "id": "7o90X4UBxib8bUp3mEDC", "name": "No Kyma application log ingestion", "severity": "2", "condition": { @@ -51,7 +50,6 @@ }, "actions": [ { - "id": "7490X4UBxib8bUp3mEDC", "name": "Notify", "destination_id": "", "message_template": { @@ -71,46 +69,5 @@ ] } } - ], - "ui_metadata": { - "schedule": { - "timezone": null, - "frequency": "interval", - "period": { - "unit": "MINUTES", - "interval": 1 - }, - "daily": 0, - "weekly": { - "tue": false, - "wed": false, - "thur": false, - "sat": false, - "fri": false, - "mon": false, - "sun": false - }, - "monthly": { - "type": "day", - "day": 1 - }, - "cronExpression": "0 */1 * * *" - }, - "search": { - "searchType": "graph", - "timeField": "@timestamp", - "aggregations": [], - "groupBy": [], - "bucketValue": 5, - "bucketUnitOfTime": "m", - "where": { - "fieldName": [], - "fieldRangeEnd": 0, - "fieldRangeStart": 0, - "fieldValue": "", - "operator": "is" - } - }, - "monitor_type": "query_level_monitor" - } + ] } \ No newline at end of file diff --git a/docs/user/integration/sap-cloud-logging/alert-health.json b/docs/user/integration/sap-cloud-logging/alert-health.json index bd2dcbe90f..11a3182fe1 100644 --- a/docs/user/integration/sap-cloud-logging/alert-health.json +++ b/docs/user/integration/sap-cloud-logging/alert-health.json @@ -22,7 +22,6 @@ "triggers": [ { "query_level_trigger": { - "id": "Ek8rl4cBIncOvUuuWmSC", "name": "Cloud Logging health is red", "severity": "1", "condition": { @@ -33,9 +32,8 @@ }, "actions": [ { - "id": "E08rl4cBIncOvUuuWmSC", "name": "Notify", - "destination_id": "", + "destination_id": "", "message_template": { "source": "Monitor {{ctx.monitor.name}} just entered alert status. For more details query the API as described in https://opensearch.org/docs/1.3/api-reference/cluster-api/cluster-health/\n - Trigger: {{ctx.trigger.name}}\n - Severity: {{ctx.trigger.severity}}\n - Period start: {{ctx.periodStart}}\n - Period end: {{ctx.periodEnd}}\n - /app/alerting", "lang": "mustache" @@ -53,47 +51,6 @@ ] } } - ], - "ui_metadata": { - "schedule": { - "timezone": null, - "frequency": "interval", - "period": { - "unit": "MINUTES", - "interval": 1 - }, - "daily": 0, - "weekly": { - "tue": false, - "wed": false, - "thur": false, - "sat": false, - "fri": false, - "mon": false, - "sun": false - }, - "monthly": { - "type": "day", - "day": 1 - }, - "cronExpression": "0 */1 * * *" - }, - "search": { - "searchType": "clusterMetrics", - "timeField": "", - "aggregations": [], - "groupBy": [], - "bucketValue": 1, - "bucketUnitOfTime": "h", - "where": { - "fieldName": [], - "fieldRangeEnd": 0, - "fieldRangeStart": 0, - "fieldValue": "", - "operator": "is" - } - }, - "monitor_type": "cluster_metrics_monitor" - } + ] } \ No newline at end of file diff --git a/docs/user/integration/sap-cloud-logging/alert-log-ingestion.json b/docs/user/integration/sap-cloud-logging/alert-log-ingestion.json new file mode 100644 index 0000000000..36adedbebe --- /dev/null +++ b/docs/user/integration/sap-cloud-logging/alert-log-ingestion.json @@ -0,0 +1,78 @@ +{ + "type": "monitor", + "schema_version": 8, + "name": "Kyma Log Ingestion", + "monitor_type": "query_level_monitor", + "enabled": true, + "schedule": { + "period": { + "interval": 1, + "unit": "MINUTES" + } + }, + "inputs": [ + { + "search": { + "indices": [ + "logs-otel-v1-*" + ], + "query": { + "size": 0, + "query": { + "bool": { + "filter": [ + { + "range": { + "time": { + "from": "{{period_end}}||-5m", + "to": "{{period_end}}", + "include_lower": true, + "include_upper": true, + "format": "epoch_millis", + "boost": 1 + } + } + } + ], + "adjust_pure_negative": true, + "boost": 1 + } + } + } + } + } + ], + "triggers": [ + { + "query_level_trigger": { + "name": "No OTLP Log Ingestion", + "severity": "2", + "condition": { + "script": { + "source": "ctx.results[0].hits.total.value < 1", + "lang": "painless" + } + }, + "actions": [ + { + "name": "Slack me", + "destination_id": "", + "message_template": { + "source": "Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.\n - Trigger: {{ctx.trigger.name}}\n - Severity: {{ctx.trigger.severity}}\n - Period start: {{ctx.periodStart}}\n - Period end: {{ctx.periodEnd}}\n - /app/alerting", + "lang": "mustache" + }, + "throttle_enabled": true, + "subject_template": { + "source": "No OTLP Logs from a Kyma runtime ingested since 5 min", + "lang": "mustache" + }, + "throttle": { + "value": 60, + "unit": "MINUTES" + } + } + ] + } + } + ] + } \ No newline at end of file diff --git a/docs/user/integration/sap-cloud-logging/alert-metric-ingestion.json b/docs/user/integration/sap-cloud-logging/alert-metric-ingestion.json index 87e0d55f48..7fbb680bbe 100644 --- a/docs/user/integration/sap-cloud-logging/alert-metric-ingestion.json +++ b/docs/user/integration/sap-cloud-logging/alert-metric-ingestion.json @@ -40,7 +40,6 @@ "triggers": [ { "query_level_trigger": { - "id": "7o90X4UBxib8bUp3mEDQ", "name": "No Kyma metric ingestion", "severity": "2", "condition": { @@ -51,7 +50,6 @@ }, "actions": [ { - "id": "7490X4UBxib8bUp3mEDQ", "name": "Notify", "destination_id": "", "message_template": { @@ -71,46 +69,5 @@ ] } } - ], - "ui_metadata": { - "schedule": { - "timezone": null, - "frequency": "interval", - "period": { - "unit": "MINUTES", - "interval": 1 - }, - "daily": 0, - "weekly": { - "tue": false, - "wed": false, - "thur": false, - "sat": false, - "fri": false, - "mon": false, - "sun": false - }, - "monthly": { - "type": "day", - "day": 1 - }, - "cronExpression": "0 */1 * * *" - }, - "search": { - "searchType": "graph", - "timeField": "time", - "aggregations": [], - "groupBy": [], - "bucketValue": 5, - "bucketUnitOfTime": "m", - "where": { - "fieldName": [], - "fieldRangeEnd": 0, - "fieldRangeStart": 0, - "fieldValue": "", - "operator": "is" - } - }, - "monitor_type": "query_level_monitor" - } + ] } \ No newline at end of file diff --git a/docs/user/integration/sap-cloud-logging/alert-rejection-in-progress.json b/docs/user/integration/sap-cloud-logging/alert-rejection-in-progress.json new file mode 100644 index 0000000000..543beaf706 --- /dev/null +++ b/docs/user/integration/sap-cloud-logging/alert-rejection-in-progress.json @@ -0,0 +1,108 @@ +{ + "name": "Rejection in progress", + "type": "monitor", + "monitor_type": "query_level_monitor", + "enabled": true, + "schedule": { + "period": { + "unit": "MINUTES", + "interval": 1 + } + }, + "inputs": [ + { + "search": { + "indices": [ + "cls-rejected-*" + ], + "query": { + "size": 0, + "aggregations": {}, + "query": { + "bool": { + "filter": [ + { + "range": { + "@timestamp": { + "gte": "{{period_end}}||-1h", + "lte": "{{period_end}}", + "format": "epoch_millis" + } + } + } + ] + } + } + } + } + } + ], + "triggers": [ + { + "query_level_trigger": { + "name": "Rejections in progress", + "severity": "2", + "condition": { + "script": { + "source": "ctx.results[0].hits.total.value > 1", + "lang": "painless" + } + }, + "actions": [ + { + "name": "Notify", + "destination_id": "", + "message_template": { + "source": "Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.\n - Trigger: {{ctx.trigger.name}}\n - Severity: {{ctx.trigger.severity}}\n - Period start: {{ctx.periodStart}}\n - Period end: {{ctx.periodEnd}}", + "lang": "mustache" + }, + "throttle_enabled": true, + "subject_template": { + "source": "Rejection in progress", + "lang": "mustache" + }, + "throttle": { + "value": 60, + "unit": "MINUTES" + } + } + ] + } + } + ], + "ui_metadata": { + "schedule": { + "timezone": null, + "frequency": "interval", + "period": { + "unit": "MINUTES", + "interval": 1 + }, + "daily": 0, + "weekly": { + "tue": false, + "wed": false, + "thur": false, + "sat": false, + "fri": false, + "mon": false, + "sun": false + }, + "monthly": { + "type": "day", + "day": 1 + }, + "cronExpression": "0 */1 * * *" + }, + "monitor_type": "query_level_monitor", + "search": { + "searchType": "graph", + "timeField": "@timestamp", + "aggregations": [], + "groupBy": [], + "bucketValue": 1, + "bucketUnitOfTime": "h", + "filters": [] + } + } +} \ No newline at end of file diff --git a/docs/user/integration/sap-cloud-logging/alert-telemetry-status.json b/docs/user/integration/sap-cloud-logging/alert-telemetry-status.json index 58d2a950f3..28b8d28903 100644 --- a/docs/user/integration/sap-cloud-logging/alert-telemetry-status.json +++ b/docs/user/integration/sap-cloud-logging/alert-telemetry-status.json @@ -66,7 +66,6 @@ "triggers": [ { "query_level_trigger": { - "id": "r88m6pABs6tXO7H26USv", "name": "Inform about module problem", "severity": "1", "condition": { @@ -77,7 +76,6 @@ }, "actions": [ { - "id": "sM8m6pABs6tXO7H26USv", "name": "Inform about module problem", "destination_id": "", "message_template": { @@ -97,56 +95,5 @@ ] } } - ], - "ui_metadata": { - "schedule": { - "timezone": null, - "frequency": "interval", - "period": { - "unit": "MINUTES", - "interval": 1 - }, - "daily": 0, - "weekly": { - "tue": false, - "wed": false, - "thur": false, - "sat": false, - "fri": false, - "mon": false, - "sun": false - }, - "monthly": { - "type": "day", - "day": 1 - }, - "cronExpression": "0 */1 * * *" - }, - "search": { - "searchType": "query", - "timeField": "@timestamp", - "aggregations": [ - { - "aggregationType": "max", - "fieldName": "audit_compliance_doc_version" - } - ], - "groupBy": [], - "bucketValue": 1, - "bucketUnitOfTime": "h", - "where": { - "fieldName": [ - { - "label": "audit_format_version", - "type": "number" - } - ], - "fieldRangeEnd": 0, - "fieldRangeStart": 0, - "fieldValue": 4, - "operator": "is" - } - }, - "monitor_type": "query_level_monitor" - } + ] } \ No newline at end of file diff --git a/docs/user/integration/sap-cloud-logging/alert-trace-ingestion.json b/docs/user/integration/sap-cloud-logging/alert-trace-ingestion.json index 03c9be0559..3e2344838c 100644 --- a/docs/user/integration/sap-cloud-logging/alert-trace-ingestion.json +++ b/docs/user/integration/sap-cloud-logging/alert-trace-ingestion.json @@ -1,28 +1,28 @@ { - "name": "Kyma trace ingestion", - "type": "monitor", - "monitor_type": "query_level_monitor", - "enabled": true, - "schedule": { - "period": { - "unit": "MINUTES", - "interval": 1 - } - }, - "inputs": [ - { - "search": { - "indices": [ - "otel-v1-apm-span-*" - ], - "query": { - "size": 0, - "aggregations": {}, - "query": { - "bool": { - "filter": [ - { - "range": { + "name": "Kyma trace ingestion", + "type": "monitor", + "monitor_type": "query_level_monitor", + "enabled": true, + "schedule": { + "period": { + "unit": "MINUTES", + "interval": 1 + } + }, + "inputs": [ + { + "search": { + "indices": [ + "otel-v1-apm-span-*" + ], + "query": { + "size": 0, + "aggregations": {}, + "query": { + "bool": { + "filter": [ + { + "range": { "startTime": { "from": "{{period_end}}||-5m", "to": "{{period_end}}", @@ -31,91 +31,48 @@ "format": "epoch_millis", "boost": 1 } - } - } - ], - "adjust_pure_negative": true, - "boost": 1 - } - } - } - } - } - ], - "triggers": [ - { - "query_level_trigger": { - "id": "7o90X4UBxib8bUp3mEDK", - "name": "No Kyma trace ingestion", - "severity": "2", - "condition": { - "script": { - "source": "ctx.results[0].hits.total.value < 1", - "lang": "painless" - } - }, - "actions": [ - { - "id": "7490X4UBxib8bUp3mEDK", - "name": "Notify", - "destination_id": "", - "message_template": { - "source": "Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.\n - Trigger: {{ctx.trigger.name}}\n - Severity: {{ctx.trigger.severity}}\n - Period start: {{ctx.periodStart}}\n - Period end: {{ctx.periodEnd}}\n - /app/alerting", - "lang": "mustache" - }, - "throttle_enabled": true, - "subject_template": { - "source": "No traces from a Kyma runtime ingested since 5 min", - "lang": "mustache" - }, - "throttle": { - "value": 60, - "unit": "MINUTES" - } - } - ] - } - } - ], - "ui_metadata": { - "schedule": { - "timezone": null, - "frequency": "interval", - "period": { - "unit": "MINUTES", - "interval": 1 - }, - "daily": 0, - "weekly": { - "tue": false, - "wed": false, - "thur": false, - "sat": false, - "fri": false, - "mon": false, - "sun": false - }, - "monthly": { - "type": "day", - "day": 1 - }, - "cronExpression": "0 */1 * * *" - }, - "search": { - "searchType": "query", - "timeField": "startTime", - "aggregations": [], - "groupBy": [], - "bucketValue": 5, - "bucketUnitOfTime": "m", - "where": { - "fieldName": [], - "fieldRangeEnd": 0, - "fieldRangeStart": 0, - "fieldValue": "", - "operator": "is" - } - }, - "monitor_type": "query_level_monitor" - } - } \ No newline at end of file + } + } + ], + "adjust_pure_negative": true, + "boost": 1 + } + } + } + } + } + ], + "triggers": [ + { + "query_level_trigger": { + "name": "No Kyma trace ingestion", + "severity": "2", + "condition": { + "script": { + "source": "ctx.results[0].hits.total.value < 1", + "lang": "painless" + } + }, + "actions": [ + { + "name": "Notify", + "destination_id": "", + "message_template": { + "source": "Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.\n - Trigger: {{ctx.trigger.name}}\n - Severity: {{ctx.trigger.severity}}\n - Period start: {{ctx.periodStart}}\n - Period end: {{ctx.periodEnd}}\n - /app/alerting", + "lang": "mustache" + }, + "throttle_enabled": true, + "subject_template": { + "source": "No traces from a Kyma runtime ingested since 5 min", + "lang": "mustache" + }, + "throttle": { + "value": 60, + "unit": "MINUTES" + } + } + ] + } + } + ] +} \ No newline at end of file diff --git a/docs/user/integration/sap-cloud-logging/dashboard-nodes.ndjson b/docs/user/integration/sap-cloud-logging/dashboard-runtime.ndjson similarity index 69% rename from docs/user/integration/sap-cloud-logging/dashboard-nodes.ndjson rename to docs/user/integration/sap-cloud-logging/dashboard-runtime.ndjson index 3eb379f4b5..9283000243 100644 --- a/docs/user/integration/sap-cloud-logging/dashboard-nodes.ndjson +++ b/docs/user/integration/sap-cloud-logging/dashboard-runtime.ndjson @@ -1,19 +1,35 @@ -{"attributes":{"fields":"[{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"_id\"],\"name\":\"_id\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"_index\"],\"name\":\"_index\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":false,\"count\":0,\"name\":\"_score\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":false,\"type\":\"number\"},{\"aggregatable\":false,\"count\":0,\"esTypes\":[\"_source\"],\"name\":\"_source\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":false,\"type\":\"_source\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"_type\"],\"name\":\"_type\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"aggregationTemporality\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"app_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"app_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"bucketCounts\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"bucketCountsList\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"buckets.count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"buckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"buckets.max\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"buckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"buckets.min\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"buckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":false,\"count\":0,\"esTypes\":[\"text\"],\"name\":\"description\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"exemplars.spanId\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"exemplars\"}},\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"date_nanos\"],\"name\":\"exemplars.time\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"exemplars\"}},\"type\":\"date\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"exemplars.traceId\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"exemplars\"}},\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"exemplars.value\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"exemplars\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"explicitBounds\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"explicitBoundsCount\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"flags\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"instance_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"instrumentationLibrary.name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"instrumentationLibrary.version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"instrumentationScope.name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"instrumentationScope.version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"boolean\"],\"name\":\"isMonotonic\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"boolean\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"kind\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.app@payment@currency\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.build_date\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.cause\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.cluster_type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.component\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.connection_security_policy\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.core\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.cpu\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.currency_code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.debug\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_app\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_canonical_revision\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_canonical_service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_cluster\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_principal\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_service_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_service_namespace\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_workload\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_workload_namespace\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.device\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.direction\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.endpoint\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.err\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.event\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.family\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.format\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.group\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.grpc_code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.grpc_method\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.grpc_response_status\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.grpc_service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.grpc_type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.host\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.http@flavor\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.http@host\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.http@method\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.http@scheme\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.http@status_code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":4,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.interface\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.method\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.model\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.net@host@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.net@host@port\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.net@peer@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.net@peer@port\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.net@protocol@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.net@protocol@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.node\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.nodejs@eventloop@state\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.operation\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.otel_scope_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.otel_scope_version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.path\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.protocol\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.reason\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.recommendation@type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.reporter\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.request_protocol\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.resource\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.response_code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.response_flags\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.result\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.revision\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.rpc@grpc@status_code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.rpc@method\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.rpc@service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.rpc@system\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.sampler_type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_app\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_canonical_revision\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_canonical_service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_cluster\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_principal\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_workload\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_workload_namespace\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.state\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.status\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.svc\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.tag\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.target\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.transport\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":2,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.v8js@gc@type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.v8js@heap@space@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"negative\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"negativeBuckets.count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"negativeBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"negativeBuckets.max\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"negativeBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"negativeBuckets.min\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"negativeBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"negativeOffset\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"organization_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"organization_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"positive\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"positiveBuckets.count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"positiveBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"positiveBuckets.max\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"positiveBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"positiveBuckets.min\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"positiveBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"positiveOffset\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"process_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"process_type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"quantileValuesCount\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"quantiles.quantile\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"quantiles\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"quantiles.value\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"quantiles\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.cloud@account@id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.cloud@availability_zone\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.cloud@platform\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.cloud@provider\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.cloud@region\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.container@id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.container@image@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.container@image@tag\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.host@arch\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.host@id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.host@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.host@type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.http@scheme\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":2,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@cluster@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":6,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@container@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@daemonset@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@daemonset@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@deployment@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@deployment@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@job@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@job@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":1,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@namespace@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":1,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@node@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@node@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@persistentvolumeclaim@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@pod@ip\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":11,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@pod@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@pod@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@replicaset@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@resource@group\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@resource@kind\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@resource@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@resource@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@statefulset@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@statefulset@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@volume@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@volume@type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.net@host@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.net@host@port\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.os@description\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.os@type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.os@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@command\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@command_args\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@executable@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@executable@path\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@owner\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@pid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@runtime@description\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@runtime@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@runtime@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@app_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@app_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@instance_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@org_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@org_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@process@id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@process@type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@space_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@space_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.server@address\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.server@port\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":1,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service@instance@id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service@namespace\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry@auto@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry@sdk@language\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry@sdk@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry@sdk@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry_sdk_language\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry_sdk_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry_sdk_version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.url@scheme\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"scale\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"schemaUrl\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"serviceName\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"space_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"space_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"date_nanos\"],\"name\":\"startTime\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"date\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"sum\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"date_nanos\"],\"name\":\"time\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"date\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"unit\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"value\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"zeroCount\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"}]","timeFieldName":"time","title":"metrics-otel-v1-*"},"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","migrationVersion":{"index-pattern":"7.6.0"},"references":[],"type":"index-pattern","updated_at":"2025-09-16T16:00:51.551Z","version":"Wzc3MywzXQ=="} -{"attributes":{"columns":["_source"],"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"highlightAll\":true,\"version\":true,\"query\":{\"query\":\"name: k8s.node.* or name: k8s.pod.*\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"sort":[],"title":"K8s Nodes","version":1},"id":"c9f6b090-9775-11ef-a1d8-af81cc9045d1","migrationVersion":{"search":"7.9.3"},"references":[{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"search","updated_at":"2025-09-16T16:00:51.551Z","version":"Wzc3NCwzXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"Nodes list2","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}","version":1,"visState":"{\"title\":\"Nodes list2\",\"type\":\"table\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"cardinality\",\"params\":{\"field\":\"resource.attributes.k8s@pod@name\",\"customLabel\":\"Pods\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@node@name\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Nodes\"},\"schema\":\"bucket\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"3-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.node.memory.working_set\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"3-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"json\":\"{\\n \\\"script\\\": {\\n \\\"inline\\\": \\\"doc['value'].value / 1024 / 1024 / 1024\\\",\\n \\\"lang\\\": \\\"painless\\\"\\n }\\n}\"}},\"customLabel\":\"Memory Usage (WSS)(GB)\"},\"schema\":\"metric\"},{\"id\":\"4\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"4-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.node.memory.available\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"4-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"json\":\"{\\n \\\"script\\\": {\\n \\\"inline\\\": \\\"doc['value'].value / 1024 / 1024 / 1024\\\",\\n \\\"lang\\\": \\\"painless\\\"\\n }\\n}\"}},\"customLabel\":\"Memory Available (GB)\"},\"schema\":\"metric\"},{\"id\":\"5\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"5-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.node.cpu.usage\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"5-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"json\":\"\"}},\"customLabel\":\"CPU Usage\"},\"schema\":\"metric\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"ebdb2f00-9776-11ef-a1d8-af81cc9045d1","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"c9f6b090-9775-11ef-a1d8-af81cc9045d1","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2025-09-16T16:00:51.551Z","version":"Wzc3NSwzXQ=="} -{"attributes":{"columns":["_source"],"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"highlightAll\":true,\"version\":true,\"query\":{\"query\":\" name: k8s.container.* or name: container.*\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"sort":[],"title":"K8s Container","version":1},"id":"d6b5f6d0-96d3-11ef-a1d8-af81cc9045d1","migrationVersion":{"search":"7.9.3"},"references":[{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"search","updated_at":"2025-09-16T16:00:51.551Z","version":"Wzc3NiwzXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"container resources","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":0,\"direction\":\"asc\"}}}}","version":1,"visState":"{\"title\":\"container resources\",\"type\":\"table\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"1-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.container.cpu_request\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"1-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"customLabel\":\"\"}},\"customLabel\":\"CPU Request\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@pod@name\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":997,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Pod\"},\"schema\":\"bucket\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@container@name\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Container\"},\"schema\":\"bucket\"},{\"id\":\"8\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"8-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: container.cpu.usage\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"8-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"CPU Usage\"},\"schema\":\"metric\"},{\"id\":\"4\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"4-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.container.cpu_limit\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"4-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"CPU Limit\"},\"schema\":\"metric\"},{\"id\":\"5\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"5-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.container.memory_request\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"5-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"json\":\"{\\n \\\"script\\\": {\\n \\\"inline\\\": \\\"doc['value'].value / 1024 / 1024\\\",\\n \\\"lang\\\": \\\"painless\\\"\\n }\\n}\"}},\"json\":\"\",\"customLabel\":\"Memory Request (MB)\"},\"schema\":\"metric\"},{\"id\":\"7\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"7-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: container.memory.working_set\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"7-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"json\":\"{\\n \\\"script\\\": {\\n \\\"inline\\\": \\\"doc['value'].value / 1024 / 1024\\\",\\n \\\"lang\\\": \\\"painless\\\"\\n }\\n}\"}},\"customLabel\":\"Memory Usage (MB)\"},\"schema\":\"metric\"},{\"id\":\"6\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"6-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.container.memory_limit\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"6-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"json\":\"{\\n \\\"script\\\": {\\n \\\"inline\\\": \\\"doc['value'].value / 1024 / 1024\\\",\\n \\\"lang\\\": \\\"painless\\\"\\n }\\n}\"}},\"customLabel\":\"Memory Limit (MB)\"},\"schema\":\"metric\"},{\"id\":\"9\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"9-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name:k8s.container.restarts\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"9-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"customLabel\":\"Restarts\"}},\"customLabel\":\"Restarts\"},\"schema\":\"metric\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":true,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"a8722220-96d4-11ef-b4ed-536715f658e8","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"d6b5f6d0-96d3-11ef-a1d8-af81cc9045d1","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2025-09-16T16:00:51.551Z","version":"Wzc3NywzXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Amount of Nodes","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Amount of Nodes\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"cardinality\",\"field\":\"resource.attributes.k8s@node@name\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Nodes\",\"type\":\"timeseries\",\"split_filters\":[{\"color\":\"#68BC00\",\"id\":\"a03ac6d0-7f81-11ef-a705-4fe2f82b8e2c\",\"filter\":{\"query\":\"\",\"language\":\"kuery\"}}],\"terms_field\":null,\"value_template\":\"{{value}}\",\"offset_time\":\"1m\",\"hide_in_legend\":0}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.node.cpu.usage\",\"language\":\"kuery\"},\"annotations\":[{\"fields\":\"\",\"template\":\"\",\"index_pattern\":\"*\",\"query_string\":{\"query\":\"\",\"language\":\"kuery\"},\"id\":\"b081cca0-7f81-11ef-a705-4fe2f82b8e2c\",\"color\":\"#F00\",\"icon\":\"fa-tag\",\"ignore_global_filters\":1,\"ignore_panel_filters\":1}],\"drop_last_bucket\":1}}"},"id":"9bd3f780-80c4-11ef-b7e6-d7428a02bdd1","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:00:51.551Z","version":"Wzc3OCwzXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Node CPU usage ","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Node CPU usage \",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"max\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Node CPU usage \",\"type\":\"timeseries\",\"split_filters\":[{\"color\":\"#68BC00\",\"id\":\"a03ac6d0-7f81-11ef-a705-4fe2f82b8e2c\",\"filter\":{\"query\":\"\",\"language\":\"kuery\"}}],\"terms_field\":\"resource.attributes.k8s@node@name\",\"value_template\":\"{{value}} CPU\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.node.cpu.usage\",\"language\":\"kuery\"},\"annotations\":[{\"fields\":\"\",\"template\":\"\",\"index_pattern\":\"*\",\"query_string\":{\"query\":\"\",\"language\":\"kuery\"},\"id\":\"b081cca0-7f81-11ef-a705-4fe2f82b8e2c\",\"color\":\"#F00\",\"icon\":\"fa-tag\",\"ignore_global_filters\":1,\"ignore_panel_filters\":1}]}}"},"id":"ddc86020-7f81-11ef-aad9-b55b1b4895ab","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:00:51.551Z","version":"Wzc3OSwzXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Node Memory Usage","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Node Memory Usage\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"max\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"0.00b\",\"chart_type\":\"line\",\"line_width\":\"2\",\"point_size\":1,\"fill\":\"0.5\",\"stacked\":\"none\",\"label\":\"Node Memory Usage\",\"type\":\"timeseries\",\"terms_field\":\"resource.attributes.k8s@node@name\",\"terms_order_by\":\"61ca57f2-469d-11e7-af02-69e470af7417\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.node.memory.usage\",\"language\":\"kuery\"}}}"},"id":"e279cfb0-7f80-11ef-aad9-b55b1b4895ab","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:00:51.551Z","version":"Wzc4MCwzXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Node Memory Available","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Node Memory Available\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"max\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"'0.00b'\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Node Memory Available\",\"type\":\"timeseries\",\"terms_field\":\"resource.attributes.k8s@node@name\",\"terms_order_by\":\"61ca57f2-469d-11e7-af02-69e470af7417\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.node.memory.available\",\"language\":\"kuery\"}}}"},"id":"535828d0-7f81-11ef-aad9-b55b1b4895ab","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:00:51.551Z","version":"Wzc4MSwzXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Node Memory Working Set","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Node Memory Working Set\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"max\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"0.00b\",\"chart_type\":\"line\",\"line_width\":\"2\",\"point_size\":1,\"fill\":\"0.5\",\"stacked\":\"none\",\"label\":\"Node Memory Usage\",\"type\":\"timeseries\",\"terms_field\":\"resource.attributes.k8s@node@name\",\"terms_order_by\":\"61ca57f2-469d-11e7-af02-69e470af7417\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.node.memory.working_set\",\"language\":\"kuery\"}}}"},"id":"eec8ff10-84bd-11ef-9853-17e21a5b68d5","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:00:51.551Z","version":"Wzc4MiwzXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Node Memory RSS","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Node Memory RSS\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"max\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"'0.00b'\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Node Memory Available\",\"type\":\"timeseries\",\"terms_field\":\"resource.attributes.k8s@node@name\",\"terms_order_by\":\"61ca57f2-469d-11e7-af02-69e470af7417\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.node.memory.rss\",\"language\":\"kuery\"}}}"},"id":"f2aa09d0-84bd-11ef-b7e6-d7428a02bdd1","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:00:51.551Z","version":"Wzc4MywzXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Node Filesystem Usage","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Node Filesystem Usage\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"sigma\":\"\",\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"max\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"'0.00b'\",\"chart_type\":\"line\",\"line_width\":\"2\",\"point_size\":1,\"fill\":\"0.6\",\"stacked\":\"none\",\"label\":\"Node Filesystem Usage\",\"type\":\"timeseries\",\"terms_field\":\"resource.attributes.k8s@node@name\",\"filter\":{\"query\":\"\",\"language\":\"kuery\"},\"value_template\":\"{{value}}\",\"steps\":1,\"terms_order_by\":\"61ca57f2-469d-11e7-af02-69e470af7417\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"background_color_rules\":[{\"id\":\"31f39bb0-7f73-11ef-a705-4fe2f82b8e2c\"}],\"bar_color_rules\":[{\"id\":\"32e3c810-7f73-11ef-a705-4fe2f82b8e2c\"}],\"gauge_color_rules\":[{\"id\":\"3357bf90-7f73-11ef-a705-4fe2f82b8e2c\"}],\"gauge_width\":10,\"gauge_inner_width\":10,\"gauge_style\":\"half\",\"filter\":{\"query\":\"name: k8s.node.filesystem.usage\",\"language\":\"kuery\"},\"annotations\":[{\"fields\":\"\",\"template\":\"\",\"index_pattern\":\"*\",\"query_string\":{\"query\":\"\",\"language\":\"kuery\"},\"id\":\"e0c8bac0-7f74-11ef-a705-4fe2f82b8e2c\",\"color\":\"#F00\",\"icon\":\"fa-tag\",\"ignore_global_filters\":1,\"ignore_panel_filters\":1}]}}"},"id":"21169010-7f76-11ef-9899-37d721678cf0","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:00:51.551Z","version":"Wzc4NCwzXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Node Filesystem Available","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Node Filesystem Available\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"max\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"'0.00b'\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Node Filesystem Available\",\"type\":\"timeseries\",\"terms_field\":\"resource.attributes.k8s@node@name\",\"terms_order_by\":\"61ca57f2-469d-11e7-af02-69e470af7417\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.node.filesystem.available\",\"language\":\"kuery\"}}}"},"id":"bee22630-7f83-11ef-aad9-b55b1b4895ab","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:00:51.551Z","version":"Wzc4NSwzXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Node Network IO Receive","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Node Network IO Receive\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#54B399\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"unit\":\"\",\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"positive_rate\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"axis_scale\":\"normal\",\"formatter\":\"bytes\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Node Network IO\",\"type\":\"timeseries\",\"terms_field\":\"resource.attributes.k8s@node@name\",\"terms_order_by\":\"_key\",\"value_template\":\"{{value}}\",\"filter\":{\"query\":\"\",\"language\":\"kuery\"},\"terms_size\":\"100\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.node.network.io AND metric.attributes.direction: receive\",\"language\":\"kuery\"}}}"},"id":"13e784c0-78e1-11f0-b9ad-d91d395f9528","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:18:12.328Z","version":"Wzg5NiwzXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Node Network Errors Receive","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Node Network Errors Receive\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#54B399\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"positive_rate\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"axis_scale\":\"normal\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Node Network Errors\",\"type\":\"timeseries\",\"terms_field\":\"resource.attributes.k8s@node@name\",\"terms_order_by\":\"_key\",\"terms_size\":\"100\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.node.network.errors AND metric.attributes.direction: receive\",\"language\":\"kuery\"}}}"},"id":"977705e0-78e1-11f0-aa07-c5df75106bf1","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:19:26.152Z","version":"WzkwOSwzXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Node Network IO Transmit","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Node Network IO Transmit\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#54B399\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"unit\":\"\",\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"positive_rate\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"axis_scale\":\"normal\",\"formatter\":\"bytes\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Node Network IO\",\"type\":\"timeseries\",\"terms_field\":\"resource.attributes.k8s@node@name\",\"terms_order_by\":\"_key\",\"value_template\":\"{{value}}\",\"filter\":{\"query\":\"\",\"language\":\"kuery\"},\"terms_size\":\"100\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.node.network.io AND metric.attributes.direction: transmit\",\"language\":\"kuery\"}}}"},"id":"f591b2f0-9318-11f0-99d5-2148ceb81347","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:20:15.674Z","version":"WzkyMSwzXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Node Network Errors Transmit","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Node Network Errors Transmit\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#54B399\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"positive_rate\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"axis_scale\":\"normal\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Node Network Errors\",\"type\":\"timeseries\",\"terms_field\":\"resource.attributes.k8s@node@name\",\"terms_order_by\":\"_key\",\"terms_size\":\"100\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.node.network.errors AND metric.attributes.direction: transmit\",\"language\":\"kuery\"}}}"},"id":"11106d00-9319-11f0-8b65-db951fa546e1","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:20:52.849Z","version":"WzkzNCwzXQ=="} -{"attributes":{"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[]}"},"optionsJSON":"{\"hidePanelTitles\":false,\"useMargins\":true}","panelsJSON":"[{\"embeddableConfig\":{\"hidePanelTitles\":false},\"gridData\":{\"h\":20,\"i\":\"b9aeea87-2caa-4c43-bd3e-707715b7e396\",\"w\":21,\"x\":0,\"y\":0},\"panelIndex\":\"b9aeea87-2caa-4c43-bd3e-707715b7e396\",\"title\":\"Nodes\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_0\"},{\"embeddableConfig\":{\"hidePanelTitles\":false},\"gridData\":{\"h\":20,\"i\":\"131bf4fb-e253-4e66-9f5f-756cae4ea247\",\"w\":27,\"x\":21,\"y\":0},\"panelIndex\":\"131bf4fb-e253-4e66-9f5f-756cae4ea247\",\"title\":\"Container Resources\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_1\"},{\"embeddableConfig\":{\"hidePanelTitles\":false},\"gridData\":{\"h\":15,\"i\":\"7b9841c5-4420-406d-947b-215e8a69c300\",\"w\":24,\"x\":0,\"y\":20},\"panelIndex\":\"7b9841c5-4420-406d-947b-215e8a69c300\",\"title\":\"Amount of Nodes\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_2\"},{\"embeddableConfig\":{},\"gridData\":{\"h\":15,\"i\":\"e97361dd-8ed4-4a34-b2a6-ecbbec424d74\",\"w\":24,\"x\":24,\"y\":20},\"panelIndex\":\"e97361dd-8ed4-4a34-b2a6-ecbbec424d74\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_3\"},{\"embeddableConfig\":{},\"gridData\":{\"h\":15,\"i\":\"97f3ae44-2376-453b-becf-ea4140b77c7d\",\"w\":24,\"x\":0,\"y\":35},\"panelIndex\":\"97f3ae44-2376-453b-becf-ea4140b77c7d\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_4\"},{\"embeddableConfig\":{},\"gridData\":{\"h\":15,\"i\":\"08421449-b5ff-4f64-9409-46852d7a241c\",\"w\":24,\"x\":24,\"y\":35},\"panelIndex\":\"08421449-b5ff-4f64-9409-46852d7a241c\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_5\"},{\"embeddableConfig\":{\"hidePanelTitles\":false},\"gridData\":{\"h\":15,\"i\":\"4c6f8098-dc08-495b-8fdb-2c8dd80d7212\",\"w\":24,\"x\":0,\"y\":50},\"panelIndex\":\"4c6f8098-dc08-495b-8fdb-2c8dd80d7212\",\"title\":\"Node Memory Working Set\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_6\"},{\"embeddableConfig\":{\"hidePanelTitles\":false},\"gridData\":{\"h\":15,\"i\":\"d59b6f1a-18cf-48f1-8f61-96ec0a98412b\",\"w\":24,\"x\":24,\"y\":50},\"panelIndex\":\"d59b6f1a-18cf-48f1-8f61-96ec0a98412b\",\"title\":\"Node Memory RSS\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_7\"},{\"embeddableConfig\":{},\"gridData\":{\"h\":16,\"i\":\"fed42e06-6338-431b-b457-60114c107396\",\"w\":24,\"x\":0,\"y\":65},\"panelIndex\":\"fed42e06-6338-431b-b457-60114c107396\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_8\"},{\"embeddableConfig\":{},\"gridData\":{\"h\":16,\"i\":\"f17e1a62-2deb-43b1-9229-3eae456f0863\",\"w\":24,\"x\":24,\"y\":65},\"panelIndex\":\"f17e1a62-2deb-43b1-9229-3eae456f0863\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_9\"},{\"embeddableConfig\":{\"hidePanelTitles\":false},\"gridData\":{\"h\":15,\"i\":\"4c0f9404-758a-4e73-bb8b-3f99e3eaa7d9\",\"w\":24,\"x\":0,\"y\":81},\"panelIndex\":\"4c0f9404-758a-4e73-bb8b-3f99e3eaa7d9\",\"title\":\"Node Network IO Receive\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_10\"},{\"embeddableConfig\":{\"hidePanelTitles\":false},\"gridData\":{\"h\":15,\"i\":\"984d0e45-5c96-4590-90aa-7aac50f41e80\",\"w\":24,\"x\":24,\"y\":81},\"panelIndex\":\"984d0e45-5c96-4590-90aa-7aac50f41e80\",\"title\":\"Node Network Errors Receive\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_11\"},{\"embeddableConfig\":{\"hidePanelTitles\":false},\"gridData\":{\"h\":15,\"i\":\"2f244a43-be22-48b3-b56e-3dd75525d97b\",\"w\":24,\"x\":0,\"y\":96},\"panelIndex\":\"2f244a43-be22-48b3-b56e-3dd75525d97b\",\"title\":\"Node Network IO Transmit\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_12\"},{\"embeddableConfig\":{\"hidePanelTitles\":false},\"gridData\":{\"h\":15,\"i\":\"10e49f55-401b-48f9-b8e5-b098069cc1ba\",\"w\":24,\"x\":24,\"y\":96},\"panelIndex\":\"10e49f55-401b-48f9-b8e5-b098069cc1ba\",\"title\":\"Node Network Errors Transmit\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_13\"}]","timeRestore":false,"title":"K8s Nodes","version":1},"id":"4ad17100-80c4-11ef-9853-17e21a5b68d5","migrationVersion":{"dashboard":"7.9.3"},"references":[{"id":"ebdb2f00-9776-11ef-a1d8-af81cc9045d1","name":"panel_0","type":"visualization"},{"id":"a8722220-96d4-11ef-b4ed-536715f658e8","name":"panel_1","type":"visualization"},{"id":"9bd3f780-80c4-11ef-b7e6-d7428a02bdd1","name":"panel_2","type":"visualization"},{"id":"ddc86020-7f81-11ef-aad9-b55b1b4895ab","name":"panel_3","type":"visualization"},{"id":"e279cfb0-7f80-11ef-aad9-b55b1b4895ab","name":"panel_4","type":"visualization"},{"id":"535828d0-7f81-11ef-aad9-b55b1b4895ab","name":"panel_5","type":"visualization"},{"id":"eec8ff10-84bd-11ef-9853-17e21a5b68d5","name":"panel_6","type":"visualization"},{"id":"f2aa09d0-84bd-11ef-b7e6-d7428a02bdd1","name":"panel_7","type":"visualization"},{"id":"21169010-7f76-11ef-9899-37d721678cf0","name":"panel_8","type":"visualization"},{"id":"bee22630-7f83-11ef-aad9-b55b1b4895ab","name":"panel_9","type":"visualization"},{"id":"13e784c0-78e1-11f0-b9ad-d91d395f9528","name":"panel_10","type":"visualization"},{"id":"977705e0-78e1-11f0-aa07-c5df75106bf1","name":"panel_11","type":"visualization"},{"id":"f591b2f0-9318-11f0-99d5-2148ceb81347","name":"panel_12","type":"visualization"},{"id":"11106d00-9319-11f0-8b65-db951fa546e1","name":"panel_13","type":"visualization"}],"type":"dashboard","updated_at":"2025-09-16T16:23:44.862Z","version":"Wzk0MywzXQ=="} -{"exportedCount":18,"missingRefCount":0,"missingReferences":[]} \ No newline at end of file +{"attributes":{"fields":"[{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"_id\"],\"name\":\"_id\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"_index\"],\"name\":\"_index\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":false,\"count\":0,\"name\":\"_score\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":false,\"type\":\"number\"},{\"aggregatable\":false,\"count\":0,\"esTypes\":[\"_source\"],\"name\":\"_source\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":false,\"type\":\"_source\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"_type\"],\"name\":\"_type\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"aggregationTemporality\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"app_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"app_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"bucketCounts\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"bucketCountsList\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"buckets.count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"buckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"buckets.max\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"buckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"buckets.min\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"buckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":false,\"count\":0,\"esTypes\":[\"text\"],\"name\":\"description\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"exemplars.spanId\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"exemplars\"}},\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"date_nanos\"],\"name\":\"exemplars.time\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"exemplars\"}},\"type\":\"date\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"exemplars.traceId\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"exemplars\"}},\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"exemplars.value\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"exemplars\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"explicitBounds\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"explicitBoundsCount\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"flags\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"instance_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"instrumentationLibrary.name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"instrumentationLibrary.version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"instrumentationScope.name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"instrumentationScope.version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"boolean\"],\"name\":\"isMonotonic\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"boolean\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"kind\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.app@payment@currency\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.build_date\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.cause\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.cluster_type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.component\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.connection_security_policy\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.core\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.cpu\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.currency_code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.debug\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_app\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_canonical_revision\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_canonical_service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_cluster\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_principal\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_service_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_service_namespace\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_workload\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_workload_namespace\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.device\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.direction\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.endpoint\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.err\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.event\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.family\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.format\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.group\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.grpc_code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.grpc_method\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.grpc_response_status\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.grpc_service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.grpc_type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.host\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.http@flavor\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.http@host\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.http@method\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.http@scheme\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.http@status_code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":4,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.interface\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.method\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.model\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.net@host@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.net@host@port\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.net@peer@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.net@peer@port\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.net@protocol@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.net@protocol@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.node\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.nodejs@eventloop@state\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.operation\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.otel_scope_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.otel_scope_version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.path\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.protocol\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.reason\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.recommendation@type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.reporter\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.request_protocol\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.resource\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.response_code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.response_flags\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.result\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.revision\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.rpc@grpc@status_code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.rpc@method\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.rpc@service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.rpc@system\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.sampler_type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_app\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_canonical_revision\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_canonical_service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_cluster\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_principal\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_workload\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_workload_namespace\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.state\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.status\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.svc\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.tag\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.target\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.transport\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":2,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.v8js@gc@type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.v8js@heap@space@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"negative\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"negativeBuckets.count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"negativeBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"negativeBuckets.max\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"negativeBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"negativeBuckets.min\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"negativeBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"negativeOffset\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"organization_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"organization_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"positive\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"positiveBuckets.count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"positiveBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"positiveBuckets.max\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"positiveBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"positiveBuckets.min\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"positiveBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"positiveOffset\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"process_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"process_type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"quantileValuesCount\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"quantiles.quantile\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"quantiles\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"quantiles.value\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"quantiles\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.cloud@account@id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.cloud@availability_zone\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.cloud@platform\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.cloud@provider\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.cloud@region\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.container@id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.container@image@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.container@image@tag\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.host@arch\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.host@id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.host@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.host@type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.http@scheme\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":2,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@cluster@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":6,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@container@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@daemonset@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@daemonset@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@deployment@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@deployment@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@job@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@job@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":1,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@namespace@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":1,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@node@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@node@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@persistentvolumeclaim@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@pod@ip\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":11,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@pod@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@pod@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@replicaset@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@resource@group\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@resource@kind\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@resource@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@resource@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@statefulset@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@statefulset@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@volume@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@volume@type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.net@host@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.net@host@port\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.os@description\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.os@type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.os@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@command\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@command_args\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@executable@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@executable@path\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@owner\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@pid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@runtime@description\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@runtime@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@runtime@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@app_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@app_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@instance_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@org_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@org_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@process@id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@process@type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@space_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@space_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.server@address\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.server@port\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":1,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service@instance@id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service@namespace\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry@auto@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry@sdk@language\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry@sdk@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry@sdk@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry_sdk_language\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry_sdk_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry_sdk_version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.url@scheme\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"scale\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"schemaUrl\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"serviceName\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"space_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"space_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"date_nanos\"],\"name\":\"startTime\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"date\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"sum\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"date_nanos\"],\"name\":\"time\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"date\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"unit\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"value\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"zeroCount\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"}]","timeFieldName":"time","title":"metrics-otel-v1-*"},"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","migrationVersion":{"index-pattern":"7.6.0"},"references":[],"type":"index-pattern","updated_at":"2025-09-16T16:33:59.060Z","version":"Wzk2MCwzXQ=="} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"title":"Clusters","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}","version":1,"visState":"{\"title\":\"Clusters\",\"type\":\"table\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"cardinality\",\"params\":{\"field\":\"resource.attributes.k8s@namespace@name\",\"customLabel\":\"Namespaces\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@cluster@name\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\"},\"schema\":\"bucket\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"fdba71f0-950e-11ef-a1d8-af81cc9045d1","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"visualization","updated_at":"2025-03-31T06:58:43.928Z","version":"WzU0LDFd"} +{"attributes":{"columns":["_source"],"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"highlightAll\":true,\"version\":true,\"query\":{\"query\":\"name: k8s.deployment.* or name: k8s.job.* or name: k8s.statefulset.* or name: k8s.daemonset.*\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"sort":[],"title":"K8s Workloads","version":1},"id":"07fe9710-9512-11ef-a1d8-af81cc9045d1","migrationVersion":{"search":"7.9.3"},"references":[{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"search","updated_at":"2025-03-31T06:58:43.928Z","version":"WzU1LDFd"} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"Workload list2","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}","version":1,"visState":"{\"title\":\"Workload list2\",\"type\":\"table\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"cardinality\",\"params\":{\"field\":\"resource.attributes.k8s@deployment@name\",\"customLabel\":\"Deployments\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@namespace@name\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":996,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Namespaces\"},\"schema\":\"bucket\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"cardinality\",\"params\":{\"field\":\"resource.attributes.k8s@statefulset@name\",\"customLabel\":\"StatefulSet\"},\"schema\":\"metric\"},{\"id\":\"4\",\"enabled\":true,\"type\":\"cardinality\",\"params\":{\"field\":\"resource.attributes.k8s@daemonset@name\",\"customLabel\":\"DaemonSets\"},\"schema\":\"metric\"},{\"id\":\"5\",\"enabled\":true,\"type\":\"cardinality\",\"params\":{\"field\":\"resource.attributes.k8s@job@name\",\"customLabel\":\"Jobs\"},\"schema\":\"metric\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"878b8140-954a-11ef-a1d8-af81cc9045d1","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"07fe9710-9512-11ef-a1d8-af81cc9045d1","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2025-03-31T06:58:43.928Z","version":"WzU2LDFd"} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Workloard Chart","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Workloard Chart\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"everything\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"cardinality\",\"field\":\"resource.attributes.k8s@deployment@name\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Deployments\",\"type\":\"timeseries\",\"filter\":{\"query\":\"name: k8s.deployment.available\",\"language\":\"kuery\"}},{\"id\":\"63c2b270-9549-11ef-9761-9b02e78a949d\",\"color\":\"rgba(96,146,192,1)\",\"split_mode\":\"everything\",\"metrics\":[{\"id\":\"63c2b271-9549-11ef-9761-9b02e78a949d\",\"type\":\"cardinality\",\"field\":\"resource.attributes.k8s@statefulset@name\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"StatefulSets\",\"type\":\"timeseries\",\"filter\":{\"query\":\"name: k8s.statefulset.desired_pods\",\"language\":\"kuery\"}},{\"id\":\"b855b2b0-9549-11ef-9761-9b02e78a949d\",\"color\":\"rgba(202,142,174,1)\",\"split_mode\":\"everything\",\"metrics\":[{\"id\":\"b855b2b1-9549-11ef-9761-9b02e78a949d\",\"type\":\"cardinality\",\"field\":\"resource.attributes.k8s@daemonset@name\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"DaemonSets\",\"type\":\"timeseries\",\"filter\":{\"query\":\"name: k8s.daemonset.desired_scheduled_nodes\",\"language\":\"kuery\"}},{\"id\":\"d80ff390-9549-11ef-9761-9b02e78a949d\",\"color\":\"rgba(185,168,136,1)\",\"split_mode\":\"everything\",\"metrics\":[{\"id\":\"d80ff391-9549-11ef-9761-9b02e78a949d\",\"type\":\"cardinality\",\"field\":\"resource.attributes.k8s@job@name\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Jobs\",\"type\":\"timeseries\",\"filter\":{\"query\":\"name: k8s.job.desired_successful_pods\",\"language\":\"kuery\"}}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.*\",\"language\":\"kuery\"}}}"},"id":"7ef9b520-9549-11ef-b4ed-536715f658e8","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-03-31T06:58:43.928Z","version":"WzU3LDFd"} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"deployments2","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":0,\"direction\":\"asc\"}}}}","version":1,"visState":"{\"title\":\"deployments2\",\"type\":\"table\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"1-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.deployment.desired\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"1-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Desired Pods\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@deployment@name\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Deployments\"},\"schema\":\"bucket\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"3-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.deployment.available\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"3-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Available Pods\"},\"schema\":\"metric\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"17c26560-96ce-11ef-a1d8-af81cc9045d1","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"07fe9710-9512-11ef-a1d8-af81cc9045d1","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2025-03-31T06:58:43.928Z","version":"WzU4LDFd"} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"deployments2 (copy)","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":0,\"direction\":\"asc\"}}}}","version":1,"visState":"{\"title\":\"deployments2 (copy)\",\"type\":\"table\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"1-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.statefulset.desired_pods\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"1-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Desired Pods\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@statefulset@name\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"StatefulSets\"},\"schema\":\"bucket\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"3-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.statefulset.ready_pods\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"3-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Available Pods\"},\"schema\":\"metric\"},{\"id\":\"4\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"4-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.statefulset.current_pods\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"4-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Current Pods\"},\"schema\":\"metric\"},{\"id\":\"5\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"5-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.statefulset.updated_pods\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"5-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Updated Pods\"},\"schema\":\"metric\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"d7d514b0-96ce-11ef-b4ed-536715f658e8","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"07fe9710-9512-11ef-a1d8-af81cc9045d1","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2025-03-31T06:58:43.928Z","version":"WzU5LDFd"} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"deployments2 (copy 1)","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":0,\"direction\":\"asc\"}}}}","version":1,"visState":"{\"title\":\"deployments2 (copy 1)\",\"type\":\"table\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"1-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.daemonset.desired_scheduled_nodes\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"1-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Desired Nodes\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@daemonset@name\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"DaemonSets\"},\"schema\":\"bucket\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"3-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.daemonset.ready_nodes\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"3-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Ready Nodes\"},\"schema\":\"metric\"},{\"id\":\"5\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"5-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.daemonset.current_scheduled_nodes\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"5-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Scheduled Nodes\"},\"schema\":\"metric\"},{\"id\":\"4\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"4-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.daemonset.misscheduled_nodes\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"4-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"customLabel\":\"\"}},\"customLabel\":\"Misscheduled Nodes\"},\"schema\":\"metric\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"f6afeab0-96d1-11ef-b4ed-536715f658e8","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"07fe9710-9512-11ef-a1d8-af81cc9045d1","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2025-03-31T06:58:43.928Z","version":"WzYwLDFd"} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"deployments2 (copy 2)","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":0,\"direction\":\"asc\"}}}}","version":1,"visState":"{\"title\":\"deployments2 (copy 2)\",\"type\":\"table\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"1-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.job.desired_successful_pods\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"1-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Desired Pods\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@job@name\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Jobs\"},\"schema\":\"bucket\"},{\"id\":\"4\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"4-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.job.successful_pods\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"4-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"customLabel\":\"\"}},\"customLabel\":\"Successful Pods\"},\"schema\":\"metric\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"3-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.job.active_pods\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"3-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Active Pods\"},\"schema\":\"metric\"},{\"id\":\"5\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"5-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.job.failed_pods\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"5-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Failed Pods\"},\"schema\":\"metric\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"2029a240-96d3-11ef-a1d8-af81cc9045d1","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"07fe9710-9512-11ef-a1d8-af81cc9045d1","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2025-03-31T06:58:43.928Z","version":"WzYxLDFd"} +{"attributes":{"columns":["_source"],"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"highlightAll\":true,\"version\":true,\"query\":{\"query\":\" name: k8s.container.* or name: container.*\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"sort":[],"title":"K8s Container","version":1},"id":"d6b5f6d0-96d3-11ef-a1d8-af81cc9045d1","migrationVersion":{"search":"7.9.3"},"references":[{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"search","updated_at":"2025-09-16T16:33:59.060Z","version":"Wzk2MywzXQ=="} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"container resources","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":0,\"direction\":\"asc\"}}}}","version":1,"visState":"{\"title\":\"container resources\",\"type\":\"table\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"1-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.container.cpu_request\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"1-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"customLabel\":\"\"}},\"customLabel\":\"CPU Request\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@pod@name\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":997,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Pod\"},\"schema\":\"bucket\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@container@name\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Container\"},\"schema\":\"bucket\"},{\"id\":\"8\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"8-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: container.cpu.usage\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"8-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"CPU Usage\"},\"schema\":\"metric\"},{\"id\":\"4\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"4-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.container.cpu_limit\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"4-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"CPU Limit\"},\"schema\":\"metric\"},{\"id\":\"5\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"5-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.container.memory_request\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"5-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"json\":\"{\\n \\\"script\\\": {\\n \\\"inline\\\": \\\"doc['value'].value / 1024 / 1024\\\",\\n \\\"lang\\\": \\\"painless\\\"\\n }\\n}\"}},\"json\":\"\",\"customLabel\":\"Memory Request (MB)\"},\"schema\":\"metric\"},{\"id\":\"7\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"7-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: container.memory.working_set\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"7-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"json\":\"{\\n \\\"script\\\": {\\n \\\"inline\\\": \\\"doc['value'].value / 1024 / 1024\\\",\\n \\\"lang\\\": \\\"painless\\\"\\n }\\n}\"}},\"customLabel\":\"Memory Usage (MB)\"},\"schema\":\"metric\"},{\"id\":\"6\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"6-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.container.memory_limit\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"6-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"json\":\"{\\n \\\"script\\\": {\\n \\\"inline\\\": \\\"doc['value'].value / 1024 / 1024\\\",\\n \\\"lang\\\": \\\"painless\\\"\\n }\\n}\"}},\"customLabel\":\"Memory Limit (MB)\"},\"schema\":\"metric\"},{\"id\":\"9\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"9-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name:k8s.container.restarts\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"9-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"customLabel\":\"Restarts\"}},\"customLabel\":\"Restarts\"},\"schema\":\"metric\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":true,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"a8722220-96d4-11ef-b4ed-536715f658e8","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"d6b5f6d0-96d3-11ef-a1d8-af81cc9045d1","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2025-09-16T16:33:59.060Z","version":"Wzk2NCwzXQ=="} +{"attributes":{"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[]}"},"optionsJSON":"{\"hidePanelTitles\":false,\"useMargins\":true}","panelsJSON":"[{\"version\":\"1.3.18\",\"gridData\":{\"x\":0,\"y\":0,\"w\":14,\"h\":16,\"i\":\"f57d14bb-fa23-40ec-96b5-af20d521ca2d\"},\"panelIndex\":\"f57d14bb-fa23-40ec-96b5-af20d521ca2d\",\"embeddableConfig\":{},\"panelRefName\":\"panel_0\"},{\"version\":\"1.3.18\",\"gridData\":{\"x\":14,\"y\":0,\"w\":16,\"h\":16,\"i\":\"dae768dc-0ff2-4106-a44a-ddf859186e19\"},\"panelIndex\":\"dae768dc-0ff2-4106-a44a-ddf859186e19\",\"embeddableConfig\":{\"title\":\"Namespaces\",\"hidePanelTitles\":false},\"title\":\"Namespaces\",\"panelRefName\":\"panel_1\"},{\"version\":\"1.3.18\",\"gridData\":{\"x\":30,\"y\":0,\"w\":18,\"h\":16,\"i\":\"fcf90cfd-6103-4056-9797-99d63742e31a\"},\"panelIndex\":\"fcf90cfd-6103-4056-9797-99d63742e31a\",\"embeddableConfig\":{\"title\":\"Workloads over Time\",\"hidePanelTitles\":false},\"title\":\"Workloads over Time\",\"panelRefName\":\"panel_2\"},{\"version\":\"1.3.18\",\"gridData\":{\"x\":0,\"y\":16,\"w\":12,\"h\":23,\"i\":\"98298537-dd2d-4eff-9789-071d74158827\"},\"panelIndex\":\"98298537-dd2d-4eff-9789-071d74158827\",\"embeddableConfig\":{\"title\":\"Deployments\",\"hidePanelTitles\":false,\"table\":null,\"vis\":{\"params\":{\"sort\":{\"columnIndex\":0,\"direction\":\"asc\"}}}},\"title\":\"Deployments\",\"panelRefName\":\"panel_3\"},{\"version\":\"1.3.18\",\"gridData\":{\"x\":12,\"y\":16,\"w\":12,\"h\":23,\"i\":\"7b47e126-84a5-4a89-84a2-d7be15e9a0d0\"},\"panelIndex\":\"7b47e126-84a5-4a89-84a2-d7be15e9a0d0\",\"embeddableConfig\":{\"title\":\"StatefulSets\",\"hidePanelTitles\":false,\"table\":null,\"vis\":{\"params\":{\"sort\":{\"columnIndex\":0,\"direction\":\"asc\"}}}},\"title\":\"StatefulSets\",\"panelRefName\":\"panel_4\"},{\"version\":\"1.3.18\",\"gridData\":{\"x\":24,\"y\":16,\"w\":12,\"h\":23,\"i\":\"b5cfabc3-5a7b-401e-a11f-3a9416afc8e8\"},\"panelIndex\":\"b5cfabc3-5a7b-401e-a11f-3a9416afc8e8\",\"embeddableConfig\":{\"title\":\"DaemonSets\",\"hidePanelTitles\":false,\"table\":null,\"vis\":{\"params\":{\"sort\":{\"columnIndex\":0,\"direction\":\"asc\"}}}},\"title\":\"DaemonSets\",\"panelRefName\":\"panel_5\"},{\"version\":\"1.3.18\",\"gridData\":{\"x\":36,\"y\":16,\"w\":12,\"h\":23,\"i\":\"f402fe32-8ab8-446c-9b1b-c0a0e13f28c3\"},\"panelIndex\":\"f402fe32-8ab8-446c-9b1b-c0a0e13f28c3\",\"embeddableConfig\":{\"title\":\"Jobs\",\"hidePanelTitles\":false,\"table\":null,\"vis\":{\"params\":{\"sort\":{\"columnIndex\":0,\"direction\":\"asc\"}}}},\"title\":\"Jobs\",\"panelRefName\":\"panel_6\"},{\"version\":\"1.3.18\",\"gridData\":{\"x\":0,\"y\":39,\"w\":36,\"h\":19,\"i\":\"b335c289-53d2-4190-8255-d228fe28503c\"},\"panelIndex\":\"b335c289-53d2-4190-8255-d228fe28503c\",\"embeddableConfig\":{\"title\":\"Container Resources\",\"hidePanelTitles\":false},\"title\":\"Container Resources\",\"panelRefName\":\"panel_7\"}]","timeRestore":false,"title":"K8s Workloads","version":1},"id":"c69f3210-950d-11ef-a1d8-af81cc9045d1","migrationVersion":{"dashboard":"7.9.3"},"references":[{"id":"fdba71f0-950e-11ef-a1d8-af81cc9045d1","name":"panel_0","type":"visualization"},{"id":"878b8140-954a-11ef-a1d8-af81cc9045d1","name":"panel_1","type":"visualization"},{"id":"7ef9b520-9549-11ef-b4ed-536715f658e8","name":"panel_2","type":"visualization"},{"id":"17c26560-96ce-11ef-a1d8-af81cc9045d1","name":"panel_3","type":"visualization"},{"id":"d7d514b0-96ce-11ef-b4ed-536715f658e8","name":"panel_4","type":"visualization"},{"id":"f6afeab0-96d1-11ef-b4ed-536715f658e8","name":"panel_5","type":"visualization"},{"id":"2029a240-96d3-11ef-a1d8-af81cc9045d1","name":"panel_6","type":"visualization"},{"id":"a8722220-96d4-11ef-b4ed-536715f658e8","name":"panel_7","type":"visualization"}],"type":"dashboard","updated_at":"2025-03-31T06:58:43.928Z","version":"WzYyLDFd"} +{"attributes":{"columns":["value","resource.attributes.k8s@pod@name"],"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"highlightAll\":true,\"version\":true,\"query\":{\"query\":\"name:k8s.volume.capacity\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"sort":[],"title":"[OTel] K8s Volume Capacity","version":1},"id":"1e517ef0-8a32-11ef-aad9-b55b1b4895ab","migrationVersion":{"search":"7.9.3"},"references":[{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"search","updated_at":"2025-03-31T06:58:53.424Z","version":"WzYzLDFd"} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"PVC List","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}","version":1,"visState":"{\"title\":\"PVC List\",\"type\":\"table\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"json\":\"{\\n \\\"script\\\" : {\\n \\\"inline\\\": \\\"doc['value'].value / 1024 / 1024\\\",\\n \\\"lang\\\": \\\"painless\\\"\\n }\\n}\",\"customLabel\":\"Capacity GB\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@cluster@name\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Cluster\"},\"schema\":\"bucket\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@namespace@name\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Namespace\"},\"schema\":\"bucket\"},{\"id\":\"4\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.service@name\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Workload\"},\"schema\":\"bucket\"},{\"id\":\"5\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@persistentvolumeclaim@name\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Volume\"},\"schema\":\"bucket\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"b792e4f0-8a32-11ef-aad9-b55b1b4895ab","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"1e517ef0-8a32-11ef-aad9-b55b1b4895ab","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2025-03-31T06:58:53.424Z","version":"WzY0LDFd"} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"K8s Volume Available","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"K8s Volume Available\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"max\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"bytes\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"filter\":{\"query\":\"name: k8s.volume.available\",\"language\":\"kuery\"},\"terms_field\":\"resource.attributes.k8s@persistentvolumeclaim@name\",\"label\":\"\",\"type\":\"timeseries\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false}}"},"id":"123ab780-8a41-11ef-9899-37d721678cf0","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-03-31T06:58:53.424Z","version":"WzY1LDFd"} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"K8s Volume Available (copy)","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"K8s Volume Available (copy)\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"max\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"bytes\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"filter\":{\"query\":\"name: k8s.volume.capacity\",\"language\":\"kuery\"},\"terms_field\":\"resource.attributes.k8s@persistentvolumeclaim@name\",\"label\":\"\",\"type\":\"timeseries\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false}}"},"id":"1b2ebd50-8a41-11ef-aad9-b55b1b4895ab","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-03-31T06:58:53.424Z","version":"WzY2LDFd"} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"K8s Volume Available (copy 1)","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"K8s Volume Available (copy 1)\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"max\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"filter\":{\"query\":\"name: k8s.volume.inodes.free\",\"language\":\"kuery\"},\"terms_field\":\"resource.attributes.k8s@persistentvolumeclaim@name\",\"label\":\"\",\"type\":\"timeseries\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false}}"},"id":"5e7d6c00-8a41-11ef-aad9-b55b1b4895ab","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-03-31T06:58:53.424Z","version":"WzY3LDFd"} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"K8s Volume Available (copy 2)","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"K8s Volume Available (copy 2)\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"max\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"filter\":{\"query\":\"name: k8s.volume.inodes\",\"language\":\"kuery\"},\"terms_field\":\"resource.attributes.k8s@persistentvolumeclaim@name\",\"label\":\"\",\"type\":\"timeseries\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false}}"},"id":"777c5900-8a41-11ef-9899-37d721678cf0","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-03-31T06:58:53.424Z","version":"WzY4LDFd"} +{"attributes":{"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[]}"},"optionsJSON":"{\"hidePanelTitles\":false,\"useMargins\":true}","panelsJSON":"[{\"embeddableConfig\":{\"hidePanelTitles\":false,\"title\":\"Volume List\"},\"gridData\":{\"h\":9,\"i\":\"2f25d4a8-99ff-4e84-a8da-1740f02e4524\",\"w\":30,\"x\":0,\"y\":0},\"panelIndex\":\"2f25d4a8-99ff-4e84-a8da-1740f02e4524\",\"title\":\"Volume List\",\"version\":\"1.3.18\",\"panelRefName\":\"panel_0\"},{\"embeddableConfig\":{\"hidePanelTitles\":false,\"title\":\"Volume Available\"},\"gridData\":{\"h\":11,\"i\":\"fa00a682-abbd-4deb-9f51-6c07809d1e33\",\"w\":24,\"x\":0,\"y\":9},\"panelIndex\":\"fa00a682-abbd-4deb-9f51-6c07809d1e33\",\"title\":\"Volume Available\",\"version\":\"1.3.18\",\"panelRefName\":\"panel_1\"},{\"embeddableConfig\":{\"hidePanelTitles\":false,\"title\":\"Volume Capacity\"},\"gridData\":{\"h\":11,\"i\":\"524534a8-db24-47e2-baf1-c86e83850f71\",\"w\":24,\"x\":24,\"y\":9},\"panelIndex\":\"524534a8-db24-47e2-baf1-c86e83850f71\",\"title\":\"Volume Capacity\",\"version\":\"1.3.18\",\"panelRefName\":\"panel_2\"},{\"embeddableConfig\":{\"hidePanelTitles\":false,\"title\":\"inodes free\"},\"gridData\":{\"h\":11,\"i\":\"2382926c-e2e2-49ef-bffc-02a1f1dc21de\",\"w\":24,\"x\":0,\"y\":20},\"panelIndex\":\"2382926c-e2e2-49ef-bffc-02a1f1dc21de\",\"title\":\"inodes free\",\"version\":\"1.3.18\",\"panelRefName\":\"panel_3\"},{\"embeddableConfig\":{\"hidePanelTitles\":false,\"title\":\"inodes total\"},\"gridData\":{\"h\":11,\"i\":\"2008bf73-29d5-4d4f-9787-17bf1c523e73\",\"w\":23,\"x\":24,\"y\":20},\"panelIndex\":\"2008bf73-29d5-4d4f-9787-17bf1c523e73\",\"title\":\"inodes total\",\"version\":\"1.3.18\",\"panelRefName\":\"panel_4\"}]","timeRestore":false,"title":"K8s Volumes","version":1},"id":"433f93e0-8a33-11ef-aad9-b55b1b4895ab","migrationVersion":{"dashboard":"7.9.3"},"references":[{"id":"b792e4f0-8a32-11ef-aad9-b55b1b4895ab","name":"panel_0","type":"visualization"},{"id":"123ab780-8a41-11ef-9899-37d721678cf0","name":"panel_1","type":"visualization"},{"id":"1b2ebd50-8a41-11ef-aad9-b55b1b4895ab","name":"panel_2","type":"visualization"},{"id":"5e7d6c00-8a41-11ef-aad9-b55b1b4895ab","name":"panel_3","type":"visualization"},{"id":"777c5900-8a41-11ef-9899-37d721678cf0","name":"panel_4","type":"visualization"}],"type":"dashboard","updated_at":"2025-03-31T06:58:53.424Z","version":"WzY5LDFd"} +{"attributes":{"columns":["_source"],"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"highlightAll\":true,\"version\":true,\"query\":{\"query\":\"name: k8s.node.* or name: k8s.pod.*\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"sort":[],"title":"K8s Nodes","version":1},"id":"c9f6b090-9775-11ef-a1d8-af81cc9045d1","migrationVersion":{"search":"7.9.3"},"references":[{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"search","updated_at":"2025-09-16T16:33:59.060Z","version":"Wzk2MSwzXQ=="} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"Nodes list2","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}","version":1,"visState":"{\"title\":\"Nodes list2\",\"type\":\"table\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"cardinality\",\"params\":{\"field\":\"resource.attributes.k8s@pod@name\",\"customLabel\":\"Pods\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@node@name\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Nodes\"},\"schema\":\"bucket\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"3-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.node.memory.working_set\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"3-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"json\":\"{\\n \\\"script\\\": {\\n \\\"inline\\\": \\\"doc['value'].value / 1024 / 1024 / 1024\\\",\\n \\\"lang\\\": \\\"painless\\\"\\n }\\n}\"}},\"customLabel\":\"Memory Usage (WSS)(GB)\"},\"schema\":\"metric\"},{\"id\":\"4\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"4-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.node.memory.available\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"4-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"json\":\"{\\n \\\"script\\\": {\\n \\\"inline\\\": \\\"doc['value'].value / 1024 / 1024 / 1024\\\",\\n \\\"lang\\\": \\\"painless\\\"\\n }\\n}\"}},\"customLabel\":\"Memory Available (GB)\"},\"schema\":\"metric\"},{\"id\":\"5\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"5-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.node.cpu.usage\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"5-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"json\":\"\"}},\"customLabel\":\"CPU Usage\"},\"schema\":\"metric\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"ebdb2f00-9776-11ef-a1d8-af81cc9045d1","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"c9f6b090-9775-11ef-a1d8-af81cc9045d1","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2025-09-16T16:33:59.060Z","version":"Wzk2MiwzXQ=="} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Amount of Nodes","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Amount of Nodes\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"cardinality\",\"field\":\"resource.attributes.k8s@node@name\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Nodes\",\"type\":\"timeseries\",\"split_filters\":[{\"color\":\"#68BC00\",\"id\":\"a03ac6d0-7f81-11ef-a705-4fe2f82b8e2c\",\"filter\":{\"query\":\"\",\"language\":\"kuery\"}}],\"terms_field\":null,\"value_template\":\"{{value}}\",\"offset_time\":\"1m\",\"hide_in_legend\":0}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.node.cpu.usage\",\"language\":\"kuery\"},\"annotations\":[{\"fields\":\"\",\"template\":\"\",\"index_pattern\":\"*\",\"query_string\":{\"query\":\"\",\"language\":\"kuery\"},\"id\":\"b081cca0-7f81-11ef-a705-4fe2f82b8e2c\",\"color\":\"#F00\",\"icon\":\"fa-tag\",\"ignore_global_filters\":1,\"ignore_panel_filters\":1}],\"drop_last_bucket\":1}}"},"id":"9bd3f780-80c4-11ef-b7e6-d7428a02bdd1","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:33:59.060Z","version":"Wzk2NSwzXQ=="} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Node CPU usage ","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Node CPU usage \",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"max\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Node CPU usage \",\"type\":\"timeseries\",\"split_filters\":[{\"color\":\"#68BC00\",\"id\":\"a03ac6d0-7f81-11ef-a705-4fe2f82b8e2c\",\"filter\":{\"query\":\"\",\"language\":\"kuery\"}}],\"terms_field\":\"resource.attributes.k8s@node@name\",\"value_template\":\"{{value}} CPU\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.node.cpu.usage\",\"language\":\"kuery\"},\"annotations\":[{\"fields\":\"\",\"template\":\"\",\"index_pattern\":\"*\",\"query_string\":{\"query\":\"\",\"language\":\"kuery\"},\"id\":\"b081cca0-7f81-11ef-a705-4fe2f82b8e2c\",\"color\":\"#F00\",\"icon\":\"fa-tag\",\"ignore_global_filters\":1,\"ignore_panel_filters\":1}]}}"},"id":"ddc86020-7f81-11ef-aad9-b55b1b4895ab","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:33:59.060Z","version":"Wzk2NiwzXQ=="} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Node Memory Usage","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Node Memory Usage\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"max\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"0.00b\",\"chart_type\":\"line\",\"line_width\":\"2\",\"point_size\":1,\"fill\":\"0.5\",\"stacked\":\"none\",\"label\":\"Node Memory Usage\",\"type\":\"timeseries\",\"terms_field\":\"resource.attributes.k8s@node@name\",\"terms_order_by\":\"61ca57f2-469d-11e7-af02-69e470af7417\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.node.memory.usage\",\"language\":\"kuery\"}}}"},"id":"e279cfb0-7f80-11ef-aad9-b55b1b4895ab","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:33:59.060Z","version":"Wzk2NywzXQ=="} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Node Memory Available","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Node Memory Available\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"max\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"'0.00b'\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Node Memory Available\",\"type\":\"timeseries\",\"terms_field\":\"resource.attributes.k8s@node@name\",\"terms_order_by\":\"61ca57f2-469d-11e7-af02-69e470af7417\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.node.memory.available\",\"language\":\"kuery\"}}}"},"id":"535828d0-7f81-11ef-aad9-b55b1b4895ab","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:33:59.060Z","version":"Wzk2OCwzXQ=="} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Node Memory Working Set","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Node Memory Working Set\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"max\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"0.00b\",\"chart_type\":\"line\",\"line_width\":\"2\",\"point_size\":1,\"fill\":\"0.5\",\"stacked\":\"none\",\"label\":\"Node Memory Usage\",\"type\":\"timeseries\",\"terms_field\":\"resource.attributes.k8s@node@name\",\"terms_order_by\":\"61ca57f2-469d-11e7-af02-69e470af7417\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.node.memory.working_set\",\"language\":\"kuery\"}}}"},"id":"eec8ff10-84bd-11ef-9853-17e21a5b68d5","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:33:59.060Z","version":"Wzk2OSwzXQ=="} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Node Memory RSS","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Node Memory RSS\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"max\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"'0.00b'\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Node Memory Available\",\"type\":\"timeseries\",\"terms_field\":\"resource.attributes.k8s@node@name\",\"terms_order_by\":\"61ca57f2-469d-11e7-af02-69e470af7417\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.node.memory.rss\",\"language\":\"kuery\"}}}"},"id":"f2aa09d0-84bd-11ef-b7e6-d7428a02bdd1","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:33:59.060Z","version":"Wzk3MCwzXQ=="} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Node Filesystem Usage","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Node Filesystem Usage\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"sigma\":\"\",\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"max\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"'0.00b'\",\"chart_type\":\"line\",\"line_width\":\"2\",\"point_size\":1,\"fill\":\"0.6\",\"stacked\":\"none\",\"label\":\"Node Filesystem Usage\",\"type\":\"timeseries\",\"terms_field\":\"resource.attributes.k8s@node@name\",\"filter\":{\"query\":\"\",\"language\":\"kuery\"},\"value_template\":\"{{value}}\",\"steps\":1,\"terms_order_by\":\"61ca57f2-469d-11e7-af02-69e470af7417\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"background_color_rules\":[{\"id\":\"31f39bb0-7f73-11ef-a705-4fe2f82b8e2c\"}],\"bar_color_rules\":[{\"id\":\"32e3c810-7f73-11ef-a705-4fe2f82b8e2c\"}],\"gauge_color_rules\":[{\"id\":\"3357bf90-7f73-11ef-a705-4fe2f82b8e2c\"}],\"gauge_width\":10,\"gauge_inner_width\":10,\"gauge_style\":\"half\",\"filter\":{\"query\":\"name: k8s.node.filesystem.usage\",\"language\":\"kuery\"},\"annotations\":[{\"fields\":\"\",\"template\":\"\",\"index_pattern\":\"*\",\"query_string\":{\"query\":\"\",\"language\":\"kuery\"},\"id\":\"e0c8bac0-7f74-11ef-a705-4fe2f82b8e2c\",\"color\":\"#F00\",\"icon\":\"fa-tag\",\"ignore_global_filters\":1,\"ignore_panel_filters\":1}]}}"},"id":"21169010-7f76-11ef-9899-37d721678cf0","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:33:59.060Z","version":"Wzk3MSwzXQ=="} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Node Filesystem Available","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Node Filesystem Available\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"max\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"'0.00b'\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Node Filesystem Available\",\"type\":\"timeseries\",\"terms_field\":\"resource.attributes.k8s@node@name\",\"terms_order_by\":\"61ca57f2-469d-11e7-af02-69e470af7417\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.node.filesystem.available\",\"language\":\"kuery\"}}}"},"id":"bee22630-7f83-11ef-aad9-b55b1b4895ab","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:33:59.060Z","version":"Wzk3MiwzXQ=="} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Node Network IO Receive","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Node Network IO Receive\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#54B399\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"unit\":\"\",\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"positive_rate\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"axis_scale\":\"normal\",\"formatter\":\"bytes\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Node Network IO\",\"type\":\"timeseries\",\"terms_field\":\"resource.attributes.k8s@node@name\",\"terms_order_by\":\"_key\",\"value_template\":\"{{value}}\",\"filter\":{\"query\":\"\",\"language\":\"kuery\"},\"terms_size\":\"100\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.node.network.io AND metric.attributes.direction: receive\",\"language\":\"kuery\"}}}"},"id":"13e784c0-78e1-11f0-b9ad-d91d395f9528","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:33:59.060Z","version":"Wzk3MywzXQ=="} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Node Network Errors Receive","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Node Network Errors Receive\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#54B399\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"positive_rate\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"axis_scale\":\"normal\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Node Network Errors\",\"type\":\"timeseries\",\"terms_field\":\"resource.attributes.k8s@node@name\",\"terms_order_by\":\"_key\",\"terms_size\":\"100\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.node.network.errors AND metric.attributes.direction: receive\",\"language\":\"kuery\"}}}"},"id":"977705e0-78e1-11f0-aa07-c5df75106bf1","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:33:59.060Z","version":"Wzk3NCwzXQ=="} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Node Network IO Transmit","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Node Network IO Transmit\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#54B399\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"unit\":\"\",\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"positive_rate\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"axis_scale\":\"normal\",\"formatter\":\"bytes\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Node Network IO\",\"type\":\"timeseries\",\"terms_field\":\"resource.attributes.k8s@node@name\",\"terms_order_by\":\"_key\",\"value_template\":\"{{value}}\",\"filter\":{\"query\":\"\",\"language\":\"kuery\"},\"terms_size\":\"100\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.node.network.io AND metric.attributes.direction: transmit\",\"language\":\"kuery\"}}}"},"id":"f591b2f0-9318-11f0-99d5-2148ceb81347","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:33:59.060Z","version":"Wzk3NSwzXQ=="} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Node Network Errors Transmit","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Node Network Errors Transmit\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#54B399\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"positive_rate\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"axis_scale\":\"normal\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Node Network Errors\",\"type\":\"timeseries\",\"terms_field\":\"resource.attributes.k8s@node@name\",\"terms_order_by\":\"_key\",\"terms_size\":\"100\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.node.network.errors AND metric.attributes.direction: transmit\",\"language\":\"kuery\"}}}"},"id":"11106d00-9319-11f0-8b65-db951fa546e1","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-09-16T16:33:59.060Z","version":"Wzk3NiwzXQ=="} +{"attributes":{"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[]}"},"optionsJSON":"{\"hidePanelTitles\":false,\"useMargins\":true}","panelsJSON":"[{\"embeddableConfig\":{\"hidePanelTitles\":false},\"gridData\":{\"h\":20,\"i\":\"b9aeea87-2caa-4c43-bd3e-707715b7e396\",\"w\":21,\"x\":0,\"y\":0},\"panelIndex\":\"b9aeea87-2caa-4c43-bd3e-707715b7e396\",\"title\":\"Nodes\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_0\"},{\"embeddableConfig\":{\"hidePanelTitles\":false},\"gridData\":{\"h\":20,\"i\":\"131bf4fb-e253-4e66-9f5f-756cae4ea247\",\"w\":27,\"x\":21,\"y\":0},\"panelIndex\":\"131bf4fb-e253-4e66-9f5f-756cae4ea247\",\"title\":\"Container Resources\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_1\"},{\"embeddableConfig\":{\"hidePanelTitles\":false},\"gridData\":{\"h\":15,\"i\":\"7b9841c5-4420-406d-947b-215e8a69c300\",\"w\":24,\"x\":0,\"y\":20},\"panelIndex\":\"7b9841c5-4420-406d-947b-215e8a69c300\",\"title\":\"Amount of Nodes\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_2\"},{\"embeddableConfig\":{},\"gridData\":{\"h\":15,\"i\":\"e97361dd-8ed4-4a34-b2a6-ecbbec424d74\",\"w\":24,\"x\":24,\"y\":20},\"panelIndex\":\"e97361dd-8ed4-4a34-b2a6-ecbbec424d74\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_3\"},{\"embeddableConfig\":{},\"gridData\":{\"h\":15,\"i\":\"97f3ae44-2376-453b-becf-ea4140b77c7d\",\"w\":24,\"x\":0,\"y\":35},\"panelIndex\":\"97f3ae44-2376-453b-becf-ea4140b77c7d\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_4\"},{\"embeddableConfig\":{},\"gridData\":{\"h\":15,\"i\":\"08421449-b5ff-4f64-9409-46852d7a241c\",\"w\":24,\"x\":24,\"y\":35},\"panelIndex\":\"08421449-b5ff-4f64-9409-46852d7a241c\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_5\"},{\"embeddableConfig\":{\"hidePanelTitles\":false},\"gridData\":{\"h\":15,\"i\":\"4c6f8098-dc08-495b-8fdb-2c8dd80d7212\",\"w\":24,\"x\":0,\"y\":50},\"panelIndex\":\"4c6f8098-dc08-495b-8fdb-2c8dd80d7212\",\"title\":\"Node Memory Working Set\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_6\"},{\"embeddableConfig\":{\"hidePanelTitles\":false},\"gridData\":{\"h\":15,\"i\":\"d59b6f1a-18cf-48f1-8f61-96ec0a98412b\",\"w\":24,\"x\":24,\"y\":50},\"panelIndex\":\"d59b6f1a-18cf-48f1-8f61-96ec0a98412b\",\"title\":\"Node Memory RSS\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_7\"},{\"embeddableConfig\":{},\"gridData\":{\"h\":16,\"i\":\"fed42e06-6338-431b-b457-60114c107396\",\"w\":24,\"x\":0,\"y\":65},\"panelIndex\":\"fed42e06-6338-431b-b457-60114c107396\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_8\"},{\"embeddableConfig\":{},\"gridData\":{\"h\":16,\"i\":\"f17e1a62-2deb-43b1-9229-3eae456f0863\",\"w\":24,\"x\":24,\"y\":65},\"panelIndex\":\"f17e1a62-2deb-43b1-9229-3eae456f0863\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_9\"},{\"embeddableConfig\":{\"hidePanelTitles\":false},\"gridData\":{\"h\":15,\"i\":\"4c0f9404-758a-4e73-bb8b-3f99e3eaa7d9\",\"w\":24,\"x\":0,\"y\":81},\"panelIndex\":\"4c0f9404-758a-4e73-bb8b-3f99e3eaa7d9\",\"title\":\"Node Network IO Receive\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_10\"},{\"embeddableConfig\":{\"hidePanelTitles\":false},\"gridData\":{\"h\":15,\"i\":\"984d0e45-5c96-4590-90aa-7aac50f41e80\",\"w\":24,\"x\":24,\"y\":81},\"panelIndex\":\"984d0e45-5c96-4590-90aa-7aac50f41e80\",\"title\":\"Node Network Errors Receive\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_11\"},{\"embeddableConfig\":{\"hidePanelTitles\":false},\"gridData\":{\"h\":15,\"i\":\"2f244a43-be22-48b3-b56e-3dd75525d97b\",\"w\":24,\"x\":0,\"y\":96},\"panelIndex\":\"2f244a43-be22-48b3-b56e-3dd75525d97b\",\"title\":\"Node Network IO Transmit\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_12\"},{\"embeddableConfig\":{\"hidePanelTitles\":false},\"gridData\":{\"h\":15,\"i\":\"10e49f55-401b-48f9-b8e5-b098069cc1ba\",\"w\":24,\"x\":24,\"y\":96},\"panelIndex\":\"10e49f55-401b-48f9-b8e5-b098069cc1ba\",\"title\":\"Node Network Errors Transmit\",\"version\":\"2.19.3\",\"panelRefName\":\"panel_13\"}]","timeRestore":false,"title":"K8s Nodes","version":1},"id":"4ad17100-80c4-11ef-9853-17e21a5b68d5","migrationVersion":{"dashboard":"7.9.3"},"references":[{"id":"ebdb2f00-9776-11ef-a1d8-af81cc9045d1","name":"panel_0","type":"visualization"},{"id":"a8722220-96d4-11ef-b4ed-536715f658e8","name":"panel_1","type":"visualization"},{"id":"9bd3f780-80c4-11ef-b7e6-d7428a02bdd1","name":"panel_2","type":"visualization"},{"id":"ddc86020-7f81-11ef-aad9-b55b1b4895ab","name":"panel_3","type":"visualization"},{"id":"e279cfb0-7f80-11ef-aad9-b55b1b4895ab","name":"panel_4","type":"visualization"},{"id":"535828d0-7f81-11ef-aad9-b55b1b4895ab","name":"panel_5","type":"visualization"},{"id":"eec8ff10-84bd-11ef-9853-17e21a5b68d5","name":"panel_6","type":"visualization"},{"id":"f2aa09d0-84bd-11ef-b7e6-d7428a02bdd1","name":"panel_7","type":"visualization"},{"id":"21169010-7f76-11ef-9899-37d721678cf0","name":"panel_8","type":"visualization"},{"id":"bee22630-7f83-11ef-aad9-b55b1b4895ab","name":"panel_9","type":"visualization"},{"id":"13e784c0-78e1-11f0-b9ad-d91d395f9528","name":"panel_10","type":"visualization"},{"id":"977705e0-78e1-11f0-aa07-c5df75106bf1","name":"panel_11","type":"visualization"},{"id":"f591b2f0-9318-11f0-99d5-2148ceb81347","name":"panel_12","type":"visualization"},{"id":"11106d00-9319-11f0-8b65-db951fa546e1","name":"panel_13","type":"visualization"}],"type":"dashboard","updated_at":"2025-09-16T16:33:59.060Z","version":"Wzk3NywzXQ=="} +{"exportedCount":34,"missingRefCount":0,"missingReferences":[]} \ No newline at end of file diff --git a/docs/user/integration/sap-cloud-logging/dashboard-status.ndjson b/docs/user/integration/sap-cloud-logging/dashboard-status.ndjson index 509546dece..ac98d478e8 100644 --- a/docs/user/integration/sap-cloud-logging/dashboard-status.ndjson +++ b/docs/user/integration/sap-cloud-logging/dashboard-status.ndjson @@ -1,22 +1,22 @@ {"attributes":{"fields":"[{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"_id\"],\"name\":\"_id\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"_index\"],\"name\":\"_index\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":false,\"count\":0,\"name\":\"_score\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":false,\"type\":\"number\"},{\"aggregatable\":false,\"count\":0,\"esTypes\":[\"_source\"],\"name\":\"_source\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":false,\"type\":\"_source\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"_type\"],\"name\":\"_type\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"aggregationTemporality\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"bucketCounts\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"bucketCountsList\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"buckets.count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"buckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"buckets.max\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"buckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"buckets.min\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"buckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":false,\"count\":0,\"esTypes\":[\"text\"],\"name\":\"description\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"exemplars.spanId\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"exemplars\"}},\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"date_nanos\"],\"name\":\"exemplars.time\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"exemplars\"}},\"type\":\"date\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"exemplars.traceId\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"exemplars\"}},\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"exemplars.value\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"exemplars\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"explicitBounds\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"explicitBoundsCount\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"flags\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"instrumentationLibrary.name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"instrumentationLibrary.version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"instrumentationScope.name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"instrumentationScope.version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"boolean\"],\"name\":\"isMonotonic\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"boolean\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"kind\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.client-id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.component\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.concurrency_policy\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.connection_security_policy\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.core\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.cpu\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.cronjob\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.customresource_group\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.customresource_kind\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.customresource_version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.daemon\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_app\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_canonical_revision\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_canonical_service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_cluster\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_principal\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_service_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_service_namespace\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_workload\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_workload_namespace\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.device\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.direction\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.exporter\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.family\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.grpc_response_status\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.http@flavor\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.http@method\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.http@status_code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.interface\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.module\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.namespace\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.net@peer@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.net@peer@port\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.node-id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.operation\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.partition\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.pool\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.processor\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.processorType\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.protocol\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.reason\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.receiver\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":1,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.reporter\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.request_protocol\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.response_code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.response_flags\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.schedule\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.scraper\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.sensor\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.service_instance_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.service_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.service_version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_app\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_canonical_revision\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_canonical_service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_cluster\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_principal\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_workload\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_workload_namespace\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":1,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.state\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.success\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.tag\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.topic\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"negative\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"negativeBuckets.count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"negativeBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"negativeBuckets.max\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"negativeBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"negativeBuckets.min\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"negativeBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"negativeOffset\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"positive\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"positiveBuckets.count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"positiveBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"positiveBuckets.max\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"positiveBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"positiveBuckets.min\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"positiveBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"positiveOffset\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"quantileValuesCount\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"quantiles.quantile\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"quantiles\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"quantiles.value\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"quantiles\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.container@id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.dt@entity@host\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.dt@entity@process_group_instance\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.dt@kubernetes@cluster@id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.dt@kubernetes@workload@kind\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.dt@kubernetes@workload@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.host@arch\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.host@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.http@scheme\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.icke\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.icke@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.icke@pod\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.icke@service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.icke@test\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":2,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@cluster@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":6,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@container@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@cronjob@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@daemonset@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@deployment@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@job@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":1,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@namespace@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":1,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@node@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@pod@ip\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":11,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@pod@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":1,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@pod@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@replicaset@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@statefulset@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.net@host@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.net@host@port\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.os@description\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.os@type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.os@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@command\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@command_args\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@command_line\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@executable@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@executable@path\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@owner\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@pid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@runtime@description\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@runtime@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@runtime@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@app_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@app_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@instance_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@org_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@org_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@process@id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@process@instance_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@process@type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@source_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@source_type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@space_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@space_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service@instance@id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service@namespace\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry@auto@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry@sdk@language\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry@sdk@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry@sdk@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry_sdk_language\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry_sdk_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry_sdk_version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"scale\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"schemaUrl\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"schemaUrl.keyword\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"multi\":{\"parent\":\"schemaUrl\"}},\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"serviceName\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"date_nanos\"],\"name\":\"startTime\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"date\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"sum\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"date_nanos\"],\"name\":\"time\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"date\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"unit\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"value\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"zeroCount\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"}]","timeFieldName":"time","title":"metrics-otel-v1-*"},"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","migrationVersion":{"index-pattern":"7.6.0"},"references":[{"id":"1","name":"index-pattern","type":"version"}],"type":"index-pattern","updated_at":"2024-10-30T16:19:09.088Z","version":"WzQzNDc5LDIyXQ=="} {"attributes":{"columns":["name"],"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"highlightAll\":true,\"version\":true,\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[{\"meta\":{\"alias\":null,\"negate\":false,\"disabled\":false,\"type\":\"phrase\",\"key\":\"instrumentationScope.name\",\"params\":{\"query\":\"io.kyma-project.telemetry/kyma\"},\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index\"},\"query\":{\"match_phrase\":{\"instrumentationScope.name\":\"io.kyma-project.telemetry/kyma\"}},\"$state\":{\"store\":\"appState\"}},{\"meta\":{\"alias\":null,\"negate\":true,\"disabled\":false,\"type\":\"phrase\",\"key\":\"resource.attributes.k8s@resource@kind\",\"params\":{\"query\":\"telemetries\"},\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.filter[1].meta.index\"},\"query\":{\"match_phrase\":{\"resource.attributes.k8s@resource@kind\":\"telemetries\"}},\"$state\":{\"store\":\"appState\"}}],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"sort":[],"title":"kyma status","version":1},"id":"2b17c860-80f0-11ef-b7e6-d7428a02bdd1","migrationVersion":{"search":"7.9.3"},"references":[{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"},{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index","type":"index-pattern"},{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.filter[1].meta.index","type":"index-pattern"}],"type":"search","updated_at":"2024-10-02T19:02:24.800Z","version":"WzM2MjE4LDIyXQ=="} {"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"cluster overview","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}","version":1,"visState":"{\"title\":\"cluster overview\",\"type\":\"table\",\"aggs\":[{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@cluster@name\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Cluster\"},\"schema\":\"bucket\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"avg\",\"params\":{\"field\":\"value\",\"customLabel\":\"Average Health\"},\"schema\":\"metric\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"e4366580-9505-11ef-a1d8-af81cc9045d1","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"2b17c860-80f0-11ef-b7e6-d7428a02bdd1","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2024-10-30T16:07:26.788Z","version":"WzQzMTczLDIyXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"MetricPipelines (copy 2)","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"MetricPipelines (copy 2)\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"gauge\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"rgba(145,112,184,1)\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"min\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"State\",\"terms_field\":\"metric.attributes.state\",\"filter\":{\"query\":\"resource.attributes.k8s@resource@kind: telemetries\",\"language\":\"kuery\"},\"terms_size\":\"10\",\"terms_direction\":\"asc\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"gauge_color_rules\":[{\"id\":\"e66db620-80f4-11ef-a27a-f17750ac535f\"}],\"gauge_width\":10,\"gauge_inner_width\":10,\"gauge_style\":\"half\",\"filter\":{\"query\":\"name: kyma.resource.status.state\",\"language\":\"kuery\"}}}"},"id":"c3d6c5b0-80f5-11ef-9853-17e21a5b68d5","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2024-10-07T16:05:10.572Z","version":"WzM3ODc5LDIyXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"MetricPipelines (copy)","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"MetricPipelines (copy)\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"gauge\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"min\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Healthy\",\"terms_field\":\"resource.attributes.k8s@resource@name\",\"filter\":{\"query\":\"resource.attributes.k8s@resource@kind: metricpipelines\",\"language\":\"kuery\"}}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"gauge_color_rules\":[{\"id\":\"e66db620-80f4-11ef-a27a-f17750ac535f\"}],\"gauge_width\":10,\"gauge_inner_width\":10,\"gauge_style\":\"half\",\"filter\":{\"query\":\"name: kyma.resource.status.conditions\",\"language\":\"kuery\"}}}"},"id":"4d358270-80f5-11ef-9853-17e21a5b68d5","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2024-10-02T19:35:17.429Z","version":"WzM2Mzk5LDIyXQ=="} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"telemetry status","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"telemetry status\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"gauge\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"rgba(145,112,184,1)\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"min\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"State\",\"terms_field\":\"metric.attributes.state\",\"filter\":{\"query\":\"resource.attributes.k8s@resource@kind: telemetries\",\"language\":\"kuery\"},\"terms_size\":\"10\",\"terms_direction\":\"asc\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"gauge_color_rules\":[{\"id\":\"e66db620-80f4-11ef-a27a-f17750ac535f\"}],\"gauge_width\":10,\"gauge_inner_width\":10,\"gauge_style\":\"half\",\"filter\":{\"query\":\"name: kyma.resource.status.state\",\"language\":\"kuery\"}}}"},"id":"c3d6c5b0-80f5-11ef-9853-17e21a5b68d5","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2024-10-07T16:05:10.572Z","version":"WzM3ODc5LDIyXQ=="} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"metricpipeline status","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"metricpipeline status\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"gauge\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"min\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Healthy\",\"terms_field\":\"resource.attributes.k8s@resource@name\",\"filter\":{\"query\":\"resource.attributes.k8s@resource@kind: metricpipelines\",\"language\":\"kuery\"}}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"gauge_color_rules\":[{\"id\":\"e66db620-80f4-11ef-a27a-f17750ac535f\"}],\"gauge_width\":10,\"gauge_inner_width\":10,\"gauge_style\":\"half\",\"filter\":{\"query\":\"name: kyma.resource.status.conditions\",\"language\":\"kuery\"}}}"},"id":"4d358270-80f5-11ef-9853-17e21a5b68d5","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2024-10-02T19:35:17.429Z","version":"WzM2Mzk5LDIyXQ=="} {"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"MetricPipelines","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"MetricPipelines\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"gauge\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"min\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Healthy\",\"terms_field\":\"resource.attributes.k8s@resource@name\",\"filter\":{\"query\":\"resource.attributes.k8s@resource@kind: tracepipelines\",\"language\":\"kuery\"}}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"gauge_color_rules\":[{\"id\":\"e66db620-80f4-11ef-a27a-f17750ac535f\"}],\"gauge_width\":10,\"gauge_inner_width\":10,\"gauge_style\":\"half\",\"filter\":{\"query\":\"name: kyma.resource.status.conditions\",\"language\":\"kuery\"}}}"},"id":"38b7fee0-80f5-11ef-9853-17e21a5b68d5","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2024-10-02T19:33:37.870Z","version":"WzM2Mzc0LDIyXQ=="} {"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"Condition list","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":0,\"direction\":\"asc\"}}}}","version":1,"visState":"{\"title\":\"Condition list\",\"type\":\"table\",\"aggs\":[{\"id\":\"6\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@resource@kind\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Pipeline Type\"},\"schema\":\"bucket\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@resource@name\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Pipeline Name\"},\"schema\":\"bucket\"},{\"id\":\"5\",\"enabled\":true,\"type\":\"avg\",\"params\":{\"field\":\"value\",\"customLabel\":\"Average Health\"},\"schema\":\"metric\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"6bfe29a0-80f0-11ef-9853-17e21a5b68d5","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"2b17c860-80f0-11ef-b7e6-d7428a02bdd1","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2024-10-30T16:07:45.980Z","version":"WzQzMTg4LDIyXQ=="} {"attributes":{"columns":["name","resource.attributes.kyma@module@name","metric.attributes.state","metric.attributes.type","metric.attributes.status"],"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"highlightAll\":true,\"version\":true,\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[{\"meta\":{\"alias\":null,\"negate\":false,\"disabled\":false,\"type\":\"phrase\",\"key\":\"instrumentationScope.name\",\"params\":{\"query\":\"io.kyma-project.telemetry/kyma\"},\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index\"},\"query\":{\"match_phrase\":{\"instrumentationScope.name\":\"io.kyma-project.telemetry/kyma\"}},\"$state\":{\"store\":\"appState\"}},{\"meta\":{\"alias\":null,\"negate\":false,\"disabled\":false,\"type\":\"phrase\",\"key\":\"name\",\"params\":{\"query\":\"kyma.resource.status.state\"},\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.filter[1].meta.index\"},\"query\":{\"match_phrase\":{\"name\":\"kyma.resource.status.state\"}},\"$state\":{\"store\":\"appState\"}}],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"sort":[],"title":"module state","version":1},"id":"5b45c020-4994-11ef-bda1-4346ee025595","migrationVersion":{"search":"7.9.3"},"references":[{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"},{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index","type":"index-pattern"},{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.filter[1].meta.index","type":"index-pattern"}],"type":"search","updated_at":"2024-09-30T15:00:56.743Z","version":"WzM1MjY3LDIyXQ=="} {"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"module states","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"module states\",\"type\":\"line\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"min\",\"params\":{\"field\":\"value\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"params\":{\"field\":\"time\",\"timeRange\":{\"from\":\"now-15m\",\"to\":\"now\"},\"useNormalizedOpenSearchInterval\":true,\"scaleMetricValues\":false,\"interval\":\"1m\",\"drop_partials\":true,\"min_doc_count\":1,\"extended_bounds\":{}},\"schema\":\"segment\"},{\"id\":\"4\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"metric.attributes.state\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":100,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\"},\"schema\":\"group\"}],\"params\":{\"type\":\"line\",\"grid\":{\"categoryLines\":false},\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"type\":\"category\",\"position\":\"bottom\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\"},\"labels\":{\"show\":true,\"filter\":true,\"truncate\":100},\"title\":{}}],\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"name\":\"LeftAxis-1\",\"type\":\"value\",\"position\":\"left\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\",\"mode\":\"normal\",\"setYExtents\":true,\"min\":0.8,\"max\":1.2},\"labels\":{\"show\":true,\"rotate\":0,\"filter\":false,\"truncate\":100},\"title\":{\"text\":\"Min value\"}}],\"seriesParams\":[{\"show\":true,\"type\":\"line\",\"mode\":\"normal\",\"data\":{\"label\":\"Min value\",\"id\":\"1\"},\"valueAxis\":\"ValueAxis-1\",\"drawLinesBetweenPoints\":true,\"lineWidth\":2,\"interpolate\":\"linear\",\"showCircles\":true}],\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"times\":[],\"addTimeMarker\":false,\"labels\":{},\"thresholdLine\":{\"show\":false,\"value\":10,\"width\":1,\"style\":\"full\",\"color\":\"#E7664C\"}}}"},"id":"e1afc6b0-4a7f-11ef-bd5a-f19eaa6d8635","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"5b45c020-4994-11ef-bda1-4346ee025595","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2024-10-08T06:45:35.705Z","version":"WzM3OTM3LDIyXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"MetricPipelines (copy 1)","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"MetricPipelines (copy 1)\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"gauge\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"min\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Healthy\",\"terms_field\":\"resource.attributes.k8s@resource@name\",\"filter\":{\"query\":\"resource.attributes.k8s@resource@kind: logpipelines\",\"language\":\"kuery\"}}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"gauge_color_rules\":[{\"id\":\"e66db620-80f4-11ef-a27a-f17750ac535f\"}],\"gauge_width\":10,\"gauge_inner_width\":10,\"gauge_style\":\"half\",\"filter\":{\"query\":\"name: kyma.resource.status.conditions\",\"language\":\"kuery\"}}}"},"id":"881c3eb0-80f5-11ef-b7e6-d7428a02bdd1","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2024-10-02T19:36:02.738Z","version":"WzM2NDE2LDIyXQ=="} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"logpipeline status","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"logpipeline status\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"gauge\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"min\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Healthy\",\"terms_field\":\"resource.attributes.k8s@resource@name\",\"filter\":{\"query\":\"resource.attributes.k8s@resource@kind: logpipelines\",\"language\":\"kuery\"}}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"gauge_color_rules\":[{\"id\":\"e66db620-80f4-11ef-a27a-f17750ac535f\"}],\"gauge_width\":10,\"gauge_inner_width\":10,\"gauge_style\":\"half\",\"filter\":{\"query\":\"name: kyma.resource.status.conditions\",\"language\":\"kuery\"}}}"},"id":"881c3eb0-80f5-11ef-b7e6-d7428a02bdd1","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2024-10-02T19:36:02.738Z","version":"WzM2NDE2LDIyXQ=="} {"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Divider LogPipelines","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Divider LogPipelines\",\"type\":\"vega\",\"aggs\":[],\"params\":{\"spec\":\"{\\n $schema: https://vega.github.io/schema/vega/v5.json\\n background: rgba(209,255,226,1)\\n padding: {\\n left: 8\\n top: 10\\n right: 8\\n bottom: 10\\n }\\n marks: [\\n {\\n type: text\\n encode: {\\n enter: {\\n fill: {\\n value: \\\"#343741\\\"\\n }\\n text: {\\n value: LogPipelines\\n }\\n }\\n update: {\\n align: {\\n value: center\\n }\\n baseline: {\\n value: alphabetic\\n }\\n font: {\\n value: sans-serif\\n }\\n fontSize: {\\n value: 18.286\\n }\\n fontWeight: {\\n value: 600\\n }\\n }\\n }\\n }\\n ]\\n config: {\\n kibana: {\\n hideWarnings: \\\"true\\\"\\n renderer: svg\\n }\\n }\\n}\"}}"},"id":"28f99f20-80f7-11ef-b7e6-d7428a02bdd1","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2024-10-02T19:47:30.450Z","version":"WzM2NjE4LDIyXQ=="} {"attributes":{"columns":["name"],"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"highlightAll\":true,\"version\":true,\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[{\"meta\":{\"alias\":null,\"negate\":false,\"disabled\":false,\"type\":\"phrase\",\"key\":\"instrumentationScope.name\",\"params\":{\"query\":\"io.kyma-project.telemetry/kyma\"},\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index\"},\"query\":{\"match_phrase\":{\"instrumentationScope.name\":\"io.kyma-project.telemetry/kyma\"}},\"$state\":{\"store\":\"appState\"}},{\"meta\":{\"alias\":null,\"negate\":false,\"disabled\":false,\"type\":\"phrase\",\"key\":\"resource.attributes.k8s@resource@kind\",\"params\":{\"query\":\"logpipelines\"},\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.filter[1].meta.index\"},\"query\":{\"match_phrase\":{\"resource.attributes.k8s@resource@kind\":\"logpipelines\"}},\"$state\":{\"store\":\"appState\"}}],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"sort":[],"title":"logpipeline conditions","version":1},"id":"bd46dcb0-80f7-11ef-9853-17e21a5b68d5","migrationVersion":{"search":"7.9.3"},"references":[{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"},{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index","type":"index-pattern"},{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.filter[1].meta.index","type":"index-pattern"}],"type":"search","updated_at":"2024-10-02T19:51:39.259Z","version":"WzM2NjYxLDIyXQ=="} {"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"logpipeline list","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}","version":1,"visState":"{\"title\":\"logpipeline list\",\"type\":\"table\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"top_hits\",\"params\":{\"field\":\"metric.attributes.reason\",\"aggregate\":\"concat\",\"size\":1,\"sortField\":\"time\",\"sortOrder\":\"desc\",\"customLabel\":\"Reason\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@resource@name\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Pipeline\"},\"schema\":\"bucket\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"metric.attributes.type\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Condition\"},\"schema\":\"bucket\"},{\"id\":\"4\",\"enabled\":true,\"type\":\"avg\",\"params\":{\"field\":\"value\",\"customLabel\":\"Average Health\"},\"schema\":\"metric\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"01234c70-80f8-11ef-9853-17e21a5b68d5","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"bd46dcb0-80f7-11ef-9853-17e21a5b68d5","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2024-10-30T16:08:05.069Z","version":"WzQzMjAyLDIyXQ=="} {"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"logpipelines chart","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"logpipelines chart\",\"type\":\"line\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"min\",\"params\":{\"field\":\"value\",\"customLabel\":\"Healthy\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"params\":{\"field\":\"time\",\"timeRange\":{\"from\":\"now-15m\",\"to\":\"now\"},\"useNormalizedOpenSearchInterval\":true,\"scaleMetricValues\":false,\"interval\":\"1m\",\"drop_partials\":true,\"min_doc_count\":1,\"extended_bounds\":{}},\"schema\":\"segment\"},{\"id\":\"4\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"metric.attributes.type\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":5,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\"},\"schema\":\"group\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@resource@name\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":5,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\"},\"schema\":\"group\"}],\"params\":{\"type\":\"line\",\"grid\":{\"categoryLines\":false},\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"type\":\"category\",\"position\":\"bottom\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\"},\"labels\":{\"show\":true,\"filter\":true,\"truncate\":100},\"title\":{}}],\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"name\":\"LeftAxis-1\",\"type\":\"value\",\"position\":\"left\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\",\"mode\":\"normal\",\"setYExtents\":true,\"min\":-0.2,\"max\":1.2},\"labels\":{\"show\":true,\"rotate\":0,\"filter\":false,\"truncate\":100},\"title\":{\"text\":\"Healthy\"}}],\"seriesParams\":[{\"show\":true,\"type\":\"line\",\"mode\":\"normal\",\"data\":{\"label\":\"Healthy\",\"id\":\"1\"},\"valueAxis\":\"ValueAxis-1\",\"drawLinesBetweenPoints\":true,\"lineWidth\":2,\"interpolate\":\"linear\",\"showCircles\":true}],\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"times\":[],\"addTimeMarker\":false,\"labels\":{},\"thresholdLine\":{\"show\":false,\"value\":10,\"width\":1,\"style\":\"full\",\"color\":\"#E7664C\"}}}"},"id":"54f91730-80f8-11ef-b7e6-d7428a02bdd1","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"bd46dcb0-80f7-11ef-9853-17e21a5b68d5","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2024-10-08T06:44:49.193Z","version":"WzM3OTIyLDIyXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Divider LogPipelines (copy)","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Divider LogPipelines (copy)\",\"type\":\"vega\",\"aggs\":[],\"params\":{\"spec\":\"{\\n $schema: https://vega.github.io/schema/vega/v5.json\\n background: rgba(209,255,226,1)\\n padding: {\\n left: 8\\n top: 10\\n right: 8\\n bottom: 10\\n }\\n marks: [\\n {\\n type: text\\n encode: {\\n enter: {\\n fill: {\\n value: \\\"#343741\\\"\\n }\\n text: {\\n value: TracePipelines\\n }\\n }\\n update: {\\n align: {\\n value: center\\n }\\n baseline: {\\n value: alphabetic\\n }\\n font: {\\n value: sans-serif\\n }\\n fontSize: {\\n value: 18.286\\n }\\n fontWeight: {\\n value: 600\\n }\\n }\\n }\\n }\\n ]\\n config: {\\n kibana: {\\n hideWarnings: \\\"true\\\"\\n renderer: svg\\n }\\n }\\n}\"}}"},"id":"ee722f00-80f8-11ef-9853-17e21a5b68d5","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2024-10-02T20:00:30.860Z","version":"WzM2NzUxLDIyXQ=="} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"divider tracepipelines","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"divider tracepipelines\",\"type\":\"vega\",\"aggs\":[],\"params\":{\"spec\":\"{\\n $schema: https://vega.github.io/schema/vega/v5.json\\n background: rgba(209,255,226,1)\\n padding: {\\n left: 8\\n top: 10\\n right: 8\\n bottom: 10\\n }\\n marks: [\\n {\\n type: text\\n encode: {\\n enter: {\\n fill: {\\n value: \\\"#343741\\\"\\n }\\n text: {\\n value: TracePipelines\\n }\\n }\\n update: {\\n align: {\\n value: center\\n }\\n baseline: {\\n value: alphabetic\\n }\\n font: {\\n value: sans-serif\\n }\\n fontSize: {\\n value: 18.286\\n }\\n fontWeight: {\\n value: 600\\n }\\n }\\n }\\n }\\n ]\\n config: {\\n kibana: {\\n hideWarnings: \\\"true\\\"\\n renderer: svg\\n }\\n }\\n}\"}}"},"id":"ee722f00-80f8-11ef-9853-17e21a5b68d5","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2024-10-02T20:00:30.860Z","version":"WzM2NzUxLDIyXQ=="} {"attributes":{"columns":["name"],"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"highlightAll\":true,\"version\":true,\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[{\"meta\":{\"alias\":null,\"negate\":false,\"disabled\":false,\"type\":\"phrase\",\"key\":\"instrumentationScope.name\",\"params\":{\"query\":\"io.kyma-project.telemetry/kyma\"},\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index\"},\"query\":{\"match_phrase\":{\"instrumentationScope.name\":\"io.kyma-project.telemetry/kyma\"}},\"$state\":{\"store\":\"appState\"}},{\"meta\":{\"alias\":null,\"negate\":false,\"disabled\":false,\"type\":\"phrase\",\"key\":\"resource.attributes.k8s@resource@kind\",\"params\":{\"query\":\"tracepipelines\"},\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.filter[1].meta.index\"},\"query\":{\"match_phrase\":{\"resource.attributes.k8s@resource@kind\":\"tracepipelines\"}},\"$state\":{\"store\":\"appState\"}}],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"sort":[],"title":"tracepipeline conditions","version":1},"id":"0c486da0-80f9-11ef-b7e6-d7428a02bdd1","migrationVersion":{"search":"7.9.3"},"references":[{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"},{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index","type":"index-pattern"},{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.filter[1].meta.index","type":"index-pattern"}],"type":"search","updated_at":"2024-10-02T20:01:01.306Z","version":"WzM2NzYxLDIyXQ=="} {"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"tracepipelines list","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}","version":1,"visState":"{\"title\":\"tracepipelines list\",\"type\":\"table\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"top_hits\",\"params\":{\"field\":\"metric.attributes.reason\",\"aggregate\":\"concat\",\"size\":1,\"sortField\":\"time\",\"sortOrder\":\"desc\",\"customLabel\":\"Reason\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@resource@name\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Pipeline\"},\"schema\":\"bucket\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"metric.attributes.type\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Condition\"},\"schema\":\"bucket\"},{\"id\":\"4\",\"enabled\":true,\"type\":\"avg\",\"params\":{\"field\":\"value\",\"customLabel\":\"Average Health\"},\"schema\":\"metric\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"d8e40950-80f9-11ef-9853-17e21a5b68d5","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"0c486da0-80f9-11ef-b7e6-d7428a02bdd1","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2024-10-30T16:08:29.256Z","version":"WzQzMjE1LDIyXQ=="} {"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"tracepipelines chart","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"tracepipelines chart\",\"type\":\"line\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"min\",\"params\":{\"field\":\"value\",\"customLabel\":\"Healthy\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"params\":{\"field\":\"time\",\"timeRange\":{\"from\":\"now-15m\",\"to\":\"now\"},\"useNormalizedOpenSearchInterval\":true,\"scaleMetricValues\":false,\"interval\":\"1m\",\"drop_partials\":true,\"min_doc_count\":1,\"extended_bounds\":{}},\"schema\":\"segment\"},{\"id\":\"4\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"metric.attributes.type\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":5,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\"},\"schema\":\"group\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@resource@name\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":5,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\"},\"schema\":\"group\"}],\"params\":{\"type\":\"line\",\"grid\":{\"categoryLines\":false},\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"type\":\"category\",\"position\":\"bottom\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\"},\"labels\":{\"show\":true,\"filter\":true,\"truncate\":100},\"title\":{}}],\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"name\":\"LeftAxis-1\",\"type\":\"value\",\"position\":\"left\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\",\"mode\":\"normal\",\"setYExtents\":true,\"min\":-0.2,\"max\":1.2},\"labels\":{\"show\":true,\"rotate\":0,\"filter\":false,\"truncate\":100},\"title\":{\"text\":\"Healthy\"}}],\"seriesParams\":[{\"show\":true,\"type\":\"line\",\"mode\":\"normal\",\"data\":{\"label\":\"Healthy\",\"id\":\"1\"},\"valueAxis\":\"ValueAxis-1\",\"drawLinesBetweenPoints\":true,\"lineWidth\":2,\"interpolate\":\"linear\",\"showCircles\":true}],\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"times\":[],\"addTimeMarker\":false,\"labels\":{},\"thresholdLine\":{\"show\":false,\"value\":10,\"width\":1,\"style\":\"full\",\"color\":\"#E7664C\"}}}"},"id":"5aa68810-80f9-11ef-9853-17e21a5b68d5","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"0c486da0-80f9-11ef-b7e6-d7428a02bdd1","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2024-10-08T06:46:07.824Z","version":"WzM3OTUwLDIyXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Divider LogPipelines (copy 1)","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Divider LogPipelines (copy 1)\",\"type\":\"vega\",\"aggs\":[],\"params\":{\"spec\":\"{\\n $schema: https://vega.github.io/schema/vega/v5.json\\n background: rgba(209,255,226,1)\\n padding: {\\n left: 8\\n top: 10\\n right: 8\\n bottom: 10\\n }\\n marks: [\\n {\\n type: text\\n encode: {\\n enter: {\\n fill: {\\n value: \\\"#343741\\\"\\n }\\n text: {\\n value: MetricPipelines\\n }\\n }\\n update: {\\n align: {\\n value: center\\n }\\n baseline: {\\n value: alphabetic\\n }\\n font: {\\n value: sans-serif\\n }\\n fontSize: {\\n value: 18.286\\n }\\n fontWeight: {\\n value: 600\\n }\\n }\\n }\\n }\\n ]\\n config: {\\n kibana: {\\n hideWarnings: \\\"true\\\"\\n renderer: svg\\n }\\n }\\n}\"}}"},"id":"07bf4690-80fa-11ef-9853-17e21a5b68d5","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2024-10-02T20:08:18.871Z","version":"WzM2ODM3LDIyXQ=="} +{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"divider metricpipelines","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"divider metricpipelines\",\"type\":\"vega\",\"aggs\":[],\"params\":{\"spec\":\"{\\n $schema: https://vega.github.io/schema/vega/v5.json\\n background: rgba(209,255,226,1)\\n padding: {\\n left: 8\\n top: 10\\n right: 8\\n bottom: 10\\n }\\n marks: [\\n {\\n type: text\\n encode: {\\n enter: {\\n fill: {\\n value: \\\"#343741\\\"\\n }\\n text: {\\n value: MetricPipelines\\n }\\n }\\n update: {\\n align: {\\n value: center\\n }\\n baseline: {\\n value: alphabetic\\n }\\n font: {\\n value: sans-serif\\n }\\n fontSize: {\\n value: 18.286\\n }\\n fontWeight: {\\n value: 600\\n }\\n }\\n }\\n }\\n ]\\n config: {\\n kibana: {\\n hideWarnings: \\\"true\\\"\\n renderer: svg\\n }\\n }\\n}\"}}"},"id":"07bf4690-80fa-11ef-9853-17e21a5b68d5","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2024-10-02T20:08:18.871Z","version":"WzM2ODM3LDIyXQ=="} {"attributes":{"columns":["name"],"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"highlightAll\":true,\"version\":true,\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[{\"meta\":{\"alias\":null,\"negate\":false,\"disabled\":false,\"type\":\"phrase\",\"key\":\"instrumentationScope.name\",\"params\":{\"query\":\"io.kyma-project.telemetry/kyma\"},\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index\"},\"query\":{\"match_phrase\":{\"instrumentationScope.name\":\"io.kyma-project.telemetry/kyma\"}},\"$state\":{\"store\":\"appState\"}},{\"meta\":{\"alias\":null,\"negate\":false,\"disabled\":false,\"type\":\"phrase\",\"key\":\"resource.attributes.k8s@resource@kind\",\"params\":{\"query\":\"metricpipelines\"},\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.filter[1].meta.index\"},\"query\":{\"match_phrase\":{\"resource.attributes.k8s@resource@kind\":\"metricpipelines\"}},\"$state\":{\"store\":\"appState\"}}],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"sort":[],"title":"metricpipeline conditions","version":1},"id":"18448350-80f9-11ef-b7e6-d7428a02bdd1","migrationVersion":{"search":"7.9.3"},"references":[{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"},{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index","type":"index-pattern"},{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.filter[1].meta.index","type":"index-pattern"}],"type":"search","updated_at":"2024-10-02T20:01:21.413Z","version":"WzM2NzY0LDIyXQ=="} {"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"metricpipeline list","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}","version":1,"visState":"{\"title\":\"metricpipeline list\",\"type\":\"table\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"top_hits\",\"params\":{\"field\":\"metric.attributes.reason\",\"aggregate\":\"concat\",\"size\":1,\"sortField\":\"time\",\"sortOrder\":\"desc\",\"customLabel\":\"Reason\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@resource@name\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Pipeline\"},\"schema\":\"bucket\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"metric.attributes.type\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Condition\"},\"schema\":\"bucket\"},{\"id\":\"4\",\"enabled\":true,\"type\":\"avg\",\"params\":{\"field\":\"value\",\"customLabel\":\"Average Health\"},\"schema\":\"metric\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"49ea0eb0-80fa-11ef-b7e6-d7428a02bdd1","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"18448350-80f9-11ef-b7e6-d7428a02bdd1","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2024-10-30T16:08:48.305Z","version":"WzQzMjI4LDIyXQ=="} {"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"metricpipeline chart","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"metricpipeline chart\",\"type\":\"line\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"min\",\"params\":{\"field\":\"value\",\"customLabel\":\"Health\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"params\":{\"field\":\"time\",\"timeRange\":{\"from\":\"now-15m\",\"to\":\"now\"},\"useNormalizedOpenSearchInterval\":true,\"scaleMetricValues\":false,\"interval\":\"1m\",\"drop_partials\":true,\"min_doc_count\":1,\"extended_bounds\":{}},\"schema\":\"segment\"},{\"id\":\"4\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"metric.attributes.type\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":5,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\"},\"schema\":\"group\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@resource@name\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":5,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\"},\"schema\":\"group\"}],\"params\":{\"type\":\"line\",\"grid\":{\"categoryLines\":false},\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"type\":\"category\",\"position\":\"bottom\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\"},\"labels\":{\"show\":true,\"filter\":true,\"truncate\":100},\"title\":{}}],\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"name\":\"LeftAxis-1\",\"type\":\"value\",\"position\":\"left\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\",\"mode\":\"normal\",\"setYExtents\":true,\"min\":-0.2,\"max\":1.2},\"labels\":{\"show\":true,\"rotate\":0,\"filter\":false,\"truncate\":100},\"title\":{\"text\":\"Health\"}}],\"seriesParams\":[{\"show\":true,\"type\":\"line\",\"mode\":\"normal\",\"data\":{\"label\":\"Health\",\"id\":\"1\"},\"valueAxis\":\"ValueAxis-1\",\"drawLinesBetweenPoints\":true,\"lineWidth\":2,\"interpolate\":\"linear\",\"showCircles\":true}],\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"times\":[],\"addTimeMarker\":false,\"labels\":{},\"thresholdLine\":{\"show\":false,\"value\":10,\"width\":1,\"style\":\"full\",\"color\":\"#E7664C\"}}}"},"id":"b3302170-80fa-11ef-9853-17e21a5b68d5","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"18448350-80f9-11ef-b7e6-d7428a02bdd1","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2024-10-08T06:47:16.771Z","version":"WzM3OTY5LDIyXQ=="} diff --git a/docs/user/integration/sap-cloud-logging/dashboard-volumes.ndjson b/docs/user/integration/sap-cloud-logging/dashboard-volumes.ndjson deleted file mode 100644 index 332c993191..0000000000 --- a/docs/user/integration/sap-cloud-logging/dashboard-volumes.ndjson +++ /dev/null @@ -1,9 +0,0 @@ -{"attributes":{"fields":"[{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"_id\"],\"name\":\"_id\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"_index\"],\"name\":\"_index\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":false,\"count\":0,\"name\":\"_score\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":false,\"type\":\"number\"},{\"aggregatable\":false,\"count\":0,\"esTypes\":[\"_source\"],\"name\":\"_source\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":false,\"type\":\"_source\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"_type\"],\"name\":\"_type\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"aggregationTemporality\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"bucketCounts\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"bucketCountsList\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"buckets.count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"buckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"buckets.max\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"buckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"buckets.min\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"buckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":false,\"count\":0,\"esTypes\":[\"text\"],\"name\":\"description\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"exemplars.spanId\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"exemplars\"}},\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"date_nanos\"],\"name\":\"exemplars.time\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"exemplars\"}},\"type\":\"date\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"exemplars.traceId\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"exemplars\"}},\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"exemplars.value\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"exemplars\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"explicitBounds\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"explicitBoundsCount\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"flags\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"instrumentationLibrary.name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"instrumentationLibrary.version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"instrumentationScope.name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"instrumentationScope.version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"boolean\"],\"name\":\"isMonotonic\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"boolean\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"kind\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.component\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.connection_security_policy\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_app\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_canonical_revision\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_canonical_service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_cluster\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_principal\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_service_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_service_namespace\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_workload\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_workload_namespace\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.direction\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.event\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.group\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.grpc_code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.grpc_method\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.grpc_response_status\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.grpc_service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.grpc_type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.interface\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.path\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.reason\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.reporter\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.request_protocol\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.resource\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.response_code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.response_flags\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.result\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_app\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_canonical_revision\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_canonical_service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_cluster\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_principal\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_workload\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_workload_namespace\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.state\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.status\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.tag\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"negative\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"negativeBuckets.count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"negativeBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"negativeBuckets.max\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"negativeBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"negativeBuckets.min\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"negativeBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"negativeOffset\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"positive\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"positiveBuckets.count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"positiveBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"positiveBuckets.max\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"positiveBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"positiveBuckets.min\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"positiveBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"positiveOffset\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"quantileValuesCount\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"quantiles.quantile\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"quantiles\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"quantiles.value\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"quantiles\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.container@id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.container@image@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.container@image@tag\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.http@scheme\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":2,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@cluster@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":6,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@container@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@daemonset@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@daemonset@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@deployment@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@deployment@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@hpa@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@hpa@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":1,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@job@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@job@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":1,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@namespace@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@namespace@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":1,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@node@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@node@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@persistentvolumeclaim@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@pod@ip\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":11,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@pod@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@pod@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@replicaset@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@replicaset@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@resource@group\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@resource@kind\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@resource@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@resource@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@statefulset@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@statefulset@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@volume@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@volume@type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.net@host@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.net@host@port\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@app_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@app_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@instance_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@org_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@org_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@process@id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@process@instance_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@process@type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@source_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@source_type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@space_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@space_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.server@address\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.server@port\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service@instance@id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry_sdk_language\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry_sdk_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry_sdk_version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.url@scheme\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"scale\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"schemaUrl\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"serviceName\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"date_nanos\"],\"name\":\"startTime\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"date\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"sum\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"date_nanos\"],\"name\":\"time\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"date\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"unit\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"value\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"zeroCount\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"}]","timeFieldName":"time","title":"metrics-otel-v1-*"},"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","migrationVersion":{"index-pattern":"7.6.0"},"references":[{"id":"1","name":"index-pattern","type":"version"}],"type":"index-pattern","updated_at":"2024-10-30T15:18:38.892Z","version":"WzQyODc2LDIyXQ=="} -{"attributes":{"columns":["value","resource.attributes.k8s@pod@name"],"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"highlightAll\":true,\"version\":true,\"query\":{\"query\":\"name:k8s.volume.capacity\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"sort":[],"title":"[OTel] K8s Volume Capacity","version":1},"id":"1e517ef0-8a32-11ef-aad9-b55b1b4895ab","migrationVersion":{"search":"7.9.3"},"references":[{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"search","updated_at":"2024-10-14T15:38:25.768Z","version":"WzM4MzA1LDIyXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"PVC List","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}","version":1,"visState":"{\"title\":\"PVC List\",\"type\":\"table\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"json\":\"{\\n \\\"script\\\" : {\\n \\\"inline\\\": \\\"doc['value'].value / 1024 / 1024\\\",\\n \\\"lang\\\": \\\"painless\\\"\\n }\\n}\",\"customLabel\":\"Capacity GB\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@cluster@name\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Cluster\"},\"schema\":\"bucket\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@namespace@name\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Namespace\"},\"schema\":\"bucket\"},{\"id\":\"4\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.service@name\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Workload\"},\"schema\":\"bucket\"},{\"id\":\"5\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@persistentvolumeclaim@name\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Volume\"},\"schema\":\"bucket\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"b792e4f0-8a32-11ef-aad9-b55b1b4895ab","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"1e517ef0-8a32-11ef-aad9-b55b1b4895ab","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2024-10-30T16:04:41.047Z","version":"WzQzMTEzLDIyXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"K8s Volume Available","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"K8s Volume Available\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"max\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"bytes\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"filter\":{\"query\":\"name: k8s.volume.available\",\"language\":\"kuery\"},\"terms_field\":\"resource.attributes.k8s@persistentvolumeclaim@name\",\"label\":\"\",\"type\":\"timeseries\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false}}"},"id":"123ab780-8a41-11ef-9899-37d721678cf0","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2024-10-15T15:08:23.141Z","version":"WzM4Mzg0LDIyXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"K8s Volume Available (copy)","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"K8s Volume Available (copy)\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"max\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"bytes\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"filter\":{\"query\":\"name: k8s.volume.capacity\",\"language\":\"kuery\"},\"terms_field\":\"resource.attributes.k8s@persistentvolumeclaim@name\",\"label\":\"\",\"type\":\"timeseries\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false}}"},"id":"1b2ebd50-8a41-11ef-aad9-b55b1b4895ab","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2024-10-14T15:38:25.768Z","version":"WzM4MzA4LDIyXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"K8s Volume Available (copy 1)","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"K8s Volume Available (copy 1)\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"max\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"filter\":{\"query\":\"name: k8s.volume.inodes.free\",\"language\":\"kuery\"},\"terms_field\":\"resource.attributes.k8s@persistentvolumeclaim@name\",\"label\":\"\",\"type\":\"timeseries\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false}}"},"id":"5e7d6c00-8a41-11ef-aad9-b55b1b4895ab","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2024-10-14T15:38:25.768Z","version":"WzM4MzA5LDIyXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"K8s Volume Available (copy 2)","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"K8s Volume Available (copy 2)\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"terms\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"max\",\"field\":\"value\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"filter\":{\"query\":\"name: k8s.volume.inodes\",\"language\":\"kuery\"},\"terms_field\":\"resource.attributes.k8s@persistentvolumeclaim@name\",\"label\":\"\",\"type\":\"timeseries\"}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false}}"},"id":"777c5900-8a41-11ef-9899-37d721678cf0","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2024-10-14T15:38:25.768Z","version":"WzM4MzEwLDIyXQ=="} -{"attributes":{"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[]}"},"optionsJSON":"{\"hidePanelTitles\":false,\"useMargins\":true}","panelsJSON":"[{\"embeddableConfig\":{\"hidePanelTitles\":false,\"title\":\"Volume List\"},\"gridData\":{\"h\":9,\"i\":\"2f25d4a8-99ff-4e84-a8da-1740f02e4524\",\"w\":30,\"x\":0,\"y\":0},\"panelIndex\":\"2f25d4a8-99ff-4e84-a8da-1740f02e4524\",\"title\":\"Volume List\",\"version\":\"1.3.18\",\"panelRefName\":\"panel_0\"},{\"embeddableConfig\":{\"hidePanelTitles\":false,\"title\":\"Volume Available\"},\"gridData\":{\"h\":11,\"i\":\"fa00a682-abbd-4deb-9f51-6c07809d1e33\",\"w\":24,\"x\":0,\"y\":9},\"panelIndex\":\"fa00a682-abbd-4deb-9f51-6c07809d1e33\",\"title\":\"Volume Available\",\"version\":\"1.3.18\",\"panelRefName\":\"panel_1\"},{\"embeddableConfig\":{\"hidePanelTitles\":false,\"title\":\"Volume Capacity\"},\"gridData\":{\"h\":11,\"i\":\"524534a8-db24-47e2-baf1-c86e83850f71\",\"w\":24,\"x\":24,\"y\":9},\"panelIndex\":\"524534a8-db24-47e2-baf1-c86e83850f71\",\"title\":\"Volume Capacity\",\"version\":\"1.3.18\",\"panelRefName\":\"panel_2\"},{\"embeddableConfig\":{\"hidePanelTitles\":false,\"title\":\"inodes free\"},\"gridData\":{\"h\":11,\"i\":\"2382926c-e2e2-49ef-bffc-02a1f1dc21de\",\"w\":24,\"x\":0,\"y\":20},\"panelIndex\":\"2382926c-e2e2-49ef-bffc-02a1f1dc21de\",\"title\":\"inodes free\",\"version\":\"1.3.18\",\"panelRefName\":\"panel_3\"},{\"embeddableConfig\":{\"hidePanelTitles\":false,\"title\":\"inodes total\"},\"gridData\":{\"h\":11,\"i\":\"2008bf73-29d5-4d4f-9787-17bf1c523e73\",\"w\":23,\"x\":24,\"y\":20},\"panelIndex\":\"2008bf73-29d5-4d4f-9787-17bf1c523e73\",\"title\":\"inodes total\",\"version\":\"1.3.18\",\"panelRefName\":\"panel_4\"}]","timeRestore":false,"title":"K8s Volumes","version":1},"id":"433f93e0-8a33-11ef-aad9-b55b1b4895ab","migrationVersion":{"dashboard":"7.9.3"},"references":[{"id":"b792e4f0-8a32-11ef-aad9-b55b1b4895ab","name":"panel_0","type":"visualization"},{"id":"123ab780-8a41-11ef-9899-37d721678cf0","name":"panel_1","type":"visualization"},{"id":"1b2ebd50-8a41-11ef-aad9-b55b1b4895ab","name":"panel_2","type":"visualization"},{"id":"5e7d6c00-8a41-11ef-aad9-b55b1b4895ab","name":"panel_3","type":"visualization"},{"id":"777c5900-8a41-11ef-9899-37d721678cf0","name":"panel_4","type":"visualization"}],"type":"dashboard","updated_at":"2024-10-30T16:04:52.456Z","version":"WzQzMTE5LDIyXQ=="} -{"exportedCount":8,"missingRefCount":1,"missingReferences":[{"id":"1","type":"version"}]} \ No newline at end of file diff --git a/docs/user/integration/sap-cloud-logging/dashboard-workloads.ndjson b/docs/user/integration/sap-cloud-logging/dashboard-workloads.ndjson deleted file mode 100644 index e203ee42ea..0000000000 --- a/docs/user/integration/sap-cloud-logging/dashboard-workloads.ndjson +++ /dev/null @@ -1,13 +0,0 @@ -{"attributes":{"fields":"[{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"_id\"],\"name\":\"_id\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"_index\"],\"name\":\"_index\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":false,\"count\":0,\"name\":\"_score\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":false,\"type\":\"number\"},{\"aggregatable\":false,\"count\":0,\"esTypes\":[\"_source\"],\"name\":\"_source\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":false,\"type\":\"_source\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"_type\"],\"name\":\"_type\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"aggregationTemporality\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"app_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"app_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"bucketCounts\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"bucketCountsList\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"buckets.count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"buckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"buckets.max\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"buckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"buckets.min\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"buckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":false,\"count\":0,\"esTypes\":[\"text\"],\"name\":\"description\",\"readFromDocValues\":false,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"exemplars.spanId\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"exemplars\"}},\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"date_nanos\"],\"name\":\"exemplars.time\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"exemplars\"}},\"type\":\"date\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"exemplars.traceId\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"exemplars\"}},\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"exemplars.value\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"exemplars\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"explicitBounds\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"explicitBoundsCount\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"flags\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"instance_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"instrumentationLibrary.name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"instrumentationLibrary.version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"instrumentationScope.name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"instrumentationScope.version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"boolean\"],\"name\":\"isMonotonic\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"boolean\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"kind\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.app@payment@currency\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.build_date\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.cause\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.cluster_type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.component\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.connection_security_policy\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.core\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.cpu\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.currency_code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.debug\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_app\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_canonical_revision\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_canonical_service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_cluster\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_principal\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_service_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_service_namespace\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_workload\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.destination_workload_namespace\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.device\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.direction\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.endpoint\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.err\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.event\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.family\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.format\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.group\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.grpc_code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.grpc_method\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.grpc_response_status\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.grpc_service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.grpc_type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.host\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.http@flavor\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.http@host\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.http@method\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.http@scheme\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.http@status_code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.interface\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.method\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.model\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.net@host@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.net@host@port\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.net@peer@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.net@peer@port\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.net@protocol@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.net@protocol@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.node\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.nodejs@eventloop@state\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.operation\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.otel_scope_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.otel_scope_version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.path\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.protocol\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.reason\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.recommendation@type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.reporter\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.request_protocol\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.resource\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.response_code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.response_flags\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.result\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.revision\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.rpc@grpc@status_code\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.rpc@method\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.rpc@service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.rpc@system\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.sampler_type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_app\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_canonical_revision\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_canonical_service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_cluster\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_principal\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_workload\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.source_workload_namespace\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.state\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.status\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.svc\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.tag\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.target\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.transport\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":1,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.v8js@gc@type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.v8js@heap@space@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"metric.attributes.version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"negative\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"negativeBuckets.count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"negativeBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"negativeBuckets.max\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"negativeBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"negativeBuckets.min\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"negativeBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"negativeOffset\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"organization_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"organization_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"positive\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"positiveBuckets.count\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"positiveBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"positiveBuckets.max\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"positiveBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"positiveBuckets.min\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"positiveBuckets\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"positiveOffset\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"process_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"process_type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"quantileValuesCount\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"quantiles.quantile\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"quantiles\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"quantiles.value\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"subType\":{\"nested\":{\"path\":\"quantiles\"}},\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.cloud@account@id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.cloud@availability_zone\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.cloud@platform\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.cloud@provider\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.cloud@region\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.container@id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.container@image@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.container@image@tag\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.host@arch\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.host@id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.host@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.host@type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.http@scheme\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":2,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@cluster@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":6,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@container@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@daemonset@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@daemonset@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@deployment@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@deployment@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@job@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@job@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":1,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@namespace@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":1,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@node@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@node@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@persistentvolumeclaim@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@pod@ip\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":11,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@pod@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@pod@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@replicaset@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@resource@group\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@resource@kind\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@resource@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@resource@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@statefulset@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@statefulset@uid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@volume@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.k8s@volume@type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.net@host@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.net@host@port\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.os@description\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.os@type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.os@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@command\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@command_args\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@executable@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@executable@path\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@owner\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@pid\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@runtime@description\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@runtime@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.process@runtime@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@app_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@app_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@instance_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@org_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@org_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@process@id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@process@type\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@space_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.sap@cf@space_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.server@address\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.server@port\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service@instance@id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service@namespace\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.service_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry@auto@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry@sdk@language\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry@sdk@name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry@sdk@version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry_sdk_language\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry_sdk_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.telemetry_sdk_version\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"resource.attributes.url@scheme\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"scale\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"schemaUrl\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"serviceName\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"space_id\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"space_name\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"date_nanos\"],\"name\":\"startTime\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"date\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"sum\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"date_nanos\"],\"name\":\"time\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"date\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"keyword\"],\"name\":\"unit\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"string\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"double\"],\"name\":\"value\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"},{\"aggregatable\":true,\"count\":0,\"esTypes\":[\"long\"],\"name\":\"zeroCount\",\"readFromDocValues\":true,\"scripted\":false,\"searchable\":true,\"type\":\"number\"}]","timeFieldName":"time","title":"metrics-otel-v1-*"},"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","migrationVersion":{"index-pattern":"7.6.0"},"references":[{"id":"1","name":"index-pattern","type":"version"}],"type":"index-pattern","updated_at":"2025-04-08T10:18:05.155Z","version":"WzExODM3LDFd"} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"title":"Clusters","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}","version":1,"visState":"{\"title\":\"Clusters\",\"type\":\"table\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"cardinality\",\"params\":{\"field\":\"resource.attributes.k8s@namespace@name\",\"customLabel\":\"Namespaces\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@cluster@name\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\"},\"schema\":\"bucket\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"fdba71f0-950e-11ef-a1d8-af81cc9045d1","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"visualization","updated_at":"2025-03-31T06:58:43.928Z","version":"WzQxMiwxXQ=="} -{"attributes":{"columns":["_source"],"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"highlightAll\":true,\"version\":true,\"query\":{\"query\":\"name: k8s.deployment.* or name: k8s.job.* or name: k8s.statefulset.* or name: k8s.daemonset.*\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"sort":[],"title":"K8s Workloads","version":1},"id":"07fe9710-9512-11ef-a1d8-af81cc9045d1","migrationVersion":{"search":"7.9.3"},"references":[{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"search","updated_at":"2025-03-31T06:58:43.928Z","version":"WzQxMywxXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"Workload list2","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}","version":1,"visState":"{\"title\":\"Workload list2\",\"type\":\"table\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"cardinality\",\"params\":{\"field\":\"resource.attributes.k8s@deployment@name\",\"customLabel\":\"Deployments\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@namespace@name\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":996,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Namespaces\"},\"schema\":\"bucket\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"cardinality\",\"params\":{\"field\":\"resource.attributes.k8s@statefulset@name\",\"customLabel\":\"StatefulSet\"},\"schema\":\"metric\"},{\"id\":\"4\",\"enabled\":true,\"type\":\"cardinality\",\"params\":{\"field\":\"resource.attributes.k8s@daemonset@name\",\"customLabel\":\"DaemonSets\"},\"schema\":\"metric\"},{\"id\":\"5\",\"enabled\":true,\"type\":\"cardinality\",\"params\":{\"field\":\"resource.attributes.k8s@job@name\",\"customLabel\":\"Jobs\"},\"schema\":\"metric\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"878b8140-954a-11ef-a1d8-af81cc9045d1","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"07fe9710-9512-11ef-a1d8-af81cc9045d1","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2025-03-31T06:58:43.928Z","version":"WzQxNCwxXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"title":"Workloard Chart","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Workloard Chart\",\"type\":\"metrics\",\"aggs\":[],\"params\":{\"id\":\"61ca57f0-469d-11e7-af02-69e470af7417\",\"type\":\"timeseries\",\"series\":[{\"id\":\"61ca57f1-469d-11e7-af02-69e470af7417\",\"color\":\"#68BC00\",\"split_mode\":\"everything\",\"split_color_mode\":\"opensearchDashboards\",\"metrics\":[{\"id\":\"61ca57f2-469d-11e7-af02-69e470af7417\",\"type\":\"cardinality\",\"field\":\"resource.attributes.k8s@deployment@name\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Deployments\",\"type\":\"timeseries\",\"filter\":{\"query\":\"name: k8s.deployment.available\",\"language\":\"kuery\"}},{\"id\":\"63c2b270-9549-11ef-9761-9b02e78a949d\",\"color\":\"rgba(96,146,192,1)\",\"split_mode\":\"everything\",\"metrics\":[{\"id\":\"63c2b271-9549-11ef-9761-9b02e78a949d\",\"type\":\"cardinality\",\"field\":\"resource.attributes.k8s@statefulset@name\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"StatefulSets\",\"type\":\"timeseries\",\"filter\":{\"query\":\"name: k8s.statefulset.desired_pods\",\"language\":\"kuery\"}},{\"id\":\"b855b2b0-9549-11ef-9761-9b02e78a949d\",\"color\":\"rgba(202,142,174,1)\",\"split_mode\":\"everything\",\"metrics\":[{\"id\":\"b855b2b1-9549-11ef-9761-9b02e78a949d\",\"type\":\"cardinality\",\"field\":\"resource.attributes.k8s@daemonset@name\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"DaemonSets\",\"type\":\"timeseries\",\"filter\":{\"query\":\"name: k8s.daemonset.desired_scheduled_nodes\",\"language\":\"kuery\"}},{\"id\":\"d80ff390-9549-11ef-9761-9b02e78a949d\",\"color\":\"rgba(185,168,136,1)\",\"split_mode\":\"everything\",\"metrics\":[{\"id\":\"d80ff391-9549-11ef-9761-9b02e78a949d\",\"type\":\"cardinality\",\"field\":\"resource.attributes.k8s@job@name\"}],\"separate_axis\":0,\"axis_position\":\"right\",\"formatter\":\"number\",\"chart_type\":\"line\",\"line_width\":1,\"point_size\":1,\"fill\":0.5,\"stacked\":\"none\",\"label\":\"Jobs\",\"type\":\"timeseries\",\"filter\":{\"query\":\"name: k8s.job.desired_successful_pods\",\"language\":\"kuery\"}}],\"time_field\":\"time\",\"index_pattern\":\"metrics-otel-v1-*\",\"interval\":\"1m\",\"axis_position\":\"left\",\"axis_formatter\":\"number\",\"axis_scale\":\"normal\",\"show_legend\":1,\"show_grid\":1,\"tooltip_mode\":\"show_all\",\"default_index_pattern\":\"logs-cfsyslog-*\",\"default_timefield\":\"@timestamp\",\"isModelInvalid\":false,\"filter\":{\"query\":\"name: k8s.*\",\"language\":\"kuery\"}}}"},"id":"7ef9b520-9549-11ef-b4ed-536715f658e8","migrationVersion":{"visualization":"7.10.0"},"references":[],"type":"visualization","updated_at":"2025-03-31T06:58:43.928Z","version":"WzQxNSwxXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"deployments2","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":0,\"direction\":\"asc\"}}}}","version":1,"visState":"{\"title\":\"deployments2\",\"type\":\"table\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"1-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.deployment.desired\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"1-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Desired Pods\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@deployment@name\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Deployments\"},\"schema\":\"bucket\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"3-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.deployment.available\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"3-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Available Pods\"},\"schema\":\"metric\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"17c26560-96ce-11ef-a1d8-af81cc9045d1","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"07fe9710-9512-11ef-a1d8-af81cc9045d1","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2025-03-31T06:58:43.928Z","version":"WzQxNiwxXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"deployments2 (copy)","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":0,\"direction\":\"asc\"}}}}","version":1,"visState":"{\"title\":\"deployments2 (copy)\",\"type\":\"table\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"1-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.statefulset.desired_pods\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"1-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Desired Pods\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@statefulset@name\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"StatefulSets\"},\"schema\":\"bucket\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"3-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.statefulset.ready_pods\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"3-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Available Pods\"},\"schema\":\"metric\"},{\"id\":\"4\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"4-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.statefulset.current_pods\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"4-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Current Pods\"},\"schema\":\"metric\"},{\"id\":\"5\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"5-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.statefulset.updated_pods\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"5-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Updated Pods\"},\"schema\":\"metric\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"d7d514b0-96ce-11ef-b4ed-536715f658e8","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"07fe9710-9512-11ef-a1d8-af81cc9045d1","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2025-03-31T06:58:43.928Z","version":"WzQxNywxXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"deployments2 (copy 1)","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":0,\"direction\":\"asc\"}}}}","version":1,"visState":"{\"title\":\"deployments2 (copy 1)\",\"type\":\"table\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"1-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.daemonset.desired_scheduled_nodes\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"1-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Desired Nodes\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@daemonset@name\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"DaemonSets\"},\"schema\":\"bucket\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"3-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.daemonset.ready_nodes\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"3-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Ready Nodes\"},\"schema\":\"metric\"},{\"id\":\"5\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"5-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.daemonset.current_scheduled_nodes\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"5-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Scheduled Nodes\"},\"schema\":\"metric\"},{\"id\":\"4\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"4-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.daemonset.misscheduled_nodes\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"4-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"customLabel\":\"\"}},\"customLabel\":\"Misscheduled Nodes\"},\"schema\":\"metric\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"f6afeab0-96d1-11ef-b4ed-536715f658e8","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"07fe9710-9512-11ef-a1d8-af81cc9045d1","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2025-03-31T06:58:43.928Z","version":"WzQxOCwxXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"deployments2 (copy 2)","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":0,\"direction\":\"asc\"}}}}","version":1,"visState":"{\"title\":\"deployments2 (copy 2)\",\"type\":\"table\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"1-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.job.desired_successful_pods\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"1-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Desired Pods\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@job@name\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Jobs\"},\"schema\":\"bucket\"},{\"id\":\"4\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"4-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.job.successful_pods\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"4-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"customLabel\":\"\"}},\"customLabel\":\"Successful Pods\"},\"schema\":\"metric\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"3-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.job.active_pods\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"3-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Active Pods\"},\"schema\":\"metric\"},{\"id\":\"5\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"5-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.job.failed_pods\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"5-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"Failed Pods\"},\"schema\":\"metric\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"2029a240-96d3-11ef-a1d8-af81cc9045d1","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"07fe9710-9512-11ef-a1d8-af81cc9045d1","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2025-03-31T06:58:43.928Z","version":"WzQxOSwxXQ=="} -{"attributes":{"columns":["_source"],"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"highlightAll\":true,\"version\":true,\"query\":{\"query\":\" name: k8s.container.* or name: container.*\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"sort":[],"title":"K8s Container","version":1},"id":"d6b5f6d0-96d3-11ef-a1d8-af81cc9045d1","migrationVersion":{"search":"7.9.3"},"references":[{"id":"maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"search","updated_at":"2025-03-31T06:59:05.579Z","version":"WzQzNCwxXQ=="} -{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"container resources","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":0,\"direction\":\"asc\"}}}}","version":1,"visState":"{\"title\":\"container resources\",\"type\":\"table\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"1-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.container.cpu_request\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"1-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"customLabel\":\"\"}},\"customLabel\":\"CPU Request\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@pod@name\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":997,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Pod\"},\"schema\":\"bucket\"},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"resource.attributes.k8s@container@name\",\"orderBy\":\"_key\",\"order\":\"desc\",\"size\":1000,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Container\"},\"schema\":\"bucket\"},{\"id\":\"8\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"8-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: container.cpu.usage\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"8-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"CPU Usage\"},\"schema\":\"metric\"},{\"id\":\"4\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"4-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.container.cpu_limit\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"4-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\"}},\"customLabel\":\"CPU Limit\"},\"schema\":\"metric\"},{\"id\":\"5\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"5-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.container.memory_request\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"5-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"json\":\"{\\n \\\"script\\\": {\\n \\\"inline\\\": \\\"doc['value'].value / 1024 / 1024\\\",\\n \\\"lang\\\": \\\"painless\\\"\\n }\\n}\"}},\"json\":\"\",\"customLabel\":\"Memory Request (MB)\"},\"schema\":\"metric\"},{\"id\":\"7\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"7-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: container.memory.working_set\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"7-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"json\":\"{\\n \\\"script\\\": {\\n \\\"inline\\\": \\\"doc['value'].value / 1024 / 1024\\\",\\n \\\"lang\\\": \\\"painless\\\"\\n }\\n}\"}},\"customLabel\":\"Memory Usage (MB)\"},\"schema\":\"metric\"},{\"id\":\"6\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"6-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name: k8s.container.memory_limit\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"6-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"json\":\"{\\n \\\"script\\\": {\\n \\\"inline\\\": \\\"doc['value'].value / 1024 / 1024\\\",\\n \\\"lang\\\": \\\"painless\\\"\\n }\\n}\"}},\"customLabel\":\"Memory Limit (MB)\"},\"schema\":\"metric\"},{\"id\":\"9\",\"enabled\":true,\"type\":\"max_bucket\",\"params\":{\"customBucket\":{\"id\":\"9-bucket\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"name:k8s.container.restarts\",\"language\":\"kuery\"},\"label\":\"\"}]}},\"customMetric\":{\"id\":\"9-metric\",\"enabled\":true,\"type\":\"max\",\"params\":{\"field\":\"value\",\"customLabel\":\"Restarts\"}},\"customLabel\":\"Restarts\"},\"schema\":\"metric\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":true,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"a8722220-96d4-11ef-b4ed-536715f658e8","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"d6b5f6d0-96d3-11ef-a1d8-af81cc9045d1","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2025-04-17T09:09:31.973Z","version":"WzEyMzg4LDFd"} -{"attributes":{"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[]}"},"optionsJSON":"{\"hidePanelTitles\":false,\"useMargins\":true}","panelsJSON":"[{\"version\":\"1.3.18\",\"gridData\":{\"x\":0,\"y\":0,\"w\":14,\"h\":16,\"i\":\"f57d14bb-fa23-40ec-96b5-af20d521ca2d\"},\"panelIndex\":\"f57d14bb-fa23-40ec-96b5-af20d521ca2d\",\"embeddableConfig\":{},\"panelRefName\":\"panel_0\"},{\"version\":\"1.3.18\",\"gridData\":{\"x\":14,\"y\":0,\"w\":16,\"h\":16,\"i\":\"dae768dc-0ff2-4106-a44a-ddf859186e19\"},\"panelIndex\":\"dae768dc-0ff2-4106-a44a-ddf859186e19\",\"embeddableConfig\":{\"title\":\"Namespaces\",\"hidePanelTitles\":false},\"title\":\"Namespaces\",\"panelRefName\":\"panel_1\"},{\"version\":\"1.3.18\",\"gridData\":{\"x\":30,\"y\":0,\"w\":18,\"h\":16,\"i\":\"fcf90cfd-6103-4056-9797-99d63742e31a\"},\"panelIndex\":\"fcf90cfd-6103-4056-9797-99d63742e31a\",\"embeddableConfig\":{\"title\":\"Workloads over Time\",\"hidePanelTitles\":false},\"title\":\"Workloads over Time\",\"panelRefName\":\"panel_2\"},{\"version\":\"1.3.18\",\"gridData\":{\"x\":0,\"y\":16,\"w\":12,\"h\":23,\"i\":\"98298537-dd2d-4eff-9789-071d74158827\"},\"panelIndex\":\"98298537-dd2d-4eff-9789-071d74158827\",\"embeddableConfig\":{\"title\":\"Deployments\",\"hidePanelTitles\":false,\"table\":null,\"vis\":{\"params\":{\"sort\":{\"columnIndex\":0,\"direction\":\"asc\"}}}},\"title\":\"Deployments\",\"panelRefName\":\"panel_3\"},{\"version\":\"1.3.18\",\"gridData\":{\"x\":12,\"y\":16,\"w\":12,\"h\":23,\"i\":\"7b47e126-84a5-4a89-84a2-d7be15e9a0d0\"},\"panelIndex\":\"7b47e126-84a5-4a89-84a2-d7be15e9a0d0\",\"embeddableConfig\":{\"title\":\"StatefulSets\",\"hidePanelTitles\":false,\"table\":null,\"vis\":{\"params\":{\"sort\":{\"columnIndex\":0,\"direction\":\"asc\"}}}},\"title\":\"StatefulSets\",\"panelRefName\":\"panel_4\"},{\"version\":\"1.3.18\",\"gridData\":{\"x\":24,\"y\":16,\"w\":12,\"h\":23,\"i\":\"b5cfabc3-5a7b-401e-a11f-3a9416afc8e8\"},\"panelIndex\":\"b5cfabc3-5a7b-401e-a11f-3a9416afc8e8\",\"embeddableConfig\":{\"title\":\"DaemonSets\",\"hidePanelTitles\":false,\"table\":null,\"vis\":{\"params\":{\"sort\":{\"columnIndex\":0,\"direction\":\"asc\"}}}},\"title\":\"DaemonSets\",\"panelRefName\":\"panel_5\"},{\"version\":\"1.3.18\",\"gridData\":{\"x\":36,\"y\":16,\"w\":12,\"h\":23,\"i\":\"f402fe32-8ab8-446c-9b1b-c0a0e13f28c3\"},\"panelIndex\":\"f402fe32-8ab8-446c-9b1b-c0a0e13f28c3\",\"embeddableConfig\":{\"title\":\"Jobs\",\"hidePanelTitles\":false,\"table\":null,\"vis\":{\"params\":{\"sort\":{\"columnIndex\":0,\"direction\":\"asc\"}}}},\"title\":\"Jobs\",\"panelRefName\":\"panel_6\"},{\"version\":\"1.3.18\",\"gridData\":{\"x\":0,\"y\":39,\"w\":36,\"h\":19,\"i\":\"b335c289-53d2-4190-8255-d228fe28503c\"},\"panelIndex\":\"b335c289-53d2-4190-8255-d228fe28503c\",\"embeddableConfig\":{\"title\":\"Container Resources\",\"hidePanelTitles\":false},\"title\":\"Container Resources\",\"panelRefName\":\"panel_7\"}]","timeRestore":false,"title":"K8s Workloads","version":1},"id":"c69f3210-950d-11ef-a1d8-af81cc9045d1","migrationVersion":{"dashboard":"7.9.3"},"references":[{"id":"fdba71f0-950e-11ef-a1d8-af81cc9045d1","name":"panel_0","type":"visualization"},{"id":"878b8140-954a-11ef-a1d8-af81cc9045d1","name":"panel_1","type":"visualization"},{"id":"7ef9b520-9549-11ef-b4ed-536715f658e8","name":"panel_2","type":"visualization"},{"id":"17c26560-96ce-11ef-a1d8-af81cc9045d1","name":"panel_3","type":"visualization"},{"id":"d7d514b0-96ce-11ef-b4ed-536715f658e8","name":"panel_4","type":"visualization"},{"id":"f6afeab0-96d1-11ef-b4ed-536715f658e8","name":"panel_5","type":"visualization"},{"id":"2029a240-96d3-11ef-a1d8-af81cc9045d1","name":"panel_6","type":"visualization"},{"id":"a8722220-96d4-11ef-b4ed-536715f658e8","name":"panel_7","type":"visualization"}],"type":"dashboard","updated_at":"2025-03-31T06:58:43.928Z","version":"WzQyMiwxXQ=="} -{"exportedCount":12,"missingRefCount":1,"missingReferences":[{"id":"1","type":"version"}]} \ No newline at end of file diff --git a/docs/user/integration/sap-cloud-logging/kyma-dashboard-configmap-otlp.yaml b/docs/user/integration/sap-cloud-logging/kyma-dashboard-configmap-otlp.yaml new file mode 100644 index 0000000000..6c697873f7 --- /dev/null +++ b/docs/user/integration/sap-cloud-logging/kyma-dashboard-configmap-otlp.yaml @@ -0,0 +1,274 @@ +kind: ConfigMap +apiVersion: v1 +metadata: + name: sap-cloud-logging + namespace: kube-public + labels: + busola.io/extension: statics + busola.io/extension-version: '0.5' +data: + dataSources: |- + clsSecret: + resource: + kind: Secret + version: v1 + namespace: sap-cloud-logging-integration + name: sap-cloud-logging + general: |- + externalNodes: + - category: SAP Cloud Logging + icon: display + children: + - label: "Dashboards" + link: "$string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/dashboards#/view/maintained-by-perfx_kyma-content-package_Kyma-Overview\"" + - label: "Discover Logs" + link: "$string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-7d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name','traceId','severityText','body'),index:maintained-by-perfx_otel-content-package_index-pattern-logs-otel-v1)\"" + - label: "Observe Traces" + link: "$string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/observability-traces#/traces\"" + - label: "Discover Metrics" + link: "$string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-7d,to:now))&_a=(columns:!(name,kind,value),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\"" + injections: |- + - name: "Logs for Namespace" + source: "'Discover Logs in SAP Cloud Logging'" + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name','traceId','severityText','body'),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-logs-otel-v1)\", + {'name': $embedResource.metadata.name} + )" + widget: ExternalLink + type: button + copyable: false + targets: + - slot: details-header + location: Namespaces + order: 1 + - name: "Logs for Pod" + source: "'Discover Logs in SAP Cloud Logging'" + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name','traceId','severityText','body'),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@pod@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-logs-otel-v1)\", + {'namespace': $embedResource.metadata.namespace, + 'name': $embedResource.metadata.name } + )" + widget: ExternalLink + type: button + copyable: false + targets: + - slot: details-header + location: Pods + order: 1 + - name: "Logs for Deployment" + source: "'Discover Logs in SAP Cloud Logging'" + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name','traceId','severityText','body'),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@deployment@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-logs-otel-v1)\", + {'namespace': $embedResource.metadata.namespace, + 'name': $embedResource.metadata.name } + )" + widget: ExternalLink + type: button + copyable: false + targets: + - slot: details-header + location: Deployments + order: 1 + - name: "Logs for DaemonSet" + source: "'Discover Logs in SAP Cloud Logging'" + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name','traceId','severityText','body'),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@daemonset@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-logs-otel-v1)\", + {'namespace': $embedResource.metadata.namespace, + 'name': $embedResource.metadata.name } + )" + widget: ExternalLink + type: button + copyable: false + targets: + - slot: details-header + location: DaemonSets + order: 1 + - name: "Logs for StatefulSet" + source: "'Discover Logs in SAP Cloud Logging'" + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name','traceId','severityText','body'),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@statefulset@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-logs-otel-v1)\", + {'namespace': $embedResource.metadata.namespace, + 'name': $embedResource.metadata.name } + )" + widget: ExternalLink + type: button + copyable: false + targets: + - slot: details-header + location: StatefulSets + order: 1 + - name: "Logs for Job" + source: "'Discover Logs in SAP Cloud Logging'" + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name','traceId','severityText','body'),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@job@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-logs-otel-v1)\", + {'namespace': $embedResource.metadata.namespace, + 'name': $embedResource.metadata.name } + )" + widget: ExternalLink + type: button + copyable: false + targets: + - slot: details-header + location: Jobs + order: 1 + - name: "Logs for CronJob" + source: "'Discover Logs in SAP Cloud Logging'" + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name','traceId','severityText','body'),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@cronjob@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-logs-otel-v1)\", + {'namespace': $embedResource.metadata.namespace, + 'name': $embedResource.metadata.name } + )" + widget: ExternalLink + type: button + copyable: false + targets: + - slot: details-header + location: CronJobs + order: 1 + - name: "Logs for Function" + source: "'Discover Logs in SAP Cloud Logging'" + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name','traceId','severityText','body'),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:(serviceName:{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-logs-otel-v1)\", + {'namespace': $embedResource.metadata.namespace, + 'name': $embedResource.metadata.name } + )" + widget: ExternalLink + type: button + copyable: false + targets: + - slot: details-header + location: Functions + order: 1 + - name: "Traces for Namespace" + source: "'Discover Traces in SAP Cloud Logging'" + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!(serviceName,traceGroup,traceId),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace})))),index:maintained-by-perfx_otel-content-package_index-pattern-otel-v1-apm-span)\", + {'namespace': $embedResource.metadata.namespace } + )" + widget: ExternalLink + type: button + copyable: false + targets: + - slot: details-header + location: Namespaces + order: 1 + - name: "Traces for Pod" + source: "'Discover Traces in SAP Cloud Logging'" + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!(serviceName,traceGroup,traceId),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@pod@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-otel-v1-apm-span)\", + {'namespace': $embedResource.metadata.namespace, + 'name': $embedResource.metadata.name } + )" + widget: ExternalLink + type: button + copyable: false + targets: + - slot: details-header + location: Pods + order: 1 + - name: "Traces for Deployment" + source: "'Discover Traces in SAP Cloud Logging'" + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!(serviceName,traceGroup,traceId),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@deployment@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-otel-v1-apm-span)\", + {'namespace': $embedResource.metadata.namespace, + 'name': $embedResource.metadata.name } + )" + widget: ExternalLink + type: button + copyable: false + targets: + - slot: details-header + location: Deployments + order: 1 + - name: "Metrics for Namespace" + source: "'Discover Metrics in SAP Cloud Logging'" + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name',name,kind,value),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace})))),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\", + {'namespace': $embedResource.metadata.name } + )" + widget: ExternalLink + type: button + copyable: false + targets: + - slot: details-header + location: Namespaces + order: 1 + - name: "Metrics for Pod" + source: "'Discover Metrics in SAP Cloud Logging'" + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name',name,kind,value),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@pod@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\", + {'namespace': $embedResource.metadata.namespace, + 'name': $embedResource.metadata.name } + )" + widget: ExternalLink + type: button + copyable: false + targets: + - slot: details-header + location: Pods + order: 1 + - name: "Metrics for Function" + source: "'Discover Metrics in SAP Cloud Logging'" + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name',name,kind,value),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('serviceName':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\", + {'namespace': $embedResource.metadata.namespace, + 'name': $embedResource.metadata.name } + )" + widget: ExternalLink + type: button + copyable: false + targets: + - slot: details-header + location: Functions + order: 1 + - name: "Metrics for Deployment" + source: "'Discover Metrics in SAP Cloud Logging'" + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name',name,kind,value),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@deployment@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\", + {'namespace': $embedResource.metadata.namespace, + 'name': $embedResource.metadata.name } + )" + widget: ExternalLink + type: button + copyable: false + targets: + - slot: details-header + location: Deployments + order: 1 + - name: "Metrics for StatefulSet" + source: "'Discover Metrics in SAP Cloud Logging'" + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name',name,kind,value),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@statefulset@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\", + {'namespace': $embedResource.metadata.namespace, + 'name': $embedResource.metadata.name } + )" + widget: ExternalLink + type: button + copyable: false + targets: + - slot: details-header + location: Statefulsets + order: 1 + - name: "Metrics for DaemonSet" + source: "'Discover Metrics in SAP Cloud Logging'" + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name',name,kind,value),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@daemonset@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\", + {'namespace': $embedResource.metadata.namespace, + 'name': $embedResource.metadata.name } + )" + widget: ExternalLink + type: button + copyable: false + targets: + - slot: details-header + location: Daemonsets + order: 1 + - name: "Metrics for Job" + source: "'Discover Metrics in SAP Cloud Logging'" + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name',name,kind,value),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@job@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\", + {'namespace': $embedResource.metadata.namespace, + 'name': $embedResource.metadata.name } + )" + widget: ExternalLink + type: button + copyable: false + targets: + - slot: details-header + location: Jobs + order: 1 + - name: "Metrics for CronJob" + source: "'Discover Metrics in SAP Cloud Logging'" + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name',name,kind,value),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@cronjob@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\", + {'namespace': $embedResource.metadata.namespace, + 'name': $embedResource.metadata.name } + )" + widget: ExternalLink + type: button + copyable: false + targets: + - slot: details-header + location: Cronjobs + order: 1 diff --git a/docs/user/integration/sap-cloud-logging/kyma-dashboard-configmap.yaml b/docs/user/integration/sap-cloud-logging/kyma-dashboard-configmap.yaml index 6c697873f7..5ae80f9326 100644 --- a/docs/user/integration/sap-cloud-logging/kyma-dashboard-configmap.yaml +++ b/docs/user/integration/sap-cloud-logging/kyma-dashboard-configmap.yaml @@ -22,15 +22,15 @@ data: - label: "Dashboards" link: "$string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/dashboards#/view/maintained-by-perfx_kyma-content-package_Kyma-Overview\"" - label: "Discover Logs" - link: "$string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-7d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name','traceId','severityText','body'),index:maintained-by-perfx_otel-content-package_index-pattern-logs-otel-v1)\"" + link: "$string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-7d,to:now))&_a=(columns:!(kubernetes.pod_name,kubernetes.container_name,log),index:maintained-by-perfx_kyma-content-package_index-pattern-logs-json-kyma)\"" - label: "Observe Traces" - link: "$string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/observability-traces#/traces\"" + link: "$string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/observability-dashboards#/trace_analytics/home\"" - label: "Discover Metrics" link: "$string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-7d,to:now))&_a=(columns:!(name,kind,value),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\"" injections: |- - - name: "Logs for Namespace" + - name: "Logs for namespace" source: "'Discover Logs in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name','traceId','severityText','body'),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-logs-otel-v1)\", + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & '/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!(kubernetes.pod_name,kubernetes.container_name,trace_id,log),filters:!((query:(match_phrase:(kubernetes.namespace_name:{name})))),index:maintained-by-perfx_kyma-content-package_index-pattern-logs-json-kyma)', {'name': $embedResource.metadata.name} )" widget: ExternalLink @@ -40,9 +40,9 @@ data: - slot: details-header location: Namespaces order: 1 - - name: "Logs for Pod" + - name: "Logs for pod" source: "'Discover Logs in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name','traceId','severityText','body'),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@pod@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-logs-otel-v1)\", + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & '/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!(kubernetes.pod_name,kubernetes.container_name,trace_id,log),filters:!((query:(match_phrase:(kubernetes.namespace_name:{namespace}))),(query:(match_phrase:(kubernetes.pod_name:{name})))),index:maintained-by-perfx_kyma-content-package_index-pattern-logs-json-kyma)', {'namespace': $embedResource.metadata.namespace, 'name': $embedResource.metadata.name } )" @@ -53,74 +53,9 @@ data: - slot: details-header location: Pods order: 1 - - name: "Logs for Deployment" + - name: "Logs for Functions" source: "'Discover Logs in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name','traceId','severityText','body'),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@deployment@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-logs-otel-v1)\", - {'namespace': $embedResource.metadata.namespace, - 'name': $embedResource.metadata.name } - )" - widget: ExternalLink - type: button - copyable: false - targets: - - slot: details-header - location: Deployments - order: 1 - - name: "Logs for DaemonSet" - source: "'Discover Logs in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name','traceId','severityText','body'),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@daemonset@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-logs-otel-v1)\", - {'namespace': $embedResource.metadata.namespace, - 'name': $embedResource.metadata.name } - )" - widget: ExternalLink - type: button - copyable: false - targets: - - slot: details-header - location: DaemonSets - order: 1 - - name: "Logs for StatefulSet" - source: "'Discover Logs in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name','traceId','severityText','body'),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@statefulset@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-logs-otel-v1)\", - {'namespace': $embedResource.metadata.namespace, - 'name': $embedResource.metadata.name } - )" - widget: ExternalLink - type: button - copyable: false - targets: - - slot: details-header - location: StatefulSets - order: 1 - - name: "Logs for Job" - source: "'Discover Logs in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name','traceId','severityText','body'),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@job@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-logs-otel-v1)\", - {'namespace': $embedResource.metadata.namespace, - 'name': $embedResource.metadata.name } - )" - widget: ExternalLink - type: button - copyable: false - targets: - - slot: details-header - location: Jobs - order: 1 - - name: "Logs for CronJob" - source: "'Discover Logs in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name','traceId','severityText','body'),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@cronjob@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-logs-otel-v1)\", - {'namespace': $embedResource.metadata.namespace, - 'name': $embedResource.metadata.name } - )" - widget: ExternalLink - type: button - copyable: false - targets: - - slot: details-header - location: CronJobs - order: 1 - - name: "Logs for Function" - source: "'Discover Logs in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name','traceId','severityText','body'),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:(serviceName:{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-logs-otel-v1)\", + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & '/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!(kubernetes.pod_name,kubernetes.container_name,trace_id,log),filters:!((query:(match_phrase:(kubernetes.namespace_name:{namespace}))),(query:(match_phrase:(kubernetes.labels.serverless_kyma-project_io_function-name:{name})))),index:maintained-by-perfx_kyma-content-package_index-pattern-logs-json-kyma)', {'namespace': $embedResource.metadata.namespace, 'name': $embedResource.metadata.name } )" @@ -131,21 +66,9 @@ data: - slot: details-header location: Functions order: 1 - - name: "Traces for Namespace" + - name: "Traces for pods" source: "'Discover Traces in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!(serviceName,traceGroup,traceId),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace})))),index:maintained-by-perfx_otel-content-package_index-pattern-otel-v1-apm-span)\", - {'namespace': $embedResource.metadata.namespace } - )" - widget: ExternalLink - type: button - copyable: false - targets: - - slot: details-header - location: Namespaces - order: 1 - - name: "Traces for Pod" - source: "'Discover Traces in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!(serviceName,traceGroup,traceId),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@pod@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-otel-v1-apm-span)\", + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!(serviceName,traceGroup,traceId),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@pod@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-otel-v1-apm-span)\", {'namespace': $embedResource.metadata.namespace, 'name': $embedResource.metadata.name } )" @@ -156,9 +79,9 @@ data: - slot: details-header location: Pods order: 1 - - name: "Traces for Deployment" + - name: "Traces for deployments" source: "'Discover Traces in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!(serviceName,traceGroup,traceId),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@deployment@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-otel-v1-apm-span)\", + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!(serviceName,traceGroup,traceId),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@deployment@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-otel-v1-apm-span)\", {'namespace': $embedResource.metadata.namespace, 'name': $embedResource.metadata.name } )" @@ -169,9 +92,9 @@ data: - slot: details-header location: Deployments order: 1 - - name: "Metrics for Namespace" + - name: "Metrics for namespace" source: "'Discover Metrics in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name',name,kind,value),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace})))),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\", + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!(name,kind,value),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace})))),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\", {'namespace': $embedResource.metadata.name } )" widget: ExternalLink @@ -181,9 +104,9 @@ data: - slot: details-header location: Namespaces order: 1 - - name: "Metrics for Pod" + - name: "Metrics for pods" source: "'Discover Metrics in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name',name,kind,value),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@pod@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\", + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!(name,kind,value),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@pod@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\", {'namespace': $embedResource.metadata.namespace, 'name': $embedResource.metadata.name } )" @@ -194,22 +117,9 @@ data: - slot: details-header location: Pods order: 1 - - name: "Metrics for Function" - source: "'Discover Metrics in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name',name,kind,value),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('serviceName':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\", - {'namespace': $embedResource.metadata.namespace, - 'name': $embedResource.metadata.name } - )" - widget: ExternalLink - type: button - copyable: false - targets: - - slot: details-header - location: Functions - order: 1 - - name: "Metrics for Deployment" + - name: "Metrics for deployments" source: "'Discover Metrics in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name',name,kind,value),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@deployment@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\", + link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!(name,kind,value),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@deployment@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\", {'namespace': $embedResource.metadata.namespace, 'name': $embedResource.metadata.name } )" @@ -220,55 +130,3 @@ data: - slot: details-header location: Deployments order: 1 - - name: "Metrics for StatefulSet" - source: "'Discover Metrics in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name',name,kind,value),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@statefulset@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\", - {'namespace': $embedResource.metadata.namespace, - 'name': $embedResource.metadata.name } - )" - widget: ExternalLink - type: button - copyable: false - targets: - - slot: details-header - location: Statefulsets - order: 1 - - name: "Metrics for DaemonSet" - source: "'Discover Metrics in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name',name,kind,value),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@daemonset@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\", - {'namespace': $embedResource.metadata.namespace, - 'name': $embedResource.metadata.name } - )" - widget: ExternalLink - type: button - copyable: false - targets: - - slot: details-header - location: Daemonsets - order: 1 - - name: "Metrics for Job" - source: "'Discover Metrics in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name',name,kind,value),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@job@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\", - {'namespace': $embedResource.metadata.namespace, - 'name': $embedResource.metadata.name } - )" - widget: ExternalLink - type: button - copyable: false - targets: - - slot: details-header - location: Jobs - order: 1 - - name: "Metrics for CronJob" - source: "'Discover Metrics in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!('serviceName','resource.attributes.k8s@pod@name',name,kind,value),sort:!(!(time,desc)),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@cronjob@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\", - {'namespace': $embedResource.metadata.namespace, - 'name': $embedResource.metadata.name } - )" - widget: ExternalLink - type: button - copyable: false - targets: - - slot: details-header - location: Cronjobs - order: 1 diff --git a/docs/user/integration/sap-cloud-logging/kyma-dashboard-http-configmap.yaml b/docs/user/integration/sap-cloud-logging/kyma-dashboard-http-configmap.yaml deleted file mode 100644 index 5ae80f9326..0000000000 --- a/docs/user/integration/sap-cloud-logging/kyma-dashboard-http-configmap.yaml +++ /dev/null @@ -1,132 +0,0 @@ -kind: ConfigMap -apiVersion: v1 -metadata: - name: sap-cloud-logging - namespace: kube-public - labels: - busola.io/extension: statics - busola.io/extension-version: '0.5' -data: - dataSources: |- - clsSecret: - resource: - kind: Secret - version: v1 - namespace: sap-cloud-logging-integration - name: sap-cloud-logging - general: |- - externalNodes: - - category: SAP Cloud Logging - icon: display - children: - - label: "Dashboards" - link: "$string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/dashboards#/view/maintained-by-perfx_kyma-content-package_Kyma-Overview\"" - - label: "Discover Logs" - link: "$string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-7d,to:now))&_a=(columns:!(kubernetes.pod_name,kubernetes.container_name,log),index:maintained-by-perfx_kyma-content-package_index-pattern-logs-json-kyma)\"" - - label: "Observe Traces" - link: "$string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/observability-dashboards#/trace_analytics/home\"" - - label: "Discover Metrics" - link: "$string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-7d,to:now))&_a=(columns:!(name,kind,value),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\"" - injections: |- - - name: "Logs for namespace" - source: "'Discover Logs in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & '/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!(kubernetes.pod_name,kubernetes.container_name,trace_id,log),filters:!((query:(match_phrase:(kubernetes.namespace_name:{name})))),index:maintained-by-perfx_kyma-content-package_index-pattern-logs-json-kyma)', - {'name': $embedResource.metadata.name} - )" - widget: ExternalLink - type: button - copyable: false - targets: - - slot: details-header - location: Namespaces - order: 1 - - name: "Logs for pod" - source: "'Discover Logs in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & '/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!(kubernetes.pod_name,kubernetes.container_name,trace_id,log),filters:!((query:(match_phrase:(kubernetes.namespace_name:{namespace}))),(query:(match_phrase:(kubernetes.pod_name:{name})))),index:maintained-by-perfx_kyma-content-package_index-pattern-logs-json-kyma)', - {'namespace': $embedResource.metadata.namespace, - 'name': $embedResource.metadata.name } - )" - widget: ExternalLink - type: button - copyable: false - targets: - - slot: details-header - location: Pods - order: 1 - - name: "Logs for Functions" - source: "'Discover Logs in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & '/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!(kubernetes.pod_name,kubernetes.container_name,trace_id,log),filters:!((query:(match_phrase:(kubernetes.namespace_name:{namespace}))),(query:(match_phrase:(kubernetes.labels.serverless_kyma-project_io_function-name:{name})))),index:maintained-by-perfx_kyma-content-package_index-pattern-logs-json-kyma)', - {'namespace': $embedResource.metadata.namespace, - 'name': $embedResource.metadata.name } - )" - widget: ExternalLink - type: button - copyable: false - targets: - - slot: details-header - location: Functions - order: 1 - - name: "Traces for pods" - source: "'Discover Traces in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!(serviceName,traceGroup,traceId),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@pod@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-otel-v1-apm-span)\", - {'namespace': $embedResource.metadata.namespace, - 'name': $embedResource.metadata.name } - )" - widget: ExternalLink - type: button - copyable: false - targets: - - slot: details-header - location: Pods - order: 1 - - name: "Traces for deployments" - source: "'Discover Traces in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!(serviceName,traceGroup,traceId),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@deployment@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-otel-v1-apm-span)\", - {'namespace': $embedResource.metadata.namespace, - 'name': $embedResource.metadata.name } - )" - widget: ExternalLink - type: button - copyable: false - targets: - - slot: details-header - location: Deployments - order: 1 - - name: "Metrics for namespace" - source: "'Discover Metrics in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!(name,kind,value),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace})))),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\", - {'namespace': $embedResource.metadata.name } - )" - widget: ExternalLink - type: button - copyable: false - targets: - - slot: details-header - location: Namespaces - order: 1 - - name: "Metrics for pods" - source: "'Discover Metrics in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!(name,kind,value),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@pod@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\", - {'namespace': $embedResource.metadata.namespace, - 'name': $embedResource.metadata.name } - )" - widget: ExternalLink - type: button - copyable: false - targets: - - slot: details-header - location: Pods - order: 1 - - name: "Metrics for deployments" - source: "'Discover Metrics in SAP Cloud Logging'" - link: "$prepareUrl($string('https://') & $base64decode($string($clsSecret().data.'dashboards-endpoint')) & \"/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:5000),time:(from:now-1d,to:now))&_a=(columns:!(name,kind,value),filters:!((query:(match_phrase:('resource.attributes.k8s@namespace@name':{namespace}))),(query:(match_phrase:('resource.attributes.k8s@deployment@name':{name})))),index:maintained-by-perfx_otel-content-package_index-pattern-metrics-otel-v1)\", - {'namespace': $embedResource.metadata.namespace, - 'name': $embedResource.metadata.name } - )" - widget: ExternalLink - type: button - copyable: false - targets: - - slot: details-header - location: Deployments - order: 1 diff --git a/docs/user/integration/sap-cloud-logging/secret-example-otlp.yaml b/docs/user/integration/sap-cloud-logging/secret-example-otlp.yaml new file mode 100644 index 0000000000..8d883aec24 --- /dev/null +++ b/docs/user/integration/sap-cloud-logging/secret-example-otlp.yaml @@ -0,0 +1,14 @@ +apiVersion: v1 +kind: Secret +metadata: + name: sap-cloud-logging + namespace: sap-cloud-logging-integration +stringData: + # To ingest telemetry data + # certs/keys should be pasted as is, keeping \n characters + ingest-otlp-endpoint: "" + ingest-otlp-key: "" + ingest-otlp-cert: "" + + # To define Kyma dashboard navigation and deep links, skip if you want to configure shipment only + dashboards-endpoint: "" diff --git a/docs/user/logs.md b/docs/user/logs.md index 01e37d06c6..3b97f2f7c7 100644 --- a/docs/user/logs.md +++ b/docs/user/logs.md @@ -1,606 +1,3 @@ -# Application Logs (OTLP) +# Logs -With application logs, you can debug an application and derive the internal state of an application. The Telemetry module supports observing your applications with logs of the correct severity level and context. - -## Overview - -The Telemetry module provides a log gateway for push-based collection of logs using OTLP and, optionally, an agent for the collection of logs of any container printing logs to the `stdout/stderr` channel running in the Kyma runtime. Kyma modules like [Istio](https://kyma-project.io/#/istio/user/README) contribute access logs. The Telemetry module enriches the data and ships them to your chosen backend (see [Vendors who natively support OpenTelemetry](https://opentelemetry.io/ecosystem/vendors/)). - -You can configure the log gateway and agent with external systems using runtime configuration with a dedicated Kubernetes API ([CRD](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#customresourcedefinitions)) named LogPipeline. -The Log feature is optional. If you don't want to use it, simply don't set up a LogPipeline. - -## Prerequisites - -- Before you can collect logs from a component, it must emit the logs. Typically, it uses a logger framework for the used language runtime (like Node.js) and prints them to the `stdout` or `stderr` channel ([Kubernetes: How nodes handle container logs](https://kubernetes.io/docs/concepts/cluster-administration/logging/#how-nodes-handle-container-logs)). Alternatively, you can use the [OTel SDK](https://opentelemetry.io/docs/languages/) to use the [push-based OTLP format](https://opentelemetry.io/docs/specs/otlp/). - -- If you want to emit the logs to the `stdout/stderr` channel, use structured logs in a JSON format with a logger library like log4J. With that, the log agent can parse your log and enrich all JSON attributes as log attributes, and a backend can use that. - -- If you prefer the push-based alternative with OTLP, also use a logger library like log4J. However, you additionally instrument that logger and bridge it to the OTel SDK. For details, see [OpenTelemetry: New First-Party Application Logs](https://opentelemetry.io/docs/specs/otel/logs/#new-first-party-application-logs). - -## Architecture - -In the Telemetry module, a central in-cluster Deployment of an [OTel Collector](https://opentelemetry.io/docs/collector/) acts as a gateway. The gateway exposes endpoints for the [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/specs/otlp/) for GRPC and HTTP-based communication using the dedicated `telemetry-otlp-logs` service, to which your applications send the logs data. - -You can choose whether you also want an agent, based on a DaemonSet of an OTel Collector. This agent can tail logs of a container from the underlying container runtime. - -![Architecture](./assets/logs-arch.drawio.svg) - -1. Application containers print JSON logs to the `stdout/stderr` channel and are stored by the Kubernetes container runtime under the `var/log` directory and its subdirectories at the related Node. Istio is configured to write access logs to `stdout` as well. -2. If you choose to use the agent, an OTel Collector runs as a [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) (one instance per Node), detects any new log files in the folder, and tails and parses them. -3. An application (exposing logs in OTLP) sends logs to the central log gateway service. Istio is configured to push access logs with OTLP as well. -4. The gateway and agent discover the metadata and enrich all received data with metadata of the source by communicating with the Kubernetes APIServer. Furthermore, they filter data according to the pipeline configuration. -5. Telemetry Manager configures the agent and gateway according to the `LogPipeline` resource specification, including the target backend. Also, it observes the logs flow to the backend and reports problems in the LogPipeline status. -8. The log agent and gateway send the data to the observability system that's specified in your `LogPipeline` resource - either within the Kyma cluster, or, if authentication is set up, to an external observability backend. -9. You can analyze the logs data with your preferred backend. - -### Telemetry Manager - -The LogPipeline resource is watched by Telemetry Manager, which is responsible for generating the custom parts of the OTel Collector configuration. - -![Manager resources](./assets/logs-resources.drawio.svg) - -1. Telemetry Manager watches all LogPipeline resources and related Secrets. -2. Furthermore, Telemetry Manager takes care of the full lifecycle of the gateway Deployment and the agent DaemonSet. Only if you defined a LogPipeline, the gateway and agent are deployed. -3. Whenever the user configuration changes, Telemetry Manager validates it and generates a single configuration for the gateway and agent. -4. Referenced Secrets are copied into one Secret that is mounted to the gateway as well. - -### Log Gateway - -In a Kyma cluster, the log gateway is the central component to which all components can send their individual logs. The gateway collects, enriches, and dispatches the data to the configured backend. For more information, see [Telemetry Gateways](./gateways.md). - -### Log Agent - -If you configure a feature in the `input` section of your LogPipeline, an additional DaemonSet is deployed acting as an agent. The agent is based on an [OTel Collector](https://opentelemetry.io/docs/collector/) and encompasses the collection and conversion of logs from the container runtime. Hereby, the workload container just prints the structured log to the `stdout/stderr` channel. The agent picks them up, parses and enriches them, and sends all data in OTLP to the configured backend. - -## Setting up a LogPipeline - -In the following steps, you can see how to construct and deploy a typical LogPipeline. Learn more about the available [parameters and attributes](resources/02-logpipeline.md). - -### 1. Create a LogPipeline - -To ship logs to a new OTLP output, create a resource of the kind `Logipeline` and save the file (named, for example, `logpipeline.yaml`). - -This configures the underlying OTel Collector with a pipeline for logs and opens a push endpoint that is accessible with the `telemetry-otlp-logs` service. For details, see [Gateway Usage](./gateways.md#usage). The following push URLs are set up: - -- GRPC: `http://telemetry-otlp-logs.kyma-system:4317` -- HTTP: `http://telemetry-otlp-logs.kyma-system:4318` - -The default protocol for shipping the data to a backend is GRPC, but you can choose HTTP instead. Depending on the configured protocol, an `otlp` or an `otlphttp` exporter is used. Ensure that the correct port is configured as part of the endpoint. - - - -#### **GRPC** - -For GRPC, use: - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: LogPipeline -metadata: - name: backend -spec: - output: - otlp: - endpoint: - value: https://backend.example.com:4317 -``` - -#### **HTTP** - -For HTTP, use the **protocol** attribute: - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: LogPipeline -metadata: - name: backend -spec: - output: - otlp: - protocol: http - endpoint: - value: https://backend.example.com:4318 -``` - - - -### 2a. Add Authentication Details From Plain Text - -To integrate with external systems, you must configure authentication details. You can use mutual TLS (mTLS), Basic Authentication, or custom headers: - - - -#### **mTLS** - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: LogPipeline -metadata: - name: backend -spec: - output: - otlp: - endpoint: - value: https://backend.example.com/otlp:4317 - tls: - cert: - value: | - -----BEGIN CERTIFICATE----- - ... - key: - value: | - -----BEGIN RSA PRIVATE KEY----- - ... -``` - -#### **Basic Authentication** - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: LogPipeline -metadata: - name: backend -spec: - output: - otlp: - endpoint: - value: https://backend.example.com/otlp:4317 - authentication: - basic: - user: - value: myUser - password: - value: myPwd -``` - -#### **Token-based authentication with custom headers** - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: LogPipeline -metadata: - name: backend -spec: - output: - otlp: - endpoint: - value: https://backend.example.com/otlp:4317 - headers: - - name: Authorization - prefix: Bearer - value: "myToken" -``` - - -### 2b. Add Authentication Details From Secrets - -Integrations into external systems usually need authentication details dealing with sensitive data. To handle that data properly in Secrets, LogPipeline supports the reference of Secrets. - -Using the **valueFrom** attribute, you can map Secret keys for mutual TLS (mTLS), Basic Authentication, or with custom headers. - -You can store the value of the token in the referenced Secret without any prefix or scheme, and you can configure it in the headers section of the LogPipeline. In the following example, the token has the prefix “Bearer”. - - - -#### **mTLS** - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: LogPipeline -metadata: - name: backend -spec: - output: - otlp: - endpoint: - value: https://backend.example.com/otlp:4317 - tls: - cert: - valueFrom: - secretKeyRef: - name: backend - namespace: default - key: cert - key: - valueFrom: - secretKeyRef: - name: backend - namespace: default - key: key -``` - -#### **Basic Authentication** - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: LogPipeline -metadata: - name: backend -spec: - output: - otlp: - endpoint: - valueFrom: - secretKeyRef: - name: backend - namespace: default - key: endpoint - authentication: - basic: - user: - valueFrom: - secretKeyRef: - name: backend - namespace: default - key: user - password: - valueFrom: - secretKeyRef: - name: backend - namespace: default - key: password -``` - -#### **Token-based authentication with custom headers** - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: LogPipeline -metadata: - name: backend -spec: - output: - otlp: - endpoint: - value: https://backend.example.com:4317 - headers: - - name: Authorization - prefix: Bearer - valueFrom: - secretKeyRef: - name: backend - namespace: default - key: token -``` - - - -The related Secret must have the referenced name, be located in the referenced namespace, and contain the mapped key. See the following example: - -```yaml -kind: Secret -apiVersion: v1 -metadata: - name: backend - namespace: default -stringData: - endpoint: https://backend.example.com:4317 - user: myUser - password: XXX - token: YYY -``` - -### 3. Rotate the Secret - -Telemetry Manager continuously watches the Secret referenced with the **secretKeyRef** construct. You can update the Secret’s values, and Telemetry Manager detects the changes and applies the new Secret to the setup. - -> [!TIP] -> If you use a Secret owned by the [SAP BTP Service Operator](https://github.com/SAP/sap-btp-service-operator), you can configure an automated rotation using a `credentialsRotationPolicy` with a specific `rotationFrequency` and don’t have to intervene manually. - -### 4. Activate Application Input - -To enable collection of logs printed by containers to the `stdout/stderr` channel, define a LogPipeline that has the `application` section enabled as input: - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: LogPipeline -metadata: - name: backend -spec: - input: - application: - enabled: true - output: - otlp: - ... -``` - -By default, input is collected from all namespaces, except the system namespaces `kube-system`, `istio-system`, `kyma-system`, which are excluded by default. - -To filter your application logs by namespace or container, use an input spec to restrict or specify which resources you want to include. For example, you can define the namespaces to include in the input collection, exclude namespaces from the input collection, or choose that only system namespaces are included. Learn more about the available [parameters and attributes](resources/02-logpipeline.md). - -The following pipeline collects input from all namespaces excluding `kyma-system` and only from the `istio-proxy` containers: - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: LogPipeline -metadata: - name: backend -spec: - input: - application: - enabled: true - namespaces: - exclude: - - myNamespace - containers: - exclude: - - myContainer - otlp: - ... -``` - -After tailing the log files from the container runtime, the payload of the log lines is transformed into an OTLP entry. Learn more about the flow of the log record through the steps and the available log attributes in the following stages: - -- [Log Tailing](#log-tailing) -- [JSON Parsing](#json-parsing) -- [Severity Parsing](#severity-parsing) -- [Trace Parsing](#trace-parsing) -- [Log Body Determination](#log-body-determination) - -The following example assumes that there’s a container `myContainer` of Pod `myPod`, running in namespace `myNamespace`, logging to `stdout` with the following log message in the JSON format: - -```json -{ - "level": "warn", - "message": "This is the actual message", - "tenant": "myTenant", - "traceID": "123" -} -``` - -#### Log Tailing - -The agent reads the log message from a log file managed by the container runtime. The file name contains namespace, Pod and Container information that will be available later as log attributes. The raw log record looks similar to the following example: - -```json -{ - "time": "2022-05-23T15:04:52.193317532Z", - "stream": "stdout", - "_p": "F", - "log": "{\"level\": \"warn\",\"message\": \"This is the actual message\",\"tenant\": \"myTenant\",\"trace_id\": \"123\"}" -} -``` - -After the tailing, the created OTLP record looks like the following example: - -```json -{ - "time": "2022-05-23T15:04:52.100000000Z", - "observedTime": "2022-05-23T15:04:52.200000000", - "attributes": { - "log.file.path": "/var/log/pods/myNamespace_myPod-/myContainer/.log", - "log.iostream": "stdout" - }, - "resourceAttributes": { - "k8s.container.name": "myContainer", - "k8s.container.restart_count": "", - "k8s.pod.name": "myPod", - "k8s.namespace.name": "myNamespace" - }, - "body": "{\"level\": \"warn\",\"message\": \"This is the actual message\",\"tenant\": \"myTenant\",\"trace_id\": \"123\"}" -} -``` - -All information identifying the source of the log (like the Container, Pod and namespace name) are enriched as resource attributes following the [Kubernetes conventions](https://opentelemetry.io/docs/specs/semconv/resource/k8s/). Further metadata - like the original file name and channel - are enriched as log attributes following the [log attribute conventions](https://opentelemetry.io/docs/specs/semconv/general/logs/). The **time** value provided in the container runtime log entry is used as **time** attribute in the new OTel record, as it is very close to the actual time when the log happened. Additionally, the **observedTime** is set with the time when the agent actual read the log record as recommended by the [OTel log specification](https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-observedtimestamp). The log payload is moved to the OTLP **body** field. - -#### JSON Parsing - -If the value of the **body** is a JSON document, the value is parsed and all JSON root attributes are enriched as additional log attributes. The original body is moved into the **log.original** attribute (managed with the LogPipeline attribute **input.application.keepOriginalBody**: `true`). - -The resulting OTLP record looks like the following example: - -```json -{ - "time": "2022-05-23T15:04:52.100000000Z", - "observedTime": "2022-05-23T15:04:52.200000000", - "attributes": { - "log.file.path": "/var/log/pods/myNamespace_myPod-/myContainer/.log", - "log.iostream": "stdout", - "log.original": "{\"level\": \"warn\",\"message\": \"This is the actual message\",\"tenant\": \"myTenant\",\"trace_id\": \"123\"}", - "level": "warn", - "tenant": "myTenant", - "trace_id": "123", - "message": "This is the actual message" - }, - "resourceAttributes": { - "k8s.container.name": "myContainer", - "k8s.container.restart_count": "", - "k8s.pod.name": "myPod", - "k8s.namespace.name": "myNamespace" - }, - "body": "" -} -``` - -#### Severity Parsing - -Typically, a log message has a log level written to a field `level`. Based on that, the agent tries to parse the log attribute **level** with a [severity parser](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/operators/severity_parser.md). If that is successful, the log attribute is transformed into the OTel attributes **severityText** and **severityNumber**. - -#### Trace Parsing - -OTLP natively supports attaching trace context to log records. If possible, the log agent parses the following log attributes according to the [W3C-Tracecontext specification](https://www.w3.org/TR/trace-context/#traceparent-header): - -- **trace_id** -- **span_id** -- **trace_flags** -- **traceparent** - -#### Log Body Determination - -Because the actual log message is typically located in the **body** attribute, the agent moves a log attribute called **message** (or **msg**) into the **body**. - -At this point, before further enrichment, the resulting overall log record looks like the following example: - -```json -{ - "time": "2022-05-23T15:04:52.100000000Z", - "observedTime": "2022-05-23T15:04:52.200000000", - "attributes": { - "log.file.path": "/var/log/pods/myNamespace_myPod-/myContainer/.log", - "log.iostream": "stdout", - "log.original": "{\"level\": \"warn\",\"message\": \"This is the actual message\",\"tenant\": \"myTenant\",\"trace_id\": \"123\"}", - "tenant": "myTenant", - }, - "resourceAttributes": { - "k8s.container.name": "myContainer", - "k8s.container.restart_count": "", - "k8s.pod.name": "myPod", - "k8s.namespace.name": "myNamespace" - }, - "body": "This is the actual message", - "severityNumber": 13, - "severityTex": "warn", - "trace_id": 123 -} -``` - -### 5. Deactivate OTLP Logs - -If you have more than one backend, you can specify from which `input` logs are pushed to each backend. For example, if OTLP logs should go to one backend and only logs from the tail input to the other backend, then disable the OTLP input for the second backend. - -By default, `otlp` input is enabled. - -To drop the push-based OTLP logs that are received by the log gateway, define a LogPipeline that has the `otlp` section disabled as an input: - -```yaml -apiVersion: telemetry.kyma-project.io/v1alpha1 -kind: LogPipeline -metadata: - name: backend -spec: - input: - application: - enabled: true - otlp: - disabled: true - output: - otlp: - endpoint: - value: https://backend.example.com:4317 -``` - -With this, the agent starts collecting all container logs, while the push-based OTLP logs are dropped by the gateway. - -### 6. Deploy the Pipeline - -To activate the LogPipeline, apply the `logpipeline.yaml` resource file in your cluster: - -```bash -kubectl apply -f logpipeline.yaml -``` - -### Result - -You activated a LogPipeline and logs start streaming to your backend. - -To check that the pipeline is running, wait until the status conditions of the LogPipeline in your cluster have status `True`: - -```bash -kubectl get logpipeline -NAME CONFIGURATION GENERATED GATEWAY HEALTHY AGENT HEALTHY FLOW HEALTHY -backend True True True True -``` - -## Kyma Modules With Logging Capabilities - -Kyma bundles modules that can be involved in user flows. If you want to collect all logs of all modules, enable the `application` input for the `kyma-system` namespace. - -### Istio - -The Istio module is crucial as it provides the [Ingress Gateway](https://istio.io/latest/docs/tasks/traffic-management/ingress/ingress-control/). Typically, this is where external requests enter the cluster scope. Furthermore, every component that’s part of the Istio Service Mesh runs an Istio proxy. Using the Istio telemetry API, you can enable access logs for the Ingress Gateway and the proxies individually. - -The Istio module is configured with an [extension provider](https://istio.io/latest/docs/tasks/observability/telemetry/) called `kyma-logs`. To activate the provider on the global mesh level using the Istio [Telemetry API](https://istio.io/latest/docs/reference/config/telemetry), place a resource to the `istio-system` namespace. - -The following example configures all Istio proxies with the `kyma-logs` extension provider, which, by default, reports access logs to the log gateway of the Telemetry module. - -```yaml -apiVersion: telemetry.istio.io/v1 -kind: Telemetry -metadata: - name: mesh-default - namespace: istio-system -spec: - accessLogging: - - providers: - - name: kyma-logs -``` - -## Operations - -A LogPipeline runs several OTel Collector instances in your cluster. This Deployment serves OTLP endpoints and ships received data to the configured backend. - -The Telemetry module ensures that the OTel Collector instances are operational and healthy at any time, for example, with buffering and retries. However, there may be situations when the instances drop logs, or cannot handle the log load. - -To detect and fix such situations, check the [pipeline status](./resources/02-logpipeline.md#logpipeline-status) and check out [Troubleshooting](#troubleshooting). If you have set up [pipeline health monitoring](./04-metrics.md#5-monitor-pipeline-health), check the alerts and reports in an integrated backend like [SAP Cloud Logging](./integration/sap-cloud-logging/README.md#use-sap-cloud-logging-alerts). - -> [! WARNING] -> It's not recommended to access the metrics endpoint of the used OTel Collector instances directly, because the exposed metrics are no official API of the Telemetry module. Breaking changes can happen if the underlying OTel Collector version introduces such. -> Instead, use the [pipeline status](./resources/02-logpipeline.md#logpipeline-status). - -## Limitations - -- **Throughput**: - - When pushing OTLP logs of an average size of 2KB to the log gateway, using its default configuration (two instances), the Telemetry module can process approximately 12,000 logs per second (LPS). To ensure availability, the log gateway runs with multiple instances. For higher throughput, manually scale out the gateway by increasing the number of replicas. See [Module Configuration and Status](https://kyma-project.io/#/telemetry-manager/user/01-manager?id=module-configuration). Ensure that the chosen scaling factor does not exceed the maximum throughput of the backend, as it may refuse logs if the rate is too high. - - For example, to scale out the gateway for scenarios like a `Large` instance of SAP Cloud Logging (up to 30,000 LPS), you can raise the throughput to about 20,000 LPS by increasing the number of replicas to 4 instances. - - The log agent, running one instance per node, handles tailing logs from stdout using the `runtime` input. When writing logs of an average size of 2KB to stdout, a single log agent instance can process approximately 9,000 LPS. -- **Load Balancing With Istio**: By design, the connections to the gateway are long-living connections (because OTLP is based on gRPC and HTTP/2). For optimal scaling of the gateway, the clients or applications must balance the connections across the available instances, which is automatically achieved if you use an Istio sidecar. If your application has no Istio sidecar, the data is always sent to one instance of the gateway. -- **Unavailability of Output**: For up to 5 minutes, a retry for data is attempted when the destination is unavailable. After that, data is dropped. -- **No Guaranteed Delivery**: The used buffers are volatile. If the gateway or agent instances crash, logs data can be lost. -- **Multiple LogPipeline Support**: The maximum amount of LogPipeline resources is 5. - -## Troubleshooting - -### No Logs Arrive at the Backend - -**Symptom**: - -- No logs arrive at the backend. -- In the LogPipeline status, the `TelemetryFlowHealthy` condition has status **GatewayAllTelemetryDataDropped** or **AgentAllTelemetryDataDropped**. - -**Cause**: Incorrect backend endpoint configuration (such as using the wrong authentication credentials) or the backend is unreachable. - -**Solution**: - -1. Check the error logs for the affected Pod by calling `kubectl logs -n kyma-system {POD_NAME}`: - - For **GatewayAllTelemetryDataDropped**, check Pod `telemetry-log-gateway`. - - For **AgentAllTelemetryDataDropped**, check Pod `telemetry-log-agent`. -2. Check if the backend is up and reachable. -3. Fix the errors. - -### Not All Logs Arrive at the Backend - -**Symptom**: - -- The backend is reachable and the connection is properly configured, but some logs are refused. -- In the LogPipeline status, the `TelemetryFlowHealthy` condition has status **GatewaySomeTelemetryDataDropped** or **AgentSomeTelemetryDataDropped**. - -**Cause**: It can happen due to a variety of reasons - for example, the backend is limiting the ingestion rate. - -**Solution**: - -1. Check the error logs for the affected Pod by calling `kubectl logs -n kyma-system {POD_NAME}`: - - For **GatewaySomeTelemetryDataDropped**, check Pod `telemetry-log-gateway`. - - For **AgentSomeTelemetryDataDropped**, check Pod `telemetry-log-agent`. -2. Check your observability backend to investigate potential causes. -3. If the backend is limiting the rate by refusing logs, try the following options: - - Option 1: Increase maximum backend ingestion rate. For example, by scaling out the SAP Cloud Logging instances. - - Option 2: Reduce emitted logs by re-configuring the LogPipeline. For example, by disabling certain inputs or applying namespace filters. - - Option 3: Reduce emitted logs in your applications. -4. Otherwise, take the actions appropriate to the cause indicated in the logs. - -### No Access Logs With Push-Based OTLP LogPipeline - -**Symptom**: The Istio telemetry resource uses a push-based OTLP LogPipeline. Even though this pipeline is healthy, access logs do not arrive at the backend. - -**Cause**: Istio cannot discover the OTLP endpoint. - -**Solution**: Recreate the Istio telemetry resource. For details, see [Istio](#istio). - -### Gateway Throttling - -**Symptom**: In the LogPipeline status, the `TelemetryFlowHealthy` condition has status **GatewayThrottling**. - -**Cause**: Gateway cannot receive logs at the given rate. - -**Solution**: Manually scale out the gateway by increasing the number of replicas for the log gateway. See [Module Configuration and Status](https://kyma-project.io/#/telemetry-manager/user/01-manager?id=module-configuration). +This content moved to [Collecting Logs](./collecting-logs/README.md). diff --git a/docs/user/monitor-pipeline-health.md b/docs/user/monitor-pipeline-health.md new file mode 100644 index 0000000000..464905bcf4 --- /dev/null +++ b/docs/user/monitor-pipeline-health.md @@ -0,0 +1,69 @@ +# Monitor Pipeline Health + +The Telemetry module is designed to be reliable and resilient. However, there may be situations when the instances drop data or cannot handle the load, and you must take action. + +## Overview + +The Telemetry module automatically handles temporary issues to prevent data loss and ensure that the OTel Collector instances of your pipelines are operational and healthy. For example, if your backend is temporarily unavailable, the module buffers your data and attempts to resend it when the connection is restored. + +The Telemetry module continuously monitors the health of your pipelines (see [Self Monitor](./architecture/README.md#self-monitor)). To ensure that your Telemetry pipelines operate reliably, you can monitor their health data in the following ways: + +- Perform manual checks by inspecting the status conditions of your pipeline resources with `kubectl`. +- Set up continuous monitoring by using a MetricPipeline to export health metrics to your observability backend, where you can set up dashboards and alerts. + +## Check Pipeline Status + +For a quick check, you can inspect the `status` of a pipeline resource directly. + +1. Run `kubectl get` for the pipeline that you want to inspect: + - For LogPipeline: `kubectl get logpipeline ` + - For TracePipeline: `kubectl get tracepipeline ` + - For MetricPipeline: `kubectl get tracepipeline ` +2. Review the output. A healthy pipeline shows `True` for all status conditions. + + ```txt + NAME CONFIGURATION GENERATED GATEWAY HEALTHY FLOW HEALTHY + backend True True True + ``` + +3. If any condition is `False`, investigate problem and fix it. + +To understand the meaning of each status condition, see the detailed reference for each pipeline type: + +- [LogPipeline Status](https://kyma-project.io/#/telemetry-manager/user/resources/02-logpipeline?id=logpipeline-status) +- [TracePipeline Status](https://kyma-project.io/#/telemetry-manager/user/resources/04-tracepipeline?id=tracepipeline-status) +- [MetricPipeline Status](https://kyma-project.io/#/telemetry-manager/user/resources/05-metricpipeline?id=metricpipeline-status) + +## Set Up Health Monitoring and Alerts + +For production environments, set up continuous monitoring by exporting the health metrics to your observability backend, where you can create dashboards and configure alerts using alert rules. For an example, see [Integrate With SAP Cloud Logging](./integration/sap-cloud-logging/README.md) + +> [!WARNING] +> Do not scrape the metrics endpoint of the OpenTelemetry Collector instances. These metrics are an internal implementation detail and are subject to breaking changes when the underlying Collector is updated. For stable health monitoring, rely on the status conditions of your LogPipeline, MetricPipeline, or TracePipeline custom resources. + +To collect these health metrics, you must have at least one active MetricPipeline in your cluster. This pipeline automatically collects and exports health data for all of your pipelines, including LogPipeline and TracePipeline resources. + +The Telemetry module emits the following metrics for health monitoring: + +- `kyma.resource.status.conditions`: Represents the status of a specific condition on a resource. It is available for all pipelines and the main `Telemetry` resource. + Values: `1` ("True"), `0` ("False"), or `-1` ("Unknown") + Specific attributes: + - `metric.attributes.type`: The type of the status condition + - `metric.attributes.status`: The status of the condition + - `metric.attributes.reason`: A programmatic identifier indicating the reason for the condition's last transition +- `kyma.resource.status.state`: Represents the overall state of the main `Telemetry` resource. + Values: `1` ("Ready") or `0` ("Not Ready") + Specific attributes: `state`: The value of the `status.state` field +- Additionally, the following attributes are attached to all health metrics to identify the source resource: + - `k8s.resource.group`: The group of the resource + - `k8s.resource.version`: The version of the resource + - `k8s.resource.kind`: The kind of the resource + - `k8s.resource.name`: The name of the resource + +To create an alert, define a rule that triggers on a specific metric value. For example, to create an alert that fires if a pipeline's `TelemetryFlowHealthy` condition becomes "False" (indicating data flow issues), use the following PromQL query: + +```txt +min by (k8s_resource_name) ((kyma_resource_status_conditions{type="TelemetryFlowHealthy",k8s_resource_kind="metricpipelines"})) == 0 +``` + +If there are issues with one of the pipelines, see [Troubleshooting for the Telemetry Module](troubleshooting.md). diff --git a/docs/user/otlp-input.md b/docs/user/otlp-input.md new file mode 100644 index 0000000000..7ba2ac3420 --- /dev/null +++ b/docs/user/otlp-input.md @@ -0,0 +1,70 @@ +# Set Up the OTLP Input + +Use the default OTLP input to collect telemetry data from your instrumented applications and customize how that data is processed by the pipeline. You can specify which input data goes to which backend. If you're using Istio, data is collected from the service mesh. + +## Overview + +When you create any LogPipeline, TracePipeline, or MetricPipeline, the Telemetry module automatically deploys the respective gateway. This opens a stable, cluster-internal [OTLP](https://opentelemetry.io/docs/specs/otel/protocol/) endpoint for each signal type, ready to receive data from your applications. + +Each endpoint listens on port `4317` for gRPC (default) and on port `4318` for HTTP. + +![OTLP Input](./assets/otlp-input.drawio.svg) + +## Configure Your Application's OTLP Exporter + +To send data from your application, first instrument your code using an [OTel SDK](https://opentelemetry.io/docs/languages/) for your programming language. The SDK's OTLP exporter sends the collected telemetry data to a backend. + +It's recommended that you configure the exporter's destination by setting the standard [environment variables](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/#otel_exporter_otlp_traces_endpoint) in your application's deployment. This method avoids hardcoding endpoints in your application code. + +Use the following environment variables to set the OTLP endpoint for each signal type: + +- Traces gRPC: `export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://telemetry-otlp-traces.kyma-system:4317"` +- Traces HTTP: `export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://telemetry-otlp-traces.kyma-system:4318/v1/traces"` +- Metrics gRPC: `export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT="http://telemetry-otlp-metrics.kyma-system:4317"` +- Metrics HTTP: `export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT="http://telemetry-otlp-metrics.kyma-system:4318/v1/metrics"` +- Logs gRPC: `export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT="http://telemetry-otlp-logs.kyma-system:4317"` +- Logs HTTP: `export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT="http://telemetry-otlp-logs.kyma-system:4318/v1/logs"` + +> [!NOTE] +> If your cluster uses Istio, communication with these endpoints is automatically secured with mTLS. For details, see [Istio Integration](./architecture/istio-integration.md). + +## Verify the Endpoints + +To see whether you've set up your gateways and their push endpoints successfully, check the status of the default `Telemetry` resource: + +```sh +kubectl -n kyma-system get telemetries.operator.kyma-project.io default -oyaml +``` + +The output shows the available endpoints and the pipeline health under the status.endpoints section: + +```yaml + endpoints: + metrics: + grpc: http://telemetry-otlp-metrics.kyma-system:4317 + http: http://telemetry-otlp-metrics.kyma-system:4318 + traces: + grpc: http://telemetry-otlp-traces.kyma-system:4317 + http: http://telemetry-otlp-traces.kyma-system:4318 + logs: + grpc: http://telemetry-otlp-logs.kyma-system:4317 + http: http://telemetry-otlp-logs.kyma-system:4318 +``` + +## Route Specific Inputs to Different Backends + +For logs and metrics: If you have multiple pipelines sending data to different backends, you can specify which inputs are active for each pipeline. This is useful if you want one pipeline to handle only OTLP data and another to handle only data from a different source. + +> [!TIP] +> For more granular control, you can also filter incoming OTLP data by namespace. For details, see [Filter Logs](./filter-and-process/filter-logs.md) and [Filter Metrics](./filter-and-process/filter-metrics.md). + +For example, if you want to analyze **otlp** input data in one backend and only data from the log-specific **application** input in another backend, then disable the **otlp** input for the second backend. By default, **otlp** input is enabled. + +```yaml +... + input: + application: + enabled: true + otlp: + disabled: true +``` diff --git a/docs/user/pipelines.md b/docs/user/pipelines.md new file mode 100644 index 0000000000..96bec2af26 --- /dev/null +++ b/docs/user/pipelines.md @@ -0,0 +1,66 @@ +# Telemetry Pipeline API + +To collect and export telemetry data from your Kyma cluster, you define one or more pipelines for each signal type (logs, traces, metrics). You choose which data to collect and to which backend it's sent. + +## Pipeline Structure + +You define Telemetry pipelines using three dedicated Kubernetes CRDs that extend the Kubernetes API: LogPipeline, TracePipeline, and MetricPipeline. + +A pipeline defines how the data flows from the original data sources through the respective inputs and a series of processing steps to the backend you defined as output. + +![structure](./assets/pipeline-structure.drawio.svg) + +The pipelines use the [OpenTelemetry Protocol](https://opentelemetry.io/docs/specs/otel/protocol/) (OTLP) as the primary method to ingest and export data, which gives you the flexibility to integrate with a wide range of observability backends. + +While each pipeline is tailored to a specific signal, they all share a common structure: + +```yaml +apiVersion: telemetry.kyma-project.io/v1alpha1 +kind: # Choose pipeline kind depending on signal type +metadata: + name: my-observability-backend +spec: + input: # Enable additional inputs depending on signal type + otlp: + ... + output: + otlp: # Integrate with your observability backend + endpoint: + ... +``` + +## Pipeline Types + +The **kind** attribute in the CRD specifies the type of telemetry data that the pipeline handles. + +- LogPipeline: Collects logs from your application containers' standard output (`stdout`) and standard error (`stderr`), and from OTLP sources. It parses these logs, extracts useful information, and forwards them to your configured backend. +- TracePipeline: Collects trace data from OTLP sources, which show how requests flow between different components of your application. To collect Istio traces generated by applications or the service mesh, you must enable tracing within Istio using the Istio `Telemetry` CRD. +- MetricPipeline: Collects metrics from OTLP sources, Prometheus-annotated workloads, the Istio service mesh, and the Kubernetes runtime. + +## Input + +In the **spec.input** section, you define the sources of your telemetry data. This section is the primary difference between the pipeline types. + +All pipelines share **otlp** as the default input and can be configured with additional, signal-specific inputs: + +By default, the **otlp** input is enabled for all signal types, which provisions a cluster-internal endpoint accepting OTLP data. For details, see [Set Up the OTLP Input](./otlp-input.md). + +Additionally, you can apply specific **input** configurations for each signal type: + +- LogPipeline: The **application** input is enabled by default. Additionally, you can collect Istio access logs through the default **otlp** input. For both inputs, you can restrict from which Kubernetes resources you want to collect signals. For details, see [Configure Application Logs](./collecting-logs/application-input.md) and [Configure Istio Access Logs](./collecting-logs/istio-support.md). +- TracePipeline: Tracing is a push-based model, so **otlp** is the only available input. The pipeline's OTLP endpoint receives span data pushed from your applications and Istio proxies. For Istio tracing, you can configure the sampling rate and apply individual settings to namespaces or workloads (see [Configure Istio Tracing](./collecting-traces/istio-support.md)). +- MetricPipeline: You can select which metrics are collected by enabling inputs: **prometheus** (for scraping annotated workloads), **runtime** (for Kubernetes resource metrics), and **istio** (for service mesh metrics). You can filter all inputs by namespace. For details, see [Collect Istio Metrics](./collecting-metrics/istio-input.md), [Collect Prometheus Metrics](./collecting-metrics/prometheus-input.md), and [Collect Runtime Metrics](./collecting-metrics/runtime-input.md). + +## Filtering and Processing + +You can control the volume and focus of your telemetry data by filtering it based on Kubernetes resources like namespaces, containers, and workloads. For details, see [Filtering and Processing Data](./filter-and-process/README.md). + +All pipelines automatically enrich telemetry data with Kubernetes resource attributes, such as Pod name, namespace, and labels. With this context information, you can easily identify the source of telemetry data in your backend. For details, see [Automatic Data Enrichment](./filter-and-process/automatic-data-enrichment.md). + +## Output + +In the **spec.output** section, you define the destination for your telemetry data. Each pipeline resource supports exactly one output, which sends data using OTLP. + +You must specify the endpoint address of your observability backend. You can also configure the protocol (gRPC or HTTP) and the authentication details required to connect securely. For details, see [Integrate With Your OTLP Backend](./integrate-otlp-backend/README.md). + +To send the same signal to multiple backends, create a separate pipeline resource for each destination. For details, see [Route Specific Inputs to Different Backends](./otlp-input.md). diff --git a/docs/user/resources/01-telemetry.md b/docs/user/resources/01-telemetry.md index d3a38c08d4..78b5f000f8 100644 --- a/docs/user/resources/01-telemetry.md +++ b/docs/user/resources/01-telemetry.md @@ -45,6 +45,9 @@ Status: metrics: grpc: http://telemetry-otlp-metrics.kyma-system:4317 http: http://telemetry-otlp-metrics.kyma-system:4318 + logs: + grpc: http://telemetry-otlp-logs.kyma-system:4317 + http: http://telemetry-otlp-logs.kyma-system:4318 conditions: - lastTransitionTime: "2023-09-01T15:28:28Z" message: All log components are running @@ -133,7 +136,7 @@ For details, see the [Telemetry specification file](https://github.com/kyma-proj -The `state` attribute of the Telemetry CR is derived from the combined state of all the subcomponents, namely, from the condition types `LogComponentsHealthy`, `TraceComponentsHealthy` and `MetricComponentsHealthy`. +The **state** attribute of the Telemetry CR is derived from the combined state of all the subcomponents, namely, from the condition types `LogComponentsHealthy`, `TraceComponentsHealthy` and `MetricComponentsHealthy`. ### Log Components State diff --git a/docs/user/resources/02-logpipeline.md b/docs/user/resources/02-logpipeline.md index 4454572e15..dc56914766 100644 --- a/docs/user/resources/02-logpipeline.md +++ b/docs/user/resources/02-logpipeline.md @@ -8,58 +8,74 @@ kubectl get crd logpipeline.telemetry.kyma-project.io -o yaml ## Sample Custom Resource -The following LogPipeline object defines a pipeline integrating with the HTTP/JSON-based output. It uses basic authentication and excludes application logs emitted by `istio-proxy` containers. +The following LogPipeline object defines a pipeline integrating with the otlp output. It uses mTLS taking connection details from a Secret, excludes OTLP logs from "namespaceA" and includes application logs emitted in "namespaceB". ```yaml apiVersion: telemetry.kyma-project.io/v1alpha1 kind: LogPipeline metadata: - name: custom-fluentd - generation: 2 + name: backend spec: + otlp: + namespaces: + exclude: + - namespaceA input: application: - containers: - exclude: - - istio-proxy - namespaces: {} + namespaces: + include: + - namespaceB + enabled: true + keepOriginalBody: true output: - http: - dedot: true - host: + otlp: + protocol: grpc + endpoint: valueFrom: secretKeyRef: - key: Fluentd-endpoint - name: custom-fluentd - namespace: default - password: - valueFrom: - secretKeyRef: - key: Fluentd-password - name: custom-fluentd - namespace: default - tls: {} - uri: /customindex/kyma - user: - valueFrom: - secretKeyRef: - key: Fluentd-username - name: custom-fluentd + key: ingest-otlp-endpoint + name: mySecret namespace: default + tls: + cert: + valueFrom: + secretKeyRef: + key: ingest-otlp-cert + name: mySecret + namespace: default + key: + valueFrom: + secretKeyRef: + key: ingest-otlp-key + name: mySecret + namespace: default status: conditions: - - lastTransitionTime: "2024-02-28T22:48:24Z" - message: Fluent Bit DaemonSet is ready - observedGeneration: 2 + - lastTransitionTime: "2025-06-13T08:58:38Z" + message: Log gateway Deployment is ready + observedGeneration: 4 + reason: GatewayReady + status: "True" + type: GatewayHealthy + - lastTransitionTime: "2025-03-30T20:32:39Z" + message: LogPipeline specification is successfully applied to the configuration + of Log gateway + observedGeneration: 4 + reason: GatewayConfigured + status: "True" + type: ConfigurationGenerated + - lastTransitionTime: "2025-03-30T20:32:39Z" + message: Log agent DaemonSet is ready + observedGeneration: 4 reason: AgentReady status: "True" type: AgentHealthy - - lastTransitionTime: "2024-02-28T22:48:11Z" - message: "" - observedGeneration: 2 - reason: ConfigurationGenerated + - lastTransitionTime: "2025-07-07T05:23:03Z" + message: No problems detected in the telemetry flow + observedGeneration: 4 + reason: FlowHealthy status: "True" - type: ConfigurationGenerated + type: TelemetryFlowHealthy ``` For further examples, see the [samples](https://github.com/kyma-project/telemetry-manager/tree/main/samples) directory. diff --git a/docs/user/resources/05-metricpipeline.md b/docs/user/resources/05-metricpipeline.md index 9d0a3f0b2f..572bdb5ef7 100644 --- a/docs/user/resources/05-metricpipeline.md +++ b/docs/user/resources/05-metricpipeline.md @@ -18,13 +18,12 @@ metadata: generation: 1 spec: input: - application: - prometheus: - enabled: false - istio: - enabled: false - runtime: - enabled: false + prometheus: + enabled: false + istio: + enabled: false + runtime: + enabled: false output: otlp: endpoint: diff --git a/docs/user/resources/README.md b/docs/user/resources/README.md index 2a6b7f1e99..6cc810f283 100644 --- a/docs/user/resources/README.md +++ b/docs/user/resources/README.md @@ -1,6 +1,6 @@ # Telemetry Resources -The API of the Telemetry module is based on Kubernetes Custom Resource Definitions (CRD), which are extend the Kubernetes API with custom additions. To inspect the specification of the Telemetry module API, see: +The API of the Telemetry module is based on Kubernetes CustomResourceDefinitions (CRDs), which extend the Kubernetes API with custom additions. To inspect the specification of the Telemetry module API, see: - [Telemetry CRD](./01-telemetry.md) - [LogPipeline CRD](./02-logpipeline.md) diff --git a/docs/user/troubleshooting.md b/docs/user/troubleshooting.md new file mode 100644 index 0000000000..08223ea471 --- /dev/null +++ b/docs/user/troubleshooting.md @@ -0,0 +1,168 @@ +# Troubleshooting for the Telemetry Module + +Troubleshoot problems related to the Telemetry module and its pipelines. + +## No Data Arrive at the Backend + +### Symptom + +- No data arrive at the backend. +- In the respective pipeline status, the `TelemetryFlowHealthy` condition has status **GatewayAllTelemetryDataDropped** or **AgentAllTelemetryDataDropped**. + +### Cause + +- Authentication Error: The credentials in your MetricPipeline output are incorrect. +- Network Unreachable: The backend URL is wrong, a firewall is blocking the connection, or there's a DNS issue preventing the agent or gateway from reaching the backend. +- Backend is Down: The observability backend itself is not running or is unhealthy. + +### Solution + +1. Identify the failing component. + - If the status is `GatewayAllTelemetryDataDropped`, the problem is with the gateway. + - If the status is `AgentAllTelemetryDataDropped`, the problem is with the agent. +2. To check the failing component's logs, call `kubectl logs -n kyma-system {POD_NAME}`: + - For the gateway, check Pod `telemetry-(log|trace|metric)-gateway`. + - For the agent, check Pod `telemetry-(log|metric)-agent`. + Look for errors related to authentication, connectivity, and DNS. +3. Check if the backend is up and reachable. +4. Based on the log messages, fix the output section of your pipeline and re-apply it. + +## Not All Data Arrive at the Backend + +### Symptom + +- The backend is reachable and the connection is properly configured, but some data points are refused. +- In the pipeline status, the `TelemetryFlowHealthy` condition has status **GatewaySomeTelemetryDataDropped** or **AgentSomeTelemetryDataDropped**. + +### Cause + +This status indicates that the telemetry gateway or agent is successfully sending data, but the backend is rejecting some of it. Common reasons are: + +- Rate Limiting: Your backend is rejecting requests because you're sending too much data at once. +- Invalid Data: Your backend is rejecting specific data due to incorrect formatting, invalid labels, or other schema violations. + +### Solution + +1. Check the error logs for the affected Pod by calling `kubectl logs -n kyma-system {POD_NAME}`: + - For **GatewaySomeTelemetryDataDropped**, check Pod `telemetry-(log|trace|metric)-gateway`. + - For **AgentSomeTelemetryDataDropped**, check Pod `telemetry-(log|metric)-agent`. +2. Go to your observability backend and investigate potential causes. +3. If the backend is limiting the rate by refusing data, try the following options: + - Increase the ingestion rate of your backend (for example, by scaling out your SAP Cloud Logging instances). + - Reduce emitted data by re-configuring the pipeline (for example, by disabling certain inputs or applying filters). + - Reduce emitted data in your applications. +4. Otherwise, fix the issues as indicated in the logs. + +## Gateway Throttling + +### Symptom + +In the pipeline status, the `TelemetryFlowHealthy` condition has status **GatewayThrottling**. + +### Cause + +The gateway is receiving data faster than it can process and forward it. + +### Solution + +Manually scale out the capacity by increasing the number of replicas for the affected gateway. For details, see [Telemetry CRD](https://kyma-project.io/#/telemetry-manager/user/01-manager?id=module-configuration). + +### Custom Spans Don’t Arrive at the Backend, but Istio Spans Do + +### Symptom + +You see traces generated by the Istio service mesh, but traces from your own application code (custom spans) are missing. + +### Cause + +The OpenTelemetry (OTel) SDK version used in your application is incompatible with the OTel Collector version. + +### Solution + +1. Check which SDK version you're using for instrumentation. +2. Investigate whether it's compatible with the OTel Collector version. +3. If necessary, upgrade to a supported SDK version. + +### Observability Backend Shows Fewer Traces than Expected + +### Symptom + +The observability backend shows significantly fewer traces than the number of requests your application receives. + +### Cause + +By default, Istio samples only 1% of requests for tracing to minimize performance overhead (see [Configure Istio Tracing](./collecting-traces/istio-support.md)). + +For example, in low-traffic environments (for development or testing) or for low-traffic services, the request volume can be so low that a 1% sample rate may result in capturing zero traces. + +### Solution + +- To see more traces in the trace backend, increase the percentage of requests that are sampled (see [Configure the Sampling Rate](./collecting-traces/istio-support.md#configure-the-sampling-rate)). + +- Alternatively, to trace a single request, force sampling by adding a traceparent HTTP header to your client request. This header contains a sampled flag that instructs the system to capture the trace, bypassing the global sampling rate (see [Trace Context: Sampled Flag](https://www.w3.org/TR/trace-context/#sampled-flag)). + +### MetricPipeline: Failed to Scrape Prometheus Endpoint + +### Symptom + +- Custom metrics don't arrive at the destination. +- The OTel Collector produces log entries saying "Failed to scrape Prometheus endpoint", such as the following example: + + ```bash + 2023-08-29T09:53:07.123Z warn internal/transaction.go:111 Failed to scrape Prometheus endpoint {"kind": "receiver", "name": "prometheus/app-pods", "data_type": "metrics", "scrape_timestamp": 1693302787120, "target_labels": "{__name__=\"up\", instance=\"10.42.0.18:8080\", job=\"app-pods\"}"} + ``` + + +### Cause + +There's a configuration or network issue between the metric agent and your application, such as: + +- The Service that exposes your metrics port doesn't specify the application protocol. +- The workload is not configured to use STRICT mTLS mode, which the metric agent uses by default. +- A deny-all NetworkPolicy in your application's namespace prevents the agent from scraping metrics from annotated workloads. + +### Solution + +- Define the application protocol in the Service port definition by either prefixing the port name with the protocol, or define the appProtocol attribute. +- If the issue is with mTLS, either configure your workload to use STRICT mTLS, or switch to unencrypted scraping by adding the prometheus.io/scheme: "http" annotation to your workload. +- Create a new NetworkPolicy to explicitly allow ingress traffic from the metric agent; such as the following example: + + ```yaml + apiVersion: networking.k8s.io/v1 + kind: NetworkPolicy + metadata: + name: allow-traffic-from-agent + spec: + podSelector: + matchLabels: + app.kubernetes.io/name: "annotated-workload" # + ingress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: kyma-system + podSelector: + matchLabels: + telemetry.kyma-project.io/metric-scrape: "true" + policyTypes: + - Ingress + + ``` + +## LogPipeline: Log Buffer Filling Up + +### Symptom + +In the LogPipeline status, the `TelemetryFlowHealthy` condition has status **AgentBufferFillingUp**. + +### Cause + +The backend ingestion rate is too low compared to the export rate of the log agent, causing data to accumulate in its buffer. + +### Solution + +You can either increase the capacity of your backend or reduce the volume of log data being sent. Try one of the following options: + +- Increase the ingestion rate of your backend (for example, by scaling out your SAP Cloud Logging instances). +- Reduce emitted data by re-configuring the pipeline (for example, by disabling certain inputs or applying namespace filters). +- Reduce the amount of log data generated by your applications. \ No newline at end of file diff --git a/renovate.json b/renovate.json index de024b4757..33747f9763 100644 --- a/renovate.json +++ b/renovate.json @@ -1,86 +1,39 @@ { - "$schema": "https://docs.renovatebot.com/renovate-schema.json", - "extends": [ - "config:recommended", - ":semanticCommits", - ":semanticCommitTypeAll(chore)", - ":semanticCommitScopeDisabled", - "helpers:pinGitHubActionDigests" - ], - "labels": ["area/dependency", "kind/chore"], - "gomod": { - "postUpdateOptions": ["gomodTidy"], - "enabled": true - }, - "kustomize": { - "enabled": false - }, - "dockerfile": { - "enabled": true - }, - "helm-values": { - "enabled": false - }, - "github-actions": { - "enabled": true - }, - "packageRules": [ - { - "matchCategories": ["golang"], - "postUpdateOptions": ["gomodTidy"], - "enabled": true + "$schema": "https://docs.renovatebot.com/renovate-schema.json", + "extends": [ + "config:recommended" + ], + "labels": [ + "area/dependency", + "kind/chore" + ], + "gomod": { + "enabled": false }, - { - "matchDatasources": ["golang-version"], - "rangeStrategy": "bump" + "kustomize": { + "enabled": false }, - { - "matchManagers": ["gomod"], - "matchDepTypes": ["indirect"], - "enabled": false + "dockerfile": { + "enabled": false }, - { - "matchManagers": ["gomod"], - "matchUpdateTypes": ["digest"], - "schedule": "every month", - "enabled": false + "helm-values": { + "enabled": false }, - { - "matchUpdateTypes": ["minor", "patch"], - "matchCurrentVersion": "!/^0/", - "automerge": true + "github-actions": { + "enabled": false }, - { - "matchManagers": ["gomod"], - "groupName": "All opentelemetry collector packages", - "matchSourceUrls": [ - "https://go.opentelemetry.io/collector{/,}**", - "https://github.com/open-telemetry/opentelemetry-collector{/,}**", - "https://github.com/open-telemetry/opentelemetry-collector-contrib{/,}**" - ] - } - ], - "customManagers": [ - { - "customType": "regex", - "fileMatch": ["^\\.env$"], - "matchStrings": [ - "ENV_ISTIO_VERSION=(?\\d+?\\.\\d+?\\.\\d+?)" - ], - "datasourceTemplate": "github-releases", - "versioningTemplate": "semver", - "depNameTemplate": "kyma-project/istio" - }, - { - "customType": "jsonata", - "fileFormat": "yaml", - "fileMatch": ["^external-images\\.yaml$"], - "matchStrings": [ - "images[].{\"depName\": $substringBefore(source, \"@\"), \"currentDigest\": $substringAfter(source, \"@\"), \"currentValue\": tag}" - ], - "datasourceTemplate": "docker", - "autoReplaceStringTemplate": "{\"source\": \"{{{depName}}}@{{{newDigest}}}\", \"tag\": \"{{{newValue}}}\"}" - } - ], - "ignorePaths": ["docs/**"] + "customManagers": [ + { + "customType": "regex", + "fileMatch": [ + "^\\.env$" + ], + "matchStrings": [ + "ENV_ISTIO_VERSION=(?\\d+?\\.\\d+?\\.\\d+?)" + ], + "datasourceTemplate": "github-releases", + "versioningTemplate": "semver", + "depNameTemplate": "kyma-project/istio" + } + ] }