Skip to content

Commit

Permalink
Add interval processor, spanlogs connector, and spanmetrics connector (
Browse files Browse the repository at this point in the history
…#1119)

* Add interval processor

Signed-off-by: Pete Wall <[email protected]>

* Start adding spanmetrics and spanlogs

Signed-off-by: Pete Wall <[email protected]>

* Rebuild the AppO11y pipeline so it's defined in a pipeline.tpl file. Much easier to modify and understand.

Signed-off-by: Pete Wall <[email protected]>

* Add spanlogs and spanmetrics connectors

Signed-off-by: Pete Wall <[email protected]>

* Add namespace field to spanmetrics

Signed-off-by: Pete Wall <[email protected]>

* Update charts/k8s-monitoring/charts/feature-application-observability/values.yaml

Co-authored-by: Patrick Easters <[email protected]>

* Update appo11y readme

Signed-off-by: Pete Wall <[email protected]>

* Update the description for the grafanaCloudMetrics connector to highlight its potential cause for billing.

Signed-off-by: Pete Wall <[email protected]>

---------

Signed-off-by: Pete Wall <[email protected]>
Co-authored-by: Patrick Easters <[email protected]>
  • Loading branch information
petewall and patrickeasters authored Jan 24, 2025
1 parent be23092 commit 538fe77
Show file tree
Hide file tree
Showing 39 changed files with 1,468 additions and 274 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,41 @@ Be sure perform actual integration testing in a live environment in the main [k8

## Values

### Connectors: Grafana Cloud Host Info

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| connectors.grafanaCloudMetrics.enabled | bool | `true` | Generate host info metrics from telemetry data. These metrics are required for using Application Observability in Grafana Cloud. Note: Enabling this may incur additional costs. See https://grafana.com/docs/grafana-cloud/monitor-applications/application-observability/pricing/ |

### Connectors: Span Logs

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| connectors.spanLogs.enabled | bool | `false` | Use a span logs connector which creates logs from spans. |
| connectors.spanLogs.labels | list | `[]` | A list of keys that will be logged as labels. |
| connectors.spanLogs.process | bool | `false` | Log one line for every process. |
| connectors.spanLogs.processAttributes | list | `[]` | Additional process attributes to log. |
| connectors.spanLogs.roots | bool | `false` | Log one line for every root span of a trace. |
| connectors.spanLogs.spanAttributes | list | `[]` | Additional span attributes to log. |
| connectors.spanLogs.spans | bool | `false` | Create a log line for each span. This can lead to a large number of logs. |

### Connectors: Span Metrics

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| connectors.spanMetrics.dimensions | list | `[]` | Define dimensions to be added. Some are set internally by default: [service.name, span.name, span.kind, status.code] Example: - name: "http.status_code" - name: "http.method" default: "GET" |
| connectors.spanMetrics.dimensionsCacheSize | int | `1000` | How many dimensions to cache |
| connectors.spanMetrics.enabled | bool | `false` | Use a span metrics connector which creates metrics from spans. |
| connectors.spanMetrics.events.enabled | bool | `false` | Capture events metrics, which track span events. |
| connectors.spanMetrics.exemplars.enabled | bool | `false` | Attach exemplars to histograms. |
| connectors.spanMetrics.exemplars.maxPerDataPoint | number | `nil` | Limits the number of exemplars that can be added to a unique dimension set. |
| connectors.spanMetrics.histogram.enabled | bool | `true` | Capture histogram metrics, derived from spans’ durations. |
| connectors.spanMetrics.histogram.explicit.buckets | list | `["2ms","4ms","6ms","8ms","10ms","50ms","100ms","200ms","400ms","800ms","1s","1400ms","2s","5s","10s","15s"]` | The histogram buckets to use. |
| connectors.spanMetrics.histogram.exponential.maxSize | int | `160` | Maximum number of buckets per positive or negative number range. |
| connectors.spanMetrics.histogram.type | string | `"explicit"` | Type of histograms to create. Must be either "explicit" or "exponential". |
| connectors.spanMetrics.histogram.unit | string | `"ms"` | The histogram unit. |
| connectors.spanMetrics.namespace | string | `"traces.span.metrics"` | The Metric namespace. |

### General settings

| Key | Type | Default | Description |
Expand All @@ -55,11 +90,14 @@ Be sure perform actual integration testing in a live environment in the main [k8
| processors.batch.size | int | `16384` | What batch size to use, in bytes |
| processors.batch.timeout | string | `"2s"` | How long before sending (Processors) |

### Processors: Grafana Cloud Host Info
### Processors: Interval

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| processors.grafanaCloudMetrics.enabled | bool | `true` | Generate host info metrics from telemetry data, used in Application Observability in Grafana Cloud. |
| processors.interval.enabled | bool | `false` | Utilize an interval processor to aggregate metrics and periodically forward the latest values to the next component in the pipeline. |
| processors.interval.interval | string | `"60s"` | The interval at which to emit aggregated metrics. |
| processors.interval.passthrough.gauge | bool | `false` | Determines whether gauge metrics should be passed through as they are or aggregated. |
| processors.interval.passthrough.summary | bool | `false` | Determines whether summary metrics should be passed through as they are or aggregated. |

### Processors: K8s Attributes

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"properties": {
"connectors": {
"properties": {
"spanMetrics": {
"properties": {
"histogram": {"properties": {"type": {"enum": ["explicit", "exponential"]}}}
}
}
}
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ otelcol.connector.host_info "{{ .name | default "default" }}" {
host_identifiers = [ "k8s.node.name" ]
output {
{{- if and .metricsOutput .Values.metrics.enabled }}
metrics = {{ .metricsOutput }}
{{- if and .metrics .Values.metrics.enabled }}
metrics = {{ .metrics }}
{{- end }}
}
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
{{/* Inputs: Values (values) metricsOutput, name */}}
{{/* https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.connector.spanlogs/ */}}
{{- define "feature.applicationObservability.connector.spanlogs.alloy.target" }}otelcol.connector.spanlogs.{{ .name | default "default" }}.input{{- end }}
{{- define "feature.applicationObservability.connector.spanlogs.alloy" }}
otelcol.connector.spanlogs "{{ .name | default "default" }}" {
{{- if .Values.connectors.spanLogs.spans }}
spans = true
{{- end }}
{{- if .Values.connectors.spanLogs.spansAttributes }}
spans_attributes = {{ .Values.connectors.spanLogs.spansAttributes | toJson }}
{{- end }}
{{- if .Values.connectors.spanLogs.roots }}
roots = true
{{- end }}
{{- if .Values.connectors.spanLogs.process }}
process = true
{{- end }}
{{- if .Values.connectors.spanLogs.processAttributes }}
process_attributes = {{ .Values.connectors.spanLogs.processAttributes | toJson }}
{{- end }}
{{- if .Values.connectors.spanLogs.labels }}
labels = {{ .Values.connectors.spanLogs.labels | toJson }}
{{- end }}

output {
{{- if and .logs .Values.logs.enabled }}
logs = {{ .logs }}
{{- end }}
}
}
{{- end }}
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
{{/* Inputs: Values (values) metricsOutput, name */}}
{{/* https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.connector.spanmetrics/ */}}
{{- define "feature.applicationObservability.connector.spanmetrics.alloy.target" }}otelcol.connector.spanmetrics.{{ .name | default "default" }}.input{{- end }}
{{- define "feature.applicationObservability.connector.spanmetrics.alloy" }}
otelcol.connector.spanmetrics "{{ .name | default "default" }}" {
{{- range $dimension := .Values.connectors.spanMetrics.dimensions }}
dimension {
name = {{ $dimension.name | quote }}
{{- if $dimension.default }}
default = {{ $dimension.default | quote }}
{{- end }}
}
{{- end }}
dimensions_cache_size = {{ .Values.connectors.spanMetrics.dimensionsCacheSize }}
namespace = {{ .Values.connectors.spanMetrics.namespace | quote }}
{{- if .Values.connectors.spanMetrics.events.enabled }}
events {
enabled = true
}
{{- end }}
{{ if .Values.connectors.spanMetrics.exemplars.enabled }}
exemplars {
enabled = true
{{- if .Values.connectors.spanMetrics.exemplars.maxPerDataPoint }}
max_per_data_point = {{ .Values.connectors.spanMetrics.exemplars.maxPerDataPoint }}
{{- end }}
}
{{- end }}
{{- if .Values.connectors.spanMetrics.histogram.enabled }}
histogram {
disable = false
unit = {{ .Values.connectors.spanMetrics.histogram.unit | quote }}
{{- if eq .Values.connectors.spanMetrics.histogram.type "explicit" }}
explicit {
buckets = {{ .Values.connectors.spanMetrics.histogram.explicit.buckets | toJson }}
}
{{- else if eq .Values.connectors.spanMetrics.histogram.type "exponential" }}
exponential {
max_size = {{ .Values.connectors.spanMetrics.histogram.exponential.maxSize }}
}
{{- end }}
}
{{- end }}

output {
{{- if and .metrics .Values.metrics.enabled }}
metrics = {{ .metrics }}
{{- end }}
}
}
{{- end }}
Original file line number Diff line number Diff line change
@@ -1,13 +1,4 @@
{{- define "feature.applicationObservability.module" }}
{{- $metricsNext := "" }}
{{- $logsNext := "" }}
{{- $resourceDetection := include "feature.applicationObservability.processor.resourcedetection.alloy.target" dict }}
{{- $k8sAttributes := include "feature.applicationObservability.processor.k8sattributes.alloy.target" dict }}
{{- $grafanaCloudMetrics := include "feature.applicationObservability.connector.host_info.alloy.target" dict }}
{{- $transform := include "feature.applicationObservability.processor.transform.alloy.target" dict }}
{{- $filter := include "feature.applicationObservability.processor.filter.alloy.target" dict }}
{{- $batch := include "feature.applicationObservability.processor.batch.alloy.target" dict }}
{{- $memoryLimiter := include "feature.applicationObservability.processor.memory_limiter.alloy.target" dict }}
declare "application_observability" {
argument "metrics_destinations" {
comment = "Must be a list of metrics destinations where collected metrics should be forwarded to"
Expand All @@ -20,66 +11,24 @@ declare "application_observability" {
argument "traces_destinations" {
comment = "Must be a list of trace destinations where collected trace should be forwarded to"
}

// Receivers --> Resource Detection Processor
{{- $next := printf "[%s]" $resourceDetection }}
{{- include "feature.applicationObservability.receiver.otlp.alloy" (dict "Values" $.Values "metricsOutput" $next "logsOutput" $next "tracesOutput" $next ) | indent 2 }}
{{- include "feature.applicationObservability.receiver.jaeger.alloy" (dict "Values" $.Values "tracesOutput" $next ) | indent 2 }}
{{- include "feature.applicationObservability.receiver.zipkin.alloy" (dict "Values" $.Values "tracesOutput" $next ) | indent 2 }}

// Resource Detection Processor --> K8s Attribute Processor
{{- $next = printf "[%s]" $k8sAttributes }}
{{- include "feature.applicationObservability.processor.resourcedetection.alloy" (dict "Values" $.Values "metricsOutput" $next "logsOutput" $next "tracesOutput" $next ) | indent 2 }}

// K8s Attribute Processor --> Transform Processor
{{- $tracesNext := list $transform }}
{{- $next = printf "[%s]" $transform }}
{{- if .Values.processors.grafanaCloudMetrics.enabled }}
// Resource Detection Processor Traces --> Host Info Connector
{{- $tracesNext = append $tracesNext $grafanaCloudMetrics }}
{{- end -}}
{{- $tracesNext = printf "[%s]" ($tracesNext | join ", ")}}
{{- include "feature.applicationObservability.processor.k8sattributes.alloy" (dict "Values" $.Values "metricsOutput" $next "logsOutput" $next "tracesOutput" $tracesNext ) | indent 2 }}

{{- if .Values.processors.grafanaCloudMetrics.enabled }}
// Host Info Connector --> Batch Processor
{{- $next = printf "[%s]" $batch }}
{{- include "feature.applicationObservability.connector.host_info.alloy" (dict "Values" $.Values "metricsOutput" $next ) | indent 2 }}
{{- end }}

{{ if eq (include "feature.applicationObservability.processor.filter.enabled" .) "true" }}
// Transform Processor --> Filter Processor
{{- $next = printf "[%s]" $filter }}
{{- else }}
// Transform Processor --> Batch Processor
{{- $next = printf "[%s]" $batch }}
{{- end }}
{{- include "feature.applicationObservability.processor.transform.alloy" (dict "Values" $.Values "metricsOutput" $next "logsOutput" $next "tracesOutput" $next ) | indent 2 }}
{{ if eq (include "feature.applicationObservability.processor.filter.enabled" .) "true" }}
// Filter Processor --> Batch Processor
{{- $next = printf "[%s]" $batch }}
{{- include "feature.applicationObservability.processor.filter.alloy" (dict "Values" $.Values "metricsOutput" $next "logsOutput" $next "tracesOutput" $next ) | indent 2 }}
{{- end }}

{{- if .Values.processors.memoryLimiter.enabled }}
// Batch Processor --> Memory Limiter
{{- $metricsNext = printf "[%s]" $memoryLimiter }}
{{- $logsNext = printf "[%s]" $memoryLimiter }}
{{- $tracesNext = printf "[%s]" $memoryLimiter }}
{{- else }}
// Batch Processor --> Destinations
{{- $metricsNext = "argument.metrics_destinations.value" }}
{{- $logsNext = "argument.logs_destinations.value" }}
{{- $tracesNext = "argument.traces_destinations.value" }}
{{- end }}
{{- include "feature.applicationObservability.processor.batch.alloy" (dict "Values" $.Values "metricsOutput" $metricsNext "logsOutput" $logsNext "tracesOutput" $tracesNext ) | indent 2 }}

{{- if .Values.processors.memoryLimiter.enabled }}
// Memory Limiter --> Destinations
{{- $metricsNext = "argument.metrics_destinations.value" }}
{{- $logsNext = "argument.logs_destinations.value" }}
{{- $tracesNext = "argument.traces_destinations.value" }}
{{- include "feature.applicationObservability.processor.memory_limiter.alloy" (dict "Values" $.Values "metricsOutput" $metricsNext "logsOutput" $logsNext "tracesOutput" $tracesNext ) | indent 2 }}
{{- $pipeline := include "feature.applicationObservability.pipeline" . | fromYamlArray }}
{{- range $component := $pipeline }}
{{- $args := (dict "Values" $.Values "name" $component.name) }}

{{- range $dataType := (list "metrics" "logs" "traces")}}
{{- if kindIs "string" (index $component.targets $dataType) }}
{{- $args = merge $args (dict $dataType (index $component.targets $dataType)) }}
{{- else if kindIs "slice" (index $component.targets $dataType) }}
{{- $targets := list }}
{{- range $target := (index $component.targets $dataType) }}
{{- $targets = append $targets (include (printf "feature.applicationObservability.%s.alloy.target" $target.component) $target) }}
{{- end }}
{{- $args = merge $args (dict $dataType (printf "[%s]" (join ", " $targets))) }}
{{- end }}
{{- end }}

// {{ $component.description }}
{{- include (printf "feature.applicationObservability.%s.alloy" $component.component) $args | indent 2 }}
{{- end }}
}
{{- end }}
Expand Down
Loading

0 comments on commit 538fe77

Please sign in to comment.