feat: Collect the containerdebug logs (#912)

* feat: Collect the containerdebug logs

* chore: Upgrade serde-saphyr

* fix: Do not hard-code the environment variable name "FILE_LOG_DIRECTORY"

* fix: Create the log directories of the info-fetcher sidecars

* chore: Update changelog

* fix: Do not create the log directories of the info-fetcher sidecars

* chore: Sort the FILE_LOG_* constants like in the other operators

Author: siegfriedweber

CHANGELOG.md

@@ -22,6 +22,8 @@ All notable changes to this project will be documented in this file.
 - A `PodDisruptionBudget` is now written out for the `servers` role when it runs as a `Deployment`,
   with `maxUnavailable: 1`. Configurable via `spec.servers.roleConfig.podDisruptionBudget` ([#873]).
 - Add `/ready` endpoint to the operator Deployment, which reports the CRD installation status ([#894]).
+- The logs of the `containerdebug` process are now collected by the Vector agent and forwarded to
+  the Vector aggregator ([#912]).
 
 ### Changed
 
@@ -101,14 +103,21 @@ All notable changes to this project will be documented in this file.
   containing a line break, is rejected with `400` ([#909]).
 - The user-info-fetcher Keycloak backend no longer drops a configured `rootPath` from the URLs it
   requests, so a Keycloak served below a sub-path, like `/auth`, can now be reached ([#909]).
+- The `containerdebug` process writes its log file again. The operator set
+  `CONTAINERDEBUG_LOG_DIRECTORY` on the OPA container, which containerdebug stopped reading in its
+  version 0.2.0 when it moved to `stackable-telemetry`; the directory is now set via
+  `FILE_LOG_DIRECTORY`, with an hourly rotation that keeps at most two files ([#912]).
+- The `tracing-rs` transform of the Vector agent no longer discards log events. A message which is
+  not a string is kept among the other fields, a `fields` object which is not an object falls back
+  to the raw line, and a missing timestamp is recorded as an error ([#912]).
 
-[#873]: https://github.com/stackabletech/opa-operator/pull/873
 [#852]: https://github.com/stackabletech/opa-operator/pull/852
 [#861]: https://github.com/stackabletech/opa-operator/pull/861
 [#863]: https://github.com/stackabletech/opa-operator/pull/863
 [#867]: https://github.com/stackabletech/opa-operator/pull/867
 [#871]: https://github.com/stackabletech/opa-operator/pull/871
 [#872]: https://github.com/stackabletech/opa-operator/pull/872
+[#873]: https://github.com/stackabletech/opa-operator/pull/873
 [#880]: https://github.com/stackabletech/opa-operator/pull/880
 [#882]: https://github.com/stackabletech/opa-operator/pull/882
 [#886]: https://github.com/stackabletech/opa-operator/pull/886
@@ -123,6 +132,7 @@ All notable changes to this project will be documented in this file.
 [#906]: https://github.com/stackabletech/opa-operator/pull/906
 [#908]: https://github.com/stackabletech/opa-operator/pull/908
 [#909]: https://github.com/stackabletech/opa-operator/pull/909
+[#912]: https://github.com/stackabletech/opa-operator/pull/912
 
 ## [26.7.0] - 2026-07-21
 

Cargo.nix

@@ -56,5 +56,13 @@ spec:
 The decision logs are still filtered by the log level set for the console and file appenders.
 Therefore, the configuration above would result in decision logs being present on file but not on console.
 
+NOTE: The logs of the xref:concepts:observability/containerdebug.adoc[`containerdebug` process] are collected as well.
+They are labelled `container: containerdebug`, although no such Kubernetes container exists.
+The process runs in the background of the `opa` container.
+The label comes from `/stackable/log/containerdebug`, the directory which the operator sets with the `FILE_LOG_DIRECTORY` environment variable and from which the Vector agent derives the container name.
+Overriding `FILE_LOG_DIRECTORY` with `envOverrides` therefore changes how these logs are aggregated.
+The agent only collects files one level below `/stackable/log`, so a sibling directory such as `/stackable/log/opa` relabels the events instead of dropping them, while any deeper or shallower path stops the aggregation.
+A path outside `/stackable/log` also leaves the size-capped log volume.
+
 Further information on how to configure logging, can be found in
 xref:concepts:logging.adoc[].

docs/modules/opa/pages/usage-guide/logging.adoc

@@ -32,4 +32,5 @@ tracing.workspace = true
 built.workspace = true
 
 [dev-dependencies]
+stackable-operator = { workspace = true, features = ["test-support"] }
 yaml_serde.workspace = true

rust/operator-binary/Cargo.toml

@@ -107,6 +107,285 @@ tests:
             source: |
               assert!(exists(.errors), "unparsable timestamp should be recorded as an error")
 
+  - name: Containerdebug tracing-rs log
+    inputs:
+      - type: log
+        insert_at: processed_files_tracing_rs
+        log_fields:
+          file: /stackable/log/containerdebug/containerdebug.2026-11-02-14.tracing-rs.json
+          message: '{"timestamp":"2026-11-02T14:49:28.028922Z","level":"INFO","fields":{"message":"found disk","disk.name":"overlay"},"target":"containerdebug::system_information::disk"}'
+          pod: test-opa-server-default-0
+          source_type: file
+          timestamp: 2026-11-02T14:49:29.473487331Z
+    outputs:
+      - extract_from: extended_logs
+        expected_event_count: 1
+        conditions:
+          - type: vrl
+            source: |
+              expected_log_event = {
+                "cluster": "opa",
+                "container": "containerdebug",
+                "file": "containerdebug.2026-11-02-14.tracing-rs.json",
+                "level": "INFO",
+                "logger": "containerdebug::system_information::disk",
+                "message": "found disk\n\ndisk.name=overlay",
+                "timestamp": t'2026-11-02T14:49:28.028922Z',
+                "namespace": "default",
+                "pod": "test-opa-server-default-0",
+                "role": "server",
+                "roleGroup": "default"
+              }
+
+              assert_eq!(expected_log_event, .)
+  - name: Containerdebug tracing-rs log which is unparsable
+    inputs:
+      - type: log
+        insert_at: processed_files_tracing_rs
+        log_fields:
+          file: /stackable/log/containerdebug/containerdebug.2026-11-02-14.tracing-rs.json
+          message: '{ unparsable'
+          pod: test-opa-server-default-0
+          source_type: file
+          timestamp: 2026-11-02T14:49:29.473487331Z
+    outputs:
+      - extract_from: extended_logs
+        expected_event_count: 1
+        conditions:
+          - type: vrl
+            source: |
+              expected_log_event = {
+                "cluster": "opa",
+                "container": "containerdebug",
+                "errors": [
+                  .errors[0]
+                ],
+                "file": "containerdebug.2026-11-02-14.tracing-rs.json",
+                "level": "INFO",
+                "logger": "",
+                "message": "{ unparsable",
+                "timestamp": .timestamp,
+                "namespace": "default",
+                "pod": "test-opa-server-default-0",
+                "role": "server",
+                "roleGroup": "default"
+              }
+
+              assert_eq!(expected_log_event, .)
+
+              # The remainder of the message contains character offsets into
+              # the VRL source, so only the stable prefix is asserted.
+              assert!(starts_with!(.errors[0], "JSON not parsable: "))
+  - name: Containerdebug tracing-rs log which is not a JSON object
+    inputs:
+      - type: log
+        insert_at: processed_files_tracing_rs
+        log_fields:
+          file: /stackable/log/containerdebug/containerdebug.2026-11-02-14.tracing-rs.json
+          message: 'false'
+          pod: test-opa-server-default-0
+          source_type: file
+          timestamp: 2026-11-02T14:49:29.473487331Z
+    outputs:
+      - extract_from: extended_logs
+        expected_event_count: 1
+        conditions:
+          - type: vrl
+            source: |
+              expected_log_event = {
+                "cluster": "opa",
+                "container": "containerdebug",
+                "errors": [
+                  "Parsed event is not a JSON object."
+                ],
+                "file": "containerdebug.2026-11-02-14.tracing-rs.json",
+                "level": "INFO",
+                "logger": "",
+                "message": "false",
+                "timestamp": .timestamp,
+                "namespace": "default",
+                "pod": "test-opa-server-default-0",
+                "role": "server",
+                "roleGroup": "default"
+              }
+
+              assert_eq!(expected_log_event, .)
+  - name: Containerdebug tracing-rs log with invalid fields
+    inputs:
+      - type: log
+        insert_at: processed_files_tracing_rs
+        log_fields:
+          file: /stackable/log/containerdebug/containerdebug.2026-11-02-14.tracing-rs.json
+          message: '{"timestamp":"unparsable","level":"INVALID","fields":"none","target":null}'
+          pod: test-opa-server-default-0
+          source_type: file
+          timestamp: 2026-11-02T14:49:29.473487331Z
+    outputs:
+      - extract_from: extended_logs
+        expected_event_count: 1
+        conditions:
+          - type: vrl
+            source: |
+              expected_log_event = {
+                "cluster": "opa",
+                "container": "containerdebug",
+                "errors": [
+                  .errors[0],
+                  "Logger/target not found.",
+                  "Level \"INVALID\" unknown, using \"INFO\" instead.",
+                  "Fields are not an object."
+                ],
+                "file": "containerdebug.2026-11-02-14.tracing-rs.json",
+                "level": "INFO",
+                "logger": "",
+                "message": "{\"timestamp\":\"unparsable\",\"level\":\"INVALID\",\"fields\":\"none\",\"target\":null}",
+                "timestamp": .timestamp,
+                "namespace": "default",
+                "pod": "test-opa-server-default-0",
+                "role": "server",
+                "roleGroup": "default"
+              }
+
+              assert_eq!(expected_log_event, .)
+
+              # The remainder of the message contains character offsets into
+              # the VRL source, so only the stable prefix is asserted.
+              assert!(starts_with!(.errors[0], "Timestamp not parsable, using current time instead: "))
+  - name: Containerdebug tracing-rs log without a timestamp
+    inputs:
+      - type: log
+        insert_at: processed_files_tracing_rs
+        log_fields:
+          file: /stackable/log/containerdebug/containerdebug.2026-11-02-14.tracing-rs.json
+          message: '{"level":"INFO","fields":{"message":"found disk"},"target":"containerdebug::system_information::disk"}'
+          pod: test-opa-server-default-0
+          source_type: file
+          timestamp: 2026-11-02T14:49:29.473487331Z
+    outputs:
+      - extract_from: extended_logs
+        expected_event_count: 1
+        conditions:
+          - type: vrl
+            source: |
+              expected_log_event = {
+                "cluster": "opa",
+                "container": "containerdebug",
+                "errors": [
+                  "Timestamp not found, using current time instead."
+                ],
+                "file": "containerdebug.2026-11-02-14.tracing-rs.json",
+                "level": "INFO",
+                "logger": "containerdebug::system_information::disk",
+                "message": "found disk",
+                "timestamp": .timestamp,
+                "namespace": "default",
+                "pod": "test-opa-server-default-0",
+                "role": "server",
+                "roleGroup": "default"
+              }
+
+              assert_eq!(expected_log_event, .)
+
+              # The current time is used and not the timestamp which the file
+              # source attached to the event. The latter reaches the transform
+              # as a string, whereas now() yields a timestamp.
+              assert!(is_timestamp(.timestamp))
+  - name: Containerdebug tracing-rs log without a message but with other fields
+    inputs:
+      - type: log
+        insert_at: processed_files_tracing_rs
+        log_fields:
+          file: /stackable/log/containerdebug/containerdebug.2026-11-02-14.tracing-rs.json
+          message: '{"timestamp":"2026-11-02T14:49:28.028922Z","level":"INFO","fields":{"disk.name":"overlay"},"target":"containerdebug::system_information::disk"}'
+          pod: test-opa-server-default-0
+          source_type: file
+          timestamp: 2026-11-02T14:49:29.473487331Z
+    outputs:
+      - extract_from: extended_logs
+        expected_event_count: 1
+        conditions:
+          - type: vrl
+            source: |
+              expected_log_event = {
+                "cluster": "opa",
+                "container": "containerdebug",
+                "file": "containerdebug.2026-11-02-14.tracing-rs.json",
+                "level": "INFO",
+                "logger": "containerdebug::system_information::disk",
+                "message": "disk.name=overlay",
+                "timestamp": t'2026-11-02T14:49:28.028922Z',
+                "namespace": "default",
+                "pod": "test-opa-server-default-0",
+                "role": "server",
+                "roleGroup": "default"
+              }
+
+              assert_eq!(expected_log_event, .)
+  - name: Containerdebug tracing-rs log with a message which is not a string
+    inputs:
+      - type: log
+        insert_at: processed_files_tracing_rs
+        log_fields:
+          file: /stackable/log/containerdebug/containerdebug.2026-11-02-14.tracing-rs.json
+          message: '{"timestamp":"2026-11-02T14:49:28.028922Z","level":"INFO","fields":{"message":42,"disk.name":"overlay"},"target":"containerdebug::system_information::disk"}'
+          pod: test-opa-server-default-0
+          source_type: file
+          timestamp: 2026-11-02T14:49:29.473487331Z
+    outputs:
+      - extract_from: extended_logs
+        expected_event_count: 1
+        conditions:
+          - type: vrl
+            source: |
+              expected_log_event = {
+                "cluster": "opa",
+                "container": "containerdebug",
+                "file": "containerdebug.2026-11-02-14.tracing-rs.json",
+                "level": "INFO",
+                "logger": "containerdebug::system_information::disk",
+                "message": "disk.name=overlay\nmessage=42",
+                "timestamp": t'2026-11-02T14:49:28.028922Z',
+                "namespace": "default",
+                "pod": "test-opa-server-default-0",
+                "role": "server",
+                "roleGroup": "default"
+              }
+
+              assert_eq!(expected_log_event, .)
+  - name: Containerdebug tracing-rs log without any fields
+    inputs:
+      - type: log
+        insert_at: processed_files_tracing_rs
+        log_fields:
+          file: /stackable/log/containerdebug/containerdebug.2026-11-02-14.tracing-rs.json
+          message: '{"timestamp":"2026-11-02T14:49:28.028922Z","level":"INFO","fields":{},"target":"containerdebug::system_information::disk"}'
+          pod: test-opa-server-default-0
+          source_type: file
+          timestamp: 2026-11-02T14:49:29.473487331Z
+    outputs:
+      - extract_from: extended_logs
+        expected_event_count: 1
+        conditions:
+          - type: vrl
+            source: |
+              expected_log_event = {
+                "cluster": "opa",
+                "container": "containerdebug",
+                "errors": [
+                  "Message not found."
+                ],
+                "file": "containerdebug.2026-11-02-14.tracing-rs.json",
+                "level": "INFO",
+                "logger": "containerdebug::system_information::disk",
+                "message": "",
+                "timestamp": t'2026-11-02T14:49:28.028922Z',
+                "namespace": "default",
+                "pod": "test-opa-server-default-0",
+                "role": "server",
+                "roleGroup": "default"
+              }
+
+              assert_eq!(expected_log_event, .)
   - name: Prepare stdout log
     inputs:
       - type: log