Skip to content

Splunk specific file based configuration fields #355

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Kielek merged 16 commits into from
Dec 11, 2025
Merged

Splunk specific file based configuration fields #355

Changes from 2 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
d437cda
add file based config draft
seemk Nov 11, 2025
1123de3
casing, move under distribution key
seemk Nov 25, 2025
f80d5aa
structure changes
seemk Nov 27, 2025
1bff279
js: move runtime metrics to the general block
seemk Nov 28, 2025
a6670b0
update to remove language specific entries, add splunk_prefix
seemk Dec 8, 2025
ea159c0
Merge branch 'main' into file-based-config
seemk Dec 8, 2025
6bb05d8
lint
seemk Dec 8, 2025
69a19c8
lint
seemk Dec 8, 2025
fda8404
nesting, js fixes
seemk Dec 8, 2025
6a0e2c8
remove splunk.general
seemk Dec 9, 2025
86e1dc9
add experimental status
seemk Dec 9, 2025
1251ba2
Merge branch 'main' into file-based-config
seemk Dec 10, 2025
0b64f05
remove nodejs example
seemk Dec 10, 2025
45a27be
change otlp_http -> otlp_log_http
Kielek Dec 11, 2025
0310b22
Add env-var for access token to the example
Kielek Dec 11, 2025
5af67f3
Update specification/configuration.md
Kielek Dec 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 69 additions & 8 deletions specification/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -239,8 +239,8 @@
a deprecation warning and suggest an alternate method. For example:
```txt
jaeger-thrift-splunk trace exporter is deprecated and may be removed in a future major release. Use the default
OTLP exporter instead, or set the SPLUNK_REALM and SPLUNK_ACCESS_TOKEN environment variables to send
jaeger-thrift-splunk trace exporter is deprecated and may be removed in a future major release. Use the default
OTLP exporter instead, or set the SPLUNK_REALM and SPLUNK_ACCESS_TOKEN environment variables to send
telemetry directly to Splunk Observability Cloud.
```
Expand All @@ -250,12 +250,73 @@
In addition to environment variables, other ways of defining configuration also exist:
- [Java System
Properties](https://docs.oracle.com/javase/tutorial/essential/environment/sysprop.html):
These properties MUST match the environment variables converting to lower
case and replacing underscores with hyphens or periods. For example:
system property `splunk.trace-response-header.enabled` is equivalent to environment
variable `SPLUNK_TRACE_RESPONSE_HEADER_ENABLED`.
#### File based configuration

Check failure on line 253 in specification/configuration.md

View workflow job for this annotation

GitHub Actions / validate-documentation

Heading levels should only increment by one level at a time 

specification/configuration.md:253 MD001/heading-increment Heading levels should only increment by one level at a time [Expected: h3; Actual: h4] https://github.com/DavidAnson/markdownlint/blob/v0.39.0/doc/md001.md
OpenTelemetry's [declarative configuration](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/configuration/README.md#declarative-configuration) SHOULD be supported via [`OTEL_EXPERIMENTAL_CONFIG_FILE`](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/configuration/sdk.md#via-otel_experimental_config_file) environment variable.

Check failure on line 255 in specification/configuration.md

View workflow job for this annotation

GitHub Actions / validate-documentation

Line length 

specification/configuration.md:255:81 MD013/line-length Line length [Expected: 80; Actual: 402] https://github.com/DavidAnson/markdownlint/blob/v0.39.0/doc/md013.md
In addition, Splunk specific configuration MUST have their own root-level configuration block named `splunk`.

Check failure on line 257 in specification/configuration.md

View workflow job for this annotation

GitHub Actions / validate-documentation

Line length 

specification/configuration.md:257:81 MD013/line-length Line length [Expected: 80; Actual: 109] https://github.com/DavidAnson/markdownlint/blob/v0.39.0/doc/md013.md
The following is an example covering the common configuration values:
```yaml
distribution:
splunk:
tracing:
disabled: false
response_header_enabled: true # SPLUNK_TRACE_RESPONSE_HEADER_ENABLED
profiling:
disabled: false # SPLUNK_PROFILER_ENABLED
endpoint: "" # SPLUNK_PROFILER_LOGS_ENDPOINT
memory_profiling_enabled: true # SPLUNK_PROFILER_MEMORY_ENABLED
callgraphs:
disabled: false # SPLUNK_SNAPSHOT_PROFILER_ENABLED
sampling_interval: 10 # SPLUNK_SNAPSHOT_SAMPLING_INTERVAL
selection_probability: 0.01 # SPLUNK_SNAPSHOT_SAMPLING_INTERVAL
Copy link
Contributor

@pellared pellared Nov 26, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be something like

Suggested change
profiling:
disabled: false # SPLUNK_PROFILER_ENABLED
endpoint: "" # SPLUNK_PROFILER_LOGS_ENDPOINT
memory_profiling_enabled: true # SPLUNK_PROFILER_MEMORY_ENABLED
callgraphs:
disabled: false # SPLUNK_SNAPSHOT_PROFILER_ENABLED
sampling_interval: 10 # SPLUNK_SNAPSHOT_SAMPLING_INTERVAL
selection_probability: 0.01 # SPLUNK_SNAPSHOT_SAMPLING_INTERVAL
profiling:
processors:
- batch:
schedule_delay: 5000 # SPLUNK_PROFILER_EXPORT_INTERVAL
export_timeout: 30000 # SPLUNK_PROFILER_EXPORT_TIMEOUT
exporter:
otlp_log_http:
endpoint: http://localhost:4318/v1/logs # SPLUNK_PROFILER_LOGS_ENDPOINT
memory_allocation_sampling: # SPLUNK_PROFILER_MEMORY_ENABLED
sampling_interval / sampling_rate: # SPLUNK_PROFILER_MEMORY_EVENT_RATE / SPLUNK_PROFILER_MAX_MEMORY_SAMPLES ?
callstack_sampling: # SPLUNK_PROFILER_ENABLED
sampling_interval: 10 # SPLUNK_PROFILER_CALL_STACK_INTERVAL
snapshoting: # SPLUNK_SNAPSHOT_PROFILER_ENABLED
callstack_sampling_interval: 10 # SPLUNK_SNAPSHOT_SAMPLING_INTERVAL
trace_selection_probability: 0.01 # SPLUNK_SNAPSHOT_SELECTION_PROBABILTY

Discussed with @lachmatt

Copy link
Contributor Author

@seemk seemk Nov 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This (memory_allocation_sampling, callstack_sampling, snapshotting) is completely different wording than we have in our documentation. Why would you not keep parity here?

Copy link
Contributor

@pellared pellared Nov 26, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Together with @lachmatt we find the current wording (from the documentation) confusing and misleading. We propose changing the names to better represent the behavior of the profilers.

Copy link
Contributor

@pellared pellared Nov 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that the proposal misses some of the configuration supported by Java.

Copy link
Contributor Author

@seemk seemk Nov 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Giving it some thought again after our discussion, I think it doesn't make sense because of 2 reasons:

  • Distros might not use the OpenTelemetry pipeline for profiling data exports. Node.js doesn't - we recently had a custom exporter (without using the OTel logs processors), but switched to an OTel logs exporter now. We are not using the logs processing pipeline and don't think we ever will.
  • While the naming might be more correct, differentiating this from our current customer terminology and documentation is bound to sow confusion.

An unrelated to your comment, but a related issue is that while going over the configuration, I noticed that you can't actually set the endpoint for callgraphs explicitly with file based config. The documentation we have refers to OTEL_EXPORTER_OTLP_ENDPOINT. However this does not make sense much with file based config when the pipelines are in their own sections.

We either have to create a new configuration option for a callgraphs endpoint or we use the one from profiler.
Now both the CPU profiler and callgraphs can be active irrespective of each other. When (as previously proposed) we don't have an enabled/disabled flag under profiling we can end up in a case where we can't set the endpoint for callgraphs, e.g.:

distribution:
  splunk:
    #profiling:
    #  endpoint: "collector:4318/v1/logs"
    callgraphs:
      trace_selection_probability: 0.05
      # Since profiling is commented out, we have no URL to use

#logger_provider:
#  processors:
#    - batch:
#        exporter:
#           otlp_http:
#              endpoint: ""

Some of the options we have:

distribution:
  splunk:
    profiling: # Signal is not enabled/disabled by the existence of the node
      disabled: true
      endpoint: "collector:4318/v1/logs"
    callgraphs:
      trace_selection_probability: 0.05

or

distribution:
  splunk:
    # profiling: # Profiling is disabled
    #  endpoint: "collector:4318/v1/logs"
    callgraphs:
      endpoint: "collector:4318/v1/logs"
      trace_selection_probability: 0.05

Copy link
Contributor

@ysolomchenko ysolomchenko Nov 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And my question about otlp_log_http.
I think we should use the term from the spec, otlp_http/otlp_grpc.

Copy link
Contributor Author

@seemk seemk Nov 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And my question about otlp_log_http. I think we should use the term from the spec, otlp_http/otlp_grpc.

Agree

Copy link
Contributor

@pellared pellared Nov 27, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should use the term from the spec, otlp_http/otlp_grpc.

The problem is that we are using a "log OTLP exporter" not "profiling OTLP expoter" which we may want to in future. This is why I added _log_ 😉

Copy link

@akubik-splunk akubik-splunk Nov 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey is there any significant reason for changing the naming of profiling related attributes?
I would preferer to keep it fairly simple and using terms which are more common in the industry (cpu/memory profiler / profiling etc)
Also using term "snapshotting" sounds bit ambiguous - as snapshot can be almost a specific time-bounded data of any kind where callgraphs or stack traces more specifically call it what it is
WDYT?

Copy link
Contributor

@lachmatt lachmatt Nov 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@akubik-splunk note we are already using term "snapshots" in e.g. env variables in the spec

# Language specific configurations
runtime:
nodejs:
metrics:
runtime_metrics:
enabled: true # SPLUNK_RUNTIME_METRICS_ENABLED
collection_interval: 30000 # SPLUNK_RUNTIME_METRICS_COLLECTION_INTERVAL
instrumentation_metrics_enabled: true # SPLUNK_INSTRUMENTATION_METRICS_ENABLED
autoinstrument_packages:
- "MyApiService"
- "MyOtherService"
java:
...
Copy link
Contributor

@pellared pellared Nov 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest moving under instrumentation node.

tracer_provider:
processors:
- batch:
exporter:
otlp_http:
endpoint: ""
headers:
- name: "X-SF-TOKEN"
- value: ""
meter_provider:
readers:
- periodic:
exporter:
otlp_http:
endpoint: ""
headers:
- name: "X-SF-TOKEN"
- value: ""
logger_provider:
processors:
- batch:
exporter:
otlp_http:
endpoint: ""
```

#### [Java SystemProperties](https://docs.oracle.com/javase/tutorial/essential/environment/sysprop.html)

These properties MUST match the environment variables converting to lower
case and replacing underscores with hyphens or periods. For example:
system property `splunk.trace-response-header.enabled` is equivalent to environment
variable `SPLUNK_TRACE_RESPONSE_HEADER_ENABLED`.

### Snapshot Profiler

Expand Down
Loading