Skip to content

Splunk specific file based configuration fields #355

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Kielek merged 16 commits into from
Dec 11, 2025
Merged

Splunk specific file based configuration fields #355

Changes from 12 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
d437cda
add file based config draft
seemk Nov 11, 2025
1123de3
casing, move under distribution key
seemk Nov 25, 2025
f80d5aa
structure changes
seemk Nov 27, 2025
1bff279
js: move runtime metrics to the general block
seemk Nov 28, 2025
a6670b0
update to remove language specific entries, add splunk_prefix
seemk Dec 8, 2025
ea159c0
Merge branch 'main' into file-based-config
seemk Dec 8, 2025
6bb05d8
lint
seemk Dec 8, 2025
69a19c8
lint
seemk Dec 8, 2025
fda8404
nesting, js fixes
seemk Dec 8, 2025
6a0e2c8
remove splunk.general
seemk Dec 9, 2025
86e1dc9
add experimental status
seemk Dec 9, 2025
1251ba2
Merge branch 'main' into file-based-config
seemk Dec 10, 2025
0b64f05
remove nodejs example
seemk Dec 10, 2025
45a27be
change otlp_http -> otlp_log_http
Kielek Dec 11, 2025
0310b22
Add env-var for access token to the example
Kielek Dec 11, 2025
5af67f3
Update specification/configuration.md
Kielek Dec 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 85 additions & 10 deletions specification/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -239,24 +239,99 @@ are required.
a deprecation warning and suggest an alternate method. For example:

```txt
jaeger-thrift-splunk trace exporter is deprecated and may be removed in a future major release. Use the default
OTLP exporter instead, or set the SPLUNK_REALM and SPLUNK_ACCESS_TOKEN environment variables to send
jaeger-thrift-splunk trace exporter is deprecated and may be removed in a future major release. Use the default
OTLP exporter instead, or set the SPLUNK_REALM and SPLUNK_ACCESS_TOKEN environment variables to send
telemetry directly to Splunk Observability Cloud.
```

- `OTEL_TRACES_SAMPLER`
- Distribution MUST default to `always_on`
(not OpenTelemetry default)

In addition to environment variables, other ways of defining configuration also
exist:
In addition to environment variables, other ways of defining configuration exist:

- [Java System
Properties](https://docs.oracle.com/javase/tutorial/essential/environment/sysprop.html):
These properties MUST match the environment variables converting to lower
case and replacing underscores with hyphens or periods. For example:
system property `splunk.trace-response-header.enabled` is equivalent to environment
variable `SPLUNK_TRACE_RESPONSE_HEADER_ENABLED`.
### File based configuration

**Status**: [Experimental](../README.md#versioning-and-status-of-the-specification)

OpenTelemetry's [declarative configuration](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/configuration/README.md#declarative-configuration)
SHOULD be supported via [`OTEL_EXPERIMENTAL_CONFIG_FILE`](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/configuration/sdk.md#via-otel_experimental_config_file)
environment variable.

In addition, Splunk specific configuration MUST have their own root-level
configuration block named `splunk`.

The following is a configuration blueprint covering the common values:

```yaml
distribution:
splunk:
profiling:
exporter:
otlp_http:
endpoint: "" # SPLUNK_PROFILER_LOGS_ENDPOINT
# always_on, cpu_profiler, memory_profiler, callgraphs may contain language specific key value pairs.
always_on:
cpu_profiler: # SPLUNK_PROFILER_ENABLED
sampling_interval: 10 # SPLUNK_PROFILER_CALL_STACK_INTERVAL
memory_profiler: # SPLUNK_PROFILER_MEMORY_ENABLED
callgraphs: # SPLUNK_SNAPSHOT_PROFILER_ENABLED
sampling_interval: 10 # SPLUNK_SNAPSHOT_SAMPLING_INTERVAL
selection_probability: 0.01 # SPLUNK_SNAPSHOT_SELECTION_PROBABILITY
tracer_provider:
processors:
- batch:
exporter:
otlp_http:
endpoint: ""
headers:
- name: "X-SF-TOKEN"
- value: ""
meter_provider:
readers:
- periodic:
exporter:
otlp_http:
endpoint: ""
headers:
- name: "X-SF-TOKEN"
- value: ""
logger_provider:
processors:
- batch:
exporter:
otlp_http:
endpoint: ""
instrumentation/development:
js:
# Language specific distro configuration.
# splunk keyword is for general distro config.
splunk:
use_bundled_instrumentations: false
package_name_filter:
- "MyApiGw"
metrics:
splunk_runtime_metrics:
collection_interval: 30000
traces:
http:
splunk_trace_response_header_enabled: true # SPLUNK_TRACE_RESPONSE_HEADER_ENABLED
Copy link
Contributor

@ysolomchenko ysolomchenko Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As it is already under traces, the property name could simply be response_header_enabled:

Copy link
Contributor Author

@seemk seemk Dec 10, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why? It's a Splunk extension and is in parity with the env var

Copy link
Contributor

@ysolomchenko ysolomchenko Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because its redundant, trace under traces node its like butter on butter

Copy link
Contributor Author

@seemk seemk Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see you meant trace_ not splunk_

splunk_capture_uri_parameters:
- "userId"
redis:
splunk_include_command_args: true # SPLUNK_REDIS_INCLUDE_COMMAND_ARGS
pg:
disabled: true
Copy link
Contributor

@ysolomchenko ysolomchenko Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we already have splunk: as a semi-node here, we can remove the splunk_ prefix from the properties.

Copy link
Contributor

@ysolomchenko ysolomchenko Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For .NET, it might be an issue to create this splunk: node under instrumentation/development:
For example we have SPLUNK_TRACE_RESPONSE_HEADER_ENABLED env var.
In yaml it should be like this

instrumentation/development:
  dotnet:
    traces: 
      aspnet:
        splunk_response_header_enabled: true
      aspnetcore:
        splunk_response_header_enabled: true

These instrumentations aspnet and aspnetcore come from upstream, not from our distro. In this case, IMO using the prefix is better than adding a separate node under instrumentation/development:{language}:.

If we add this node, it will look like this:

instrumentation/development:
  dotnet:
    traces: 
      aspnet:
      aspnetcore:
    splunk: 
      traces:
        response_header_enabled: true

As you can see, this structure looks a bit ugly and inconsistent compared to the first proposal.

Copy link

@robsunday robsunday Dec 10, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my opnion under instrumentation/development.{language} node we do not have to be fully consistent, however it would be good to have consistency on common settings.
SPLUNK_TRACE_RESPONSE_HEADER_ENABLED seems to be the only property besides profiling that is common across all the languages.

Actually I thought this would be good enough:

instrumentation/development:
  java:
    splunk: 
      trace_response_header_enabled: true
  dotnet:
    splunk: 
      trace_response_header_enabled: true
  nodejs:
    splunk: 
      trace_response_header_enabled: true
  etc...

trace_response_header_enabled is a valid name in my opinion because this setting according to the docs:

Enables adding server trace information to HTTP response headers

Copy link
Contributor Author

@seemk seemk Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my opnion under instrumentation/development.{language} node we do not have to be fully consistent, however it would be good to have consistency on common settings. SPLUNK_TRACE_RESPONSE_HEADER_ENABLED seems to be the only property besides profiling that is common across all the languages.

Actually I thought this would be good enough:

instrumentation/development:
  java:
    splunk: 
      trace_response_header_enabled: true
  dotnet:
    splunk: 
      trace_response_header_enabled: true
  nodejs:
    splunk: 
      trace_response_header_enabled: true
  etc...

trace_response_header_enabled is a valid name in my opinion because this setting according to the docs:

Enables adding server trace information to HTTP response headers

So why not have it under under distribution.splunk then? The spec now has a top level node for distro specific features.

Copy link
Contributor

@ysolomchenko ysolomchenko Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@seemk So my opinion it should be

instrumentation/development:
  dotnet:
    traces: 
      aspnet:
        splunk_response_header_enabled: true
      aspnetcore:
        splunk_response_header_enabled: true
  js:
    traces:
      http:
        splunk_response_header_enabled: true
  java:
    ...

Copy link
Contributor Author

@seemk seemk Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 2 issues I have with this:

  1. Duplicated configuration option (aspnet, aspnetcore), correct me if I'm wrong about .net, but on Node both programmatic and env var based config has the option to toggle it in a single place.
  2. As an user, is it intuitive that you need to know the instrumentation names for our distro specific config? One point of our distros has been to remove the gnarlyness of OTel behind sensible, easy options

Copy link
Contributor

@ysolomchenko ysolomchenko Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two instrumentation libraries because of .NET version differences. For .NET Framework, we use aspnet, and for .NET (Core), we use aspnetcore. response_header_enabled will not working without aspnet or aspnetcore so the end users should know the instrumentation name anyway if they want use the file based configuration with response_header_enabled

Copy link
Contributor Author

@seemk seemk Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And when the setting is in the distro configuration (once), why can't the distro figure it out for the user? Do you explicitly have to list the instrumentations you use for .net?

Copy link

@robsunday robsunday Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In fact, I think the lang distros can put whatever they want in their lang block. We're spending way too much time on trivial configuration fields.

I think it provides good solution for this.

Copy link
Contributor

@ysolomchenko ysolomchenko Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And when the setting is in the distro configuration (once), why can't the distro figure it out for the user? Do you explicitly have to list the instrumentations you use for .net?

Yes, you need to explicitly list them to enable.

There is an issue with the templates that were already created, but I think it’s a bit stuck. Because main idea of file-based configuration is that what you see is what you get.

```

Splunk specific keys MUST have the `splunk_` prefix.

#### [Java SystemProperties](https://docs.oracle.com/javase/tutorial/essential/environment/sysprop.html)

These properties MUST match the environment variables converting to lower
case and replacing underscores with hyphens or periods. For example:
system property `splunk.trace-response-header.enabled` is equivalent to environment
variable `SPLUNK_TRACE_RESPONSE_HEADER_ENABLED`.

### Snapshot Profiler

Expand Down