[confmap] - new feature flag for append merging strategy

[VihasMakwana]

Koanf's default merging strategy currently overrides static values such as slices, numbers, and strings. However, lists of components should be treated as a special case. This pull request introduces a new command line option to allow for merging lists instead of discarding the existing ones.

With this new merging strategy:

• All the lists are merged rather than replaced.
• The merging logic is name-aware, meaning that if components with the same name appear in both lists, they will only appear once in the final merged list.

Link to tracking issue

Related issues:

• Append components flag #8754
• Allow instantiating extensions in separate configs #8394
• [ConfMap] - Providing custom merge method for koanf's Map.Merge #10370

Testing

• Added

Documentation

• Added under readme.

Example

Consider the following configs,

# main.yaml
receivers:
  otlp/in:
processors:
  batch:
exporters:
  otlp/out:
extensions:
  file_storage:

service:
  pipelines:
    traces:
      receivers: [ otlp/in ]
      processors: [ batch ]
      exporters: [ otlp/out ]
  extensions: [ file_storage ]

# extra_extension.yaml
extensions:
  healthcheckv2:

service:
  extensions: [ healthcheckv2 ]

If you run the collector with following command,

otelcol --config=main.yaml --config=extra_extension.yaml --feature-gates=-confmap.enableMergeAppendOption

then the final configuration after config resolution will look like following:

# main.yaml
receivers:
  otlp/in:
processors:
  batch:
exporters:
  otlp/out:
extensions:
  file_storage:
  healthcheckv2:

service:
  pipelines:
    traces:
      receivers: [ otlp/in ]
      processors: [ batch ]
      exporters: [ otlp/out ]
  extensions: [ file_storage, healthcheckv2 ]

For backward compatibly, the default behaviour is not to merge lists. Users who want to explicitly merge lists can enable the command line option.

Note: I’d appreciate your feedback on this 🙏

[VihasMakwana]

@Aneurysm9 @evan-bradley @mx-psi can you take another look?

i've removed feature flag and added command line argument to control the paths user wants merged.

I'll work on the documentation once we've established a common ground.

[codecov]

Codecov Report

Attention: Patch coverage is 98.18182% with 1 line in your changes missing coverage. Please review.

Project coverage is 92.01%. Comparing base (dc62746) to head (86c64d3).

Files with missing lines	Patch %	Lines
confmap/resolver.go	85.71%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #12097      +/-   ##
==========================================
+ Coverage   92.00%   92.01%   +0.01%     
==========================================
  Files         469      470       +1     
  Lines       25355    25409      +54     
==========================================
+ Hits        23327    23381      +54     
  Misses       1619     1619              
  Partials      409      409              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mx-psi]

Thanks for taking this Vihas! I have two comments:

1. I think this is the kind of thing that is hard to get right on the first try, so I would want to do this under some sort of experimental flag.
2. I would like to do this if possible with little to no modification of stable confmap APIs. confmap is 1.0, it is a very important piece of the Collector, and if we get it wrong it would be quite difficult to reverse course.

[VihasMakwana]

@mx-psi Thanks for your comments!

1. Makes sesne. I will move it under experimental flag.
2. I understand. I will again take a fresh look and make sure we tick all the boxes.

[dmitryax]

The order of the config files defines this, just like the order defines which file overrides which right now. We can document this if it is unclear.

My concern is not only about the ordering but about controlling that it's applied to configuration parts where it's expected. For example:

I want to define a component in another config file and add it to a "standard" pipeline defined in a "base" config, but the "base" config may already have it as well (I don't have control over it). In the case of a conflict, I'd like to be able to choose whether to override the original component's config or keep it as is. Overriding maps is currently impossible, given that we have parts with meaningful null values like receivers.otlp.protocols.grpc (something we can eventually address). But at least we can control overriding lists in the component's config. This all-or-nothing approach doesn't allow that.

@VihasMakwana @mx-psi I'm ok with this as an initial version. I'm just afraid that it'll be stuck in this state for a long time, and users will be relying on the behavior of this feature gate going forward. Can we have another issue where we track the progress? @VihasMakwana, will you be able to keep working on this until it's stabilized?

[VihasMakwana]

The order of the config files defines this, just like the order defines which file overrides which right now. We can document this if it is unclear.

My concern is not only about the ordering but about controlling that it's applied to configuration parts where it's expected. For example:

I want to define a component in another config file and add it to a "standard" pipeline defined in a "base" config, but the "base" config may already have it as well (I don't have control over it). In the case of a conflict, I'd like to be able to choose whether to override the original component's config or keep it as is. Overriding maps is currently impossible, given that we have parts with meaningful null values like receivers.otlp.protocols.grpc (something we can eventually address). But at least we can control overriding lists in the component's config. This all-or-nothing approach doesn't allow that.

@VihasMakwana @mx-psi I'm ok with this as an initial version. I'm just afraid that it'll be stuck in this state for a long time, and users will be relying on the behavior of this feature gate going forward. Can we have another issue where we track the progress? @VihasMakwana, will you be able to keep working on this until it's stabilized?

yes. This is one of my top priorities and I'll open up an follow up issue to track progress.

[VihasMakwana]

@mx-psi @evan-bradley @dmitryax I think we can go forward this PR as an initial version. I have reverted the command line for now.

I'll open up an RFC as a follow-up and discuss how to address concerns related to ordering and controlling of specific parts of configuration.

[dmitryax]

LGTM

[dmitryax]

Please update the nolint comment to fix CI

[VihasMakwana]

A few of test cases are failing but they seem unrelated to my PR.