feat(secrets): Add new secrets management package #797

hsmatulis · 2025-06-27T14:16:05Z

This PR introduces a new package, secrets, to prometheus/common. This package provides a unified way to handle secrets within configuration files for Prometheus and its ecosystem components. It is designed to be extensible and observable. See the proposal here

Signed-off-by: Henrique Spanoudis Matulis <[email protected]>

hsmatulis · 2025-10-02T14:19:08Z

@pintohutch @bernot-dev @bwplotka PTAL, should be ready!

bwplotka

Thanks, amazing work!

Generally, it's great - maybe first big question is if we want SecretField to store so much state or do we want to move some of this state to manager (or and providers). This might be more composable and easier to reason about. Prometheus discovery is doing some of this. I mentioned one idea (A) 1 and 2 in comments.

However, it's not a blocker, even in the current state, I would say we could try this out on Prometheus (and someone might try on AM side!). What matters is that I see a clean YAML surface format, clean code, idiomatic struct reflection to find fields and healthy amount of test, so amazing! With this we can iterate.

No matter if you want to try (A) or skip it for now, I think I would try to check before merging:

Limit the global variable spread (https://github.com/prometheus/common/pull/797/files#r2414525826)
If we can skip validator and timeout based validator state per field, if it's YAGNI (see https://github.com/prometheus/common/pull/797/files#r2414536374)
Use testable examples: https://github.com/prometheus/common/pull/797/files#r2414499436

Then I would like to review in more depth the manager code, but generally... we could start with this! 🎉 Thanks!

secrets/field.go

bwplotka · 2025-10-08T14:22:24Z

secrets/field.go

+}
+
+type SecretFieldSettings struct {
+	RefreshInterval time.Duration `yaml:"refreshInterval,omitempty"`


question: Hm, good question if refresh is really relevant to all providers... E.g it's not for inline and.. also not for file based? Refreshing in general is a key concept here so we at least needs a good commentary in the code for this field on how it suppose to be implemented and consumed?

Hmm I guess it is not relevant to inline but could be relevant to file. Although we might want to think about listening to filesystem changes for a file based provider instead, but I found that to complicate the API quite a bit...

secrets/resolve_test.go

bwplotka · 2025-10-08T17:02:17Z

secrets/README.md

+
+In your configuration struct, use the `secrets.SecretField` type for any fields that should contain secrets.
+
+```go


Sustainability suggestion (non-blocking):

This is well documented, thanks!

However, it might be even better if we would replace the "How to use" section with a single or set of buildable examples as per https://go.dev/blog/examples .. and commentary in the key public code structures.

bwplotka · 2025-10-08T17:26:53Z

secrets/field.go

+	}
+}
+
+func (s *SecretField) Get() string {


Nice! This make caller unaware of errors which can be both great and bad, depending what we do. I think it would be great to test out in this state 👍🏽

One alternative to make things leaner on SecretField (decompose responsibilities and state), would be if SecretField would allow storing (atomically) and getting secret value string. This way we could have common package global free and SecretField manager free (manager could be responsible to set correct things in config).

It could be even provider agnostic/free combined with https://github.com/prometheus/common/pull/797/files#r2414589581 idea (and avoid globals in this package)

secrets/resolve.go

bwplotka · 2025-10-08T17:28:39Z

secrets/manager.go

+	return manager, nil
+}
+
+func (m *Manager) registerMetrics(r prometheus.Registerer) {


Thanks for metrics, super important!

bwplotka · 2025-10-08T17:29:57Z

secrets/manager.go

+	return secret.secret
+}
+
+func (m *Manager) triggerRefresh(s *SecretField) {


note to myself: Manager code needs deeper review, quite many locks 🙈 prone to errors, but perhaps needed.

yeah it was very complicated before with the verify logic, I think it should be a bit simpler now.. Still need some cleanup there, but should be more readable

bwplotka · 2025-10-08T17:43:15Z

secrets/field.go

+	return nil
+}
+
+func (s *SecretField) UnmarshalYAML(unmarshal func(interface{}) error) error {


Have we considered doing plainSecret thingy, but then unmarshal to any or map[string]any and let manager.ApplyConfig do the final parsing? This would make SecretField leaner and manager/provider agnostic (together with setter/getter for secret values https://github.com/prometheus/common/pull/797/files#r2414554264)

…terfaces Signed-off-by: Henrique Matulis <[email protected]>

Remove all verifier logic, as it is not needed for now. This removes the interface and all related logic for validating secrets before they are activated. This simplifies the secret management lifecycle by removing the pending/verification state. Fetched secrets will now become active immediately.

Prevent global propagation Remove unused default value

bwplotka

Looks solid, thanks!

Still some few comments to address, but overall looks close to be merged!

secrets/manager.go

bwplotka · 2025-11-14T15:49:53Z

secrets/field.go

+type Field struct {
+	providerName   string
+	providerConfig ProviderConfig
+	manager        *Manager


idea during 1:1, it would be nice to remove bi-direction and have lazy refresh perhaps

Alternatively just use interface for Refresh()

bwplotka · 2025-11-14T15:52:20Z

secrets/README.md

+  * `ProviderConfig`: An interface for configuring a secret provider. It includes methods to create a new `Provider` instance and to clone the configuration.
+  * `ProviderConfigId`: An interface for uniquely identifying a `ProviderConfig` instance.
+  * `Provider`: An interface for fetching secrets from a specific source (e.g., inline string, file on disk). The package comes with built-in providers, and new ones can be registered.
+  * `Manager`: A component that discovers all `SecretField` instances within a configuration struct, manages their lifecycle, and handles periodic refreshing of secrets.


Can we put those into comments? Especially for library, unlikely anyone will notice README. In go it's either doc.go or code structure for godocs.

https://go.dev/blog/godoc

https://go.dev/blog/examples

Signed-off-by: Henrique Matulis <[email protected]>

hsmatulis force-pushed the main branch 3 times, most recently from cf118e6 to 24ec0f7 Compare July 7, 2025 11:07

hsmatulis force-pushed the main branch 18 times, most recently from 1882b4c to 694bf26 Compare October 2, 2025 07:35

feat(secrets): Add new package for managing secrets

ef24c3d

Signed-off-by: Henrique Spanoudis Matulis <[email protected]>

hsmatulis force-pushed the main branch from 694bf26 to ef24c3d Compare October 2, 2025 07:40

hsmatulis changed the title ~~feat(secrets): Introduce new package for dynamic secret management~~ feat(secrets): Add new secrets management package Oct 2, 2025

hsmatulis marked this pull request as ready for review October 2, 2025 07:42

bwplotka self-requested a review October 8, 2025 15:15

bwplotka reviewed Oct 8, 2025

View reviewed changes

hsmatulis added 2 commits November 13, 2025 03:25

refactor(secrets): Split Provider into ProviderConfig and Provider in…

57fea57

…terfaces Signed-off-by: Henrique Matulis <[email protected]>

hsmatulis added 5 commits November 13, 2025 03:59

refactor(secrets): Store secret values in SecretField

f55d309

refactor(secrets): Clean up nits

c8805b0

Prevent global propagation Remove unused default value

fix(secrets): formatting

5205ea8

fix(secrets): Clean up linter errors

285c284

refactor(secrets): Eat Secret in naming

d8cf227

bwplotka reviewed Nov 14, 2025

View reviewed changes

fix(secrets): improve code style

c6cca35

Signed-off-by: Henrique Matulis <[email protected]>


		In your configuration struct, use the `secrets.SecretField` type for any fields that should contain secrets.

		```go

feat(secrets): Add new secrets management package #797

Are you sure you want to change the base?

feat(secrets): Add new secrets management package #797

Uh oh!

Conversation

hsmatulis commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hsmatulis commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bwplotka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bwplotka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

hsmatulis commented Jun 27, 2025 •

edited

Loading

hsmatulis commented Oct 2, 2025 •

edited

Loading