Support change notifications

[jmlMetaswitch]

Feature Request:
There are a number of good reasons why it would be very useful if Vault supported change notifications for secrets and tokens.

Even in a generic microservice architecture, where only one microservice type accesses any given data stored in a Vault cluster, multiple microservice instances are accessing multiple Vault instances, so change notifications are the cleanest way to keep all microservice instances updated.

A suitable notification mechanism might be the one used by etcd: long polls (though preferably with a configurable time-out). The client issues "GET .........?wait=true&wait_timeout=30" for instance, and the response comes when the data changes or the timeout expires (in which case 204 "No Content" is returned).

As discussed in #1558, many back-ends support change notification, so this shouldn't be too hard to implement in many common scenarios. If the back-end support is not available, it could either be implemented with a polling mechanism or the request for notifications could be rejected (with 501 "Not Implemented" for instance).

I assume the only recommended workarounds remain

• watching the underlying store, as discussed in How best to watch for changes? #1558
• providing a proxy or another "over the top" mechanism for the client making the change to notify all other clients.

Neither seems very satisfactory, particularly in a generic microservice architecture.

Do you agree that this would be a useful feature for Vault, and that there isn't an easy workaround?

References:
#1558 asks about this, but is not a feature request and has been closed without a positive resolution.

[jefferai]

We want to offer something like this eventually although it's unlikely to be via the mechanism you described above. Allowing clients to hold arbitrary numbers of long-lived connections open is a great way for a DoS (accidental or not) and it will not surprise you that Vault doesn't like opening that kind of possibility if it can be helped. Also, unlike etcd Vault isn't just a key/value store; paths represent potentially arbitrarily complex values under the hood, which doesn't make it straightforward. Some kind of capability like this is on the roadmap, but there's no time estimate right now.

[jmlMetaswitch]

Okay, we'll look for an alternative approach. Shame, but thanks for the prompt response.

[jmlMetaswitch]

@jefferai - I'll let you decide whether to leave this open or close it.

[jefferai]

@jmlMetaswitch To be clear, we do want to have this functionality, it's just actually very complex to do within Vault, because it's a much more complex system than K/V stores due to the arbitrary nature of its backends and its complex security layers. There's nothing we can add in any kind of rapid fashion.

[colceagus]

👍

[colceagus]

It would facilitate the integration with spring-boot-actuator :)

[trulyliu]

Is this feature implemented?

[jefferai]

Not currently.

[onorua]

@jefferai I've come here after trying to solve tracking of the secrets (several hundreds of them) for changes.
Do you have an idea how do you see this feature?
Currently, having your inputs from above comments, I come up with the following solution:

1. We create separate service which is pubsub server
2. Every change to any of engines comes through some middleware, which sends notification to that pubsub server
3. Inside the notification we send full secret name.
4. It is possible to encrypt key name if necessary (in our case it is not required).
5. One pubsub server could be chained and may have a limit of 10k subscribers or similar sane limit.
   BTW, Consul may be that pubsub server.

In this case we solve several problems:

1. We don't have thousands of clients waiting for events from vault (we have only one).
2. Vault core changes become trivial, as it only need to know gRPC endpoint and send it over there.
3. It doesn't go against idea to have single API REST interface to Vault.

What do you think? Does it sounds like a reasonable solution to you? Do you see any problems? Maybe you have some other ideas how to get to the point when we can track secret changes?

[mkozjak]

I think it would be good to reopen this issue as it seems basic, yet so critical for any sane use-case where a secure approach of fetching tokens is leveraged.

I have just opened a topic on forums regarding vault agent auto-auth token revocation handling, which seems to go along with this issue. You can find it here: https://groups.google.com/forum/#!topic/vault-tool/gsN8LT3wJqQ

tldr; If you use vault agent auto-auth (with kubernetes auth...) you currently cannot act on events such as token revocation where the agent is used as a sidecar which provides vault tokens to the main app (in a k8s pod). Is that true? Meaning we can just set token TTL to three seconds and pray the app which uses those tokens isn't hit bad with traffic. Is there any way to get any events out of Vault server?

[jefferai]

@mkozjak We have plans for an event system but it's orthogonal to your problem. Follow-up can happen there.

[mkozjak]

@jefferai Thanks for the update! Anywhere I can track the feature?

[jefferai]

Not currently. We've been thinking through it conceptually. There is no hard roadmap commitment and no design docs.

[cpoole]

we'd also love to be able to subscribe to a filter of events.

We're currently investgating building a tcp "demuxer" that listens to the audit socket and forwards events to kafka topics.

So far so good, but it would be great to have something more baked in.

[Eslamanwar]

Is this feature is implemented? , it is very common practice in kubernetes to restart all pods when a secrets is changed or the application will not be consistent

[mailtoyasi]

any updates or plans on this feature?

[theromis]

Please implement it, everybody wants it.

[martinm82]

We are as well interested in this feature. A very generic solution would be in the form of webhooks, that get triggered when specific events occur. Similar like Artifactory does: https://www.jfrog.com/confluence/display/JFROG/Webhooks.

Any chance this issue get's more attention?

[booleanbetrayal]

Just wanted to chime in to say that this is something we currently feel the need for in order to form an appropriate observability / security posture around Vault use.

[victorstewart]

+1, this is a critical missing feature

[heatherezell]

Re-opening for my own tracking. I will bring this up with product again; I can't make any guarantees, but since folks very much want this I will make sure they're also tracking this request. Thanks for everyone's input!

[victorstewart]

@hsimon-hashicorp actually i just watched the hashicon vault roadmap video yesterday, and they said this feature is coming.

you can do things like have a secret autorotated on a specified schedule and the new secret pushed to you. like new certificates every 72 hours.

[heatherezell]

@hsimon-hashicorp actually i just watched the hashicon vault roadmap video yesterday, and they said this feature is coming.

you can do things like have a secret autorotated on a specified schedule and the new secret pushed to you. like new certificates every 72 hours.

That's correct - will that be sufficient for your request? Part of why I reopened this was because of a little bit of "and alsos", so if there's specific enhancements that people who've been following along would like, please let me know! I can aggregate them and see if we can get it planned out and scheduled. :)

[victorstewart]

@hsimon-hashicorp one can imagine other change needs such as... revoking my 3rd party API token and needing to write the new version into Vault and push the new value to every subscriber.

not sure the new features include this ability?

otherwise it has to be done manually by writing a script to discover every instance and ping it to refresh from Vault. or all the programs have to be torn down and rebooted.

[alldayalone]

+1 for this feature.
In my case, I'd like to receive a slack notification every time a new version of a secret is created. For monitoring purposes (big team)

[powhapki]

This one is really needed for enhancing the observability of current Vault infra and usages.

[fdaruos]

I think this is also an important missing capabilities on the Vault Agent side. Vault Agent should be able to download being informed for credential or secrets that are downloaded / retrieved by Vault Agent.

[raphaeldavidf]

any updates on this @jefferai?

[riveryc]

https://developer.hashicorp.com/vault/docs/concepts/events#subscribing-to-event-notifications
They added in their enterprise edition.