Caching

[jlambatl]

Hello,

Reading the documentation, I see this: https://github.com/vectordotdev/vrl/blob/main/DESIGN.md#performance-warning, which talks about running a cache. I see that there are also in-memory tables that can potentially be used in a similar situation.

Has there been any thought to adding VRL functions for some of the common caches, or potentially adding something like groupcache functions/support?

As we transition some of our pipelines over to Vector and use VRL heavily, there are some use cases where being able to access data centrally would be a big win. We currently use Flink for this use case; however, the complexity of Flink for simple global counters and similar atomic counters is costly and probably overkill.

[pront]

Hi @jlambatl, this has been discussed in the past. I am personally a fan of this idea.

For example, see this thread:
vectordotdev/vector#4517 (comment)

And this issue:
#137