Epic: IDistributedCache updates in .NET 9

[adityamandaleeka]

Update:

HybridCache has relocated to dotnet/extensions:dev; it does not ship in .NET 9 RC1, as a few missing and necessary features are still in development; however, we expect to ship either alongside or very-shortly-after .NET 9! ("extensions" has a different release train, that allows additional changes beyond the limit usually reserved for in-box packages; HybridCache has always been described as out-of-box - i.e. a NuGet package - so: there is no reason for us to limit ourselves by the runtime restrictions)

---

Status: feedback eagerly sought

• API proposal part 1, core API: Hybrid Cache API proposal #54647 and Add HybridCache public API #55084
• Initial cut of proposal part 1: Implement minimal implementation of HybridCache #55147 and fixes: HybridCache - minor post-PR fixes #55251
• API proposal part 2, tags and invalidation: HybridCache - tags and invalidation #55308
• By nomenclature: HybridCache - rename RemoveKeyAsync and RemoveTagAsync to "By" #55332

Tl;Dr

• add a new HybridCache API (and supporting pieces) to support more convenient and efficient distributed cache usage
• support read-through caching with lambda callbacks
• support flexible serialization
• support stampede protection
• support L1/L2 cache scenarios
• build on top of IDistributedCache so that all existing cache backends work without change (although they could optionally add support for new features)
• support comparable expiration concepts to IDistributedCache

Problem statement

The distributed cache in asp.net (i.e. IDistributedCache) is not particularly developed; it is inconvenient to use, lacks many desirable features, and is inefficient. We would like this API to be a "no-brainer", easy to get right feature, making it desirable to use - giving better performance, and a better experience with the framework.

Typical usage is shown here; being explicit about the problems:

Inconvenient usage

The usage right now is extremely manual; you need to:

• attempt to read a stored value (as byte[])
• check that value for null ("no value")
  • if not null:
    • fetch the value
    • serialize it
    • store the value
  • return the value

This is a lot of verbose boilerplate, and while it can be abstracted inside projects using utility methods (often extension methods), the vanilla experience is very poor.

Inefficiencies

The existing API is solely based on byte[]; the demand for right-sized arrays means no pooled buffers can be used. This broadly works for in-process memory-based caches, since the same byte[] can be returned repeatedly (although this implicitly assumes the code doesn't mutate the data in the byte[]), but for out-of-process caches this is extremely inefficient, requiring constant allocation.

Missing features

The existing API is extremely limited; the concrete and implementation-specific IDistributedCache implementation is handed directly to callers, which means there is no shared code reuse to help provide these features in a central way. In particular, there is no mechanism for helping with "stampede" scenarios - i.e. multiple concurrent requests for the same non-cached value, causing concurrent backend load for the same data, whether due to a cold-start empty cache, or key invalidation. There are multiple best-practice approaches that can mitigate this scenario, which we do not currently employ.

Likewise, we currently assume an in-process or out-of-process cache implementation, but caching almost always benefits from multi-tier storage, with a limited in-process (L1) cache supplemented by a separate (usually larger) out-of-process (L2) cache; this gives the "best of both" world, where the majority of fetches are served efficiently from L1, but cold-start and less-frequently-accessed data still doesn't hammer the underlying backend, thanks to L2. Multi-tier caching can sometimes additionally exploit cache-invalidation support from the L2 implementation, to provide prompt L1 invalidation as required.

This epic proposes changes to fill these gaps

Current code layout

At the moment the code is split over multiple components, in the main runtime, asp.net, and external packages (only key APIs shown):

• Microsoft.Extensions.Caching.Abstractions

  • src: https://github.com/dotnet/runtime/tree/main/src/libraries/Microsoft.Extensions.Caching.Abstractions
  • pkg: https://www.nuget.org/packages/Microsoft.Extensions.Caching.Abstractions/
  • adds IDistributedCache
  • adds DistributedCacheEntryOptions

• Microsoft.Extensions.Caching.Memory

  • src: https://github.com/dotnet/runtime/tree/main/src/libraries/Microsoft.Extensions.Caching.Memory
  • pkg: https://www.nuget.org/packages/Microsoft.Extensions.Caching.Memory/
  • adds AddDistributedMemoryCache
  • adds MemoryDistributedCache : IDistributedCache

• Microsoft.Extensions.Caching.StackExchangeRedis

  • src: https://github.com/dotnet/aspnetcore/tree/main/src/Caching/StackExchangeRedis
  • pkg: https://www.nuget.org/packages/Microsoft.Extensions.Caching.StackExchangeRedis
  • adds AddStackExchangeRedisCache

• Microsoft.Extensions.Caching.SqlServer

  • src: https://github.com/dotnet/aspnetcore/tree/main/src/Caching/SqlServer
  • pkg: https://www.nuget.org/packages/Microsoft.Extensions.Caching.SqlServer
  • adds AddDistributedSqlServerCache

• Microsoft.Extensions.Caching.Cosmos

  • src: https://github.com/Azure/Microsoft.Extensions.Caching.Cosmos
  • pkg: https://www.nuget.org/packages/Microsoft.Extensions.Caching.Cosmos
  • adds AddCosmosCache

• Alachisoft.NCache.OpenSource.SDK

  • src: https://www.alachisoft.com/ncache/
  • pkg: https://www.nuget.org/packages/Alachisoft.NCache.OpenSource.SDK/
  • adds AddNCacheDistributedCache

• AWS

  • https://github.com/awslabs/aws-dotnet-distributed-cache-provider

This list is not exhaustive - other 3rd-party and private implementations of IDistributedCache exist, and we should avoid breaking the world.

Proposal

The key proposal here is to add a new caching abstraction that is more focused, HybridCache, in Microsoft.Extensions.Caching.Abstractions; this API is designed to act more as a read-through cache, building on top[ of the existing IDistributedCache implementation, providing all the implementation details required for a rich experience. Additionally, while simple defaults are provided for the serializer, it is an explicit aim to make such concerns fully configurable, allowing for json, protobuf, xml, etc serialization as appropriate to the consumer.

namespace Microsoft.Extensions.Caching.Distributed;

public abstract class HybridCache // default concrete impl provided by service registration
{
    protected HybridCache() { }

    // read-thru usage
    public abstract ValueTask<T> GetOrCreateAsync<TState, T>(string key, TState state, Func<TState, CancellationToken, ValueTask<T>> callback, HybridCacheEntryOptions? options = null, ReadOnlyMemory<string> tags = default, CancellationToken cancellationToken = default);
    public virtual ValueTask<T> GetOrCreateAsync<T>(string key, Func<CancellationToken, ValueTask<T>> callback,
    HybridCacheEntryOptions? options = null, ReadOnlyMemory<string> tags = default, CancellationToken cancellationToken = default)
    { /* shared default implementation uses TState/T impl */ }

    // manual usage
    public abstract ValueTask<(bool Exists, T Value)> GetAsync<T>(string key, HybridCacheEntryOptions? options = null, CancellationToken cancellationToken = default);
    public abstract ValueTask SetAsync<T>(string key, T value, HybridCacheEntryOptions? options = null, ReadOnlyMemory<string> tags = default, CancellationToken cancellationToken = default);

    // key invalidation
    public abstract ValueTask RemoveKeyAsync(string key, CancellationToken cancellationToken = default);
    public virtual ValueTask RemoveKeysAsync(ReadOnlyMemory<string> keys, CancellationToken cancellationToken = default)
    { /* shared default implementation uses RemoveKeyAsync */ }

    // tag invalidation
    public virtual ValueTask RemoveTagAsync(string tag, CancellationToken cancellationToken = default)
    { /* shared default implementation uses RemoveTagsAsync */ }
    public virtual ValueTask RemoveTagsAsync(ReadOnlyMemory<string> tags, CancellationToken cancellationToken = default) => default;
}

Notes:

• the intent is that instead of requesting IDistributedCache, consumers might use HybridCache; to enable this, the consumer must additionally perform a services.AddHybridCache(...); step during registration
• the naming of GetOrCreateAsync<T> is for parity with MemoryCache.GetOrCreateAsync<T>
• RemoveAsync and RefreshAsync mirror the similarIDistributedCache methods
• it is expected that the callback (when invoked) will return a non-null value; consistent with MemoryCache et-al, null is not a supported value, and an appropriate runtime error will be raised

Usage of this API is then via a read-through approach using lambda; the simplest (but slightly less efficient) approach would be simply:

// HybridCache injected via DI
var data = await cache.GetOrCreateAsync(key, _ => /* some backend read */, [expiration etc], [cancellation]);

In this simple usage, it is anticipated that "captured variables" etc are used to convey the additional state required, as is common for lambda scenarios. A second "stateful" API is provided for more advanced scenarios where the caller wishes to trade convenience for efficiency; this usage is slightly more verbose but will be immediately familiar to the users who would want this feature:

// HybridCache injected via DI
var data = await cache.GetOrCreateAsync(key, (some state here), static (state, _) => /* some backend read */, [expiration etc], [cancellation]);

This has been prototyped and works successfully with type inference etc.

The implementation (see later) deals with all the backend fetch, testing, serialization etc aspects internally.

(in both examples, the "discard" (_) is conveying the CancellationToken for the backend read, and can be used by providing a receiving lambda parameter)

An internal implementation of this API would be registered and injected via a new AddHybridCache API (Microsoft.Extensions.Caching.Abstractions):

namespace Microsoft.Extensions.Caching.Distributed;

public static class HybridCacheServiceExtensions
{
    public static IServiceCollection AddHybridCache(this IServiceCollection services, Action<HybridCacheOptions> setupAction)
    {...}

    public static IServiceCollection AddHybridCache(this IServiceCollection services)
    {...}
}

The internal implementation behind this would receive IDistributedCache for the backend, as it exists currently; this means that the new implementation can use all existing distributed cache backends. By default, AddDistributedMemoryCache is also assumed and applied automatically, but it is intended that this API be effective with arbitrary IDistributedCache backends such as redis, SQL Server, etc. However, to address the issue of byte[] inefficiency, a new entirely optional API is provided and tested for; if the new backend is detected, lower-allocation usage is possible. This follows the pattern used for output-cache in net8:

namespace Microsoft.Extensions.Caching.Distributed;

public interface IBufferDistributedCache : IDistributedCache
{
    ValueTask<CacheGetResult> GetAsync(string key, IBufferWriter<byte> destination, CancellationToken cancellationToken);
    ValueTask SetAsync(string key, ReadOnlySequence<byte> value, DistributedCacheEntryOptions options, CancellationToken cancellationToken);
}

public readonly struct CacheGetResult
{
    public CacheGetResult(bool exists);
    public CacheGetResult(DateTime expiry);

    public CacheGetResult(TimeSpan expiry);

    public bool Exists { get; }
    public TimeSpan? ExpiryRelative { get; }
    public DateTime? ExpiryAbsolute { get; }
}

(the intent of the usual members here is to convey expiration in the most appropriate way for the backend, relative vs absolute, although only one can be specified; the internals are an implementation detail, likely to use overlapped 8-bytes for the DateTime/TimeSpan, with a discriminator)

In the event that the backend cache implementation does not yet implement this API, the byte[] API is used instead, which is exactly the status-quo, so: no harm. The purpose of CacheGetResult is to allow the backend to convey backend expiration information, relevant for L1+L2 scenarios (design note: async precludes out TimeSpan?; tuple-type result would be simpler, but is hard to tweak later). The expiry is entirely optional and some backends may not be able to convey it, and we need to handle it lacking when IBufferDistributedCache is not supported - in either event, the inbound expiration relative to now will be assumed for L1 - not ideal, but the best we have.

Serialization

For serialization, a new API is proposed, designed to be trivially implemented by most serializers - again, preferring modern buffer APIs:

namespace Microsoft.Extensions.Caching.Distributed;

public interface IHybridCacheSerializer<T>
{
    T Deserialize(ReadOnlySequence<byte> source);
    void Serialize(T value, IBufferWriter<byte> target);
}

Inbuilt handlers would be provided for string and byte[] (and possibly BinaryData if references allow); an extensible serialization configuration API supports other types - by default, an inbuilt object serializer using System.Text.Json would be assumed, but it is intended that alternative serializers can be provided globally or per-type. This is likely to be for more efficient bandwidth scenarios, such as protobuf (Google.Protobuf or protobuf-net) etc, but could also be to help match pre-existing serialization choices. While manually registering a specific IHybridCacheSerializer<Foo> should work, it is also intended to generalize the problem of serializer selection, via an ordered set of serializer factories, specifically by registering some number of:

namespace Microsoft.Extensions.Caching.Distributed;

public interface IHybridCacheSerializerFactory
{
    bool TryCreateSerializer<T>([NotNullWhen(true)] out IHybridCacheSerializer<T>? serializer);
}

By default, we will register a specific serializer for string, and a single factory that uses System.Text.Json, however external library implementations are possible, for example:

namespace Microsoft.Extensions.Caching.Distributed;

[SuppressMessage("ApiDesign", "RS0016:Add public types and members to the declared API", Justification = "demo code only")]
public static class ProtobufDistributedCacheServiceExtensions
{
    public static IServiceCollection AddHybridCacheSerializerProtobufNet(this IServiceCollection services)
    {
        ArgumentNullException.ThrowIfNull(services);
        services.AddSingleton<IHybridCacheSerializerFactory, ProtobufNetSerializerFactory>();
        return services;
    }

    private sealed class ProtobufNetSerializerFactory : IHybridCacheSerializerFactory
    {
        public bool TryCreateSerializer<T>([NotNullWhen(true)] out IHybridCacheSerializer<T>? serializer)
        {
            // in real implementation, would use library rules
            if (Attribute.IsDefined(typeof(T), typeof(DataContractAttribute)))
            {
                serializer = new ProtobufNetSerializer<T>();
                return true;
            }
            serializer = null;
            return false;
        }
    }
    internal sealed class ProtobufNetSerializer<T> : IHybridCacheSerializer<T>
    {
        // in real implementation, would use library serializer
        public T Deserialize(ReadOnlySequence<byte> source) => throw new NotImplementedException();

        public void Serialize(T value, IBufferWriter<byte> target) => throw new NotImplementedException();
    }
}

The internal implementation of HybridCache would lookup T as needed, caching locally to prevent constantly using the factory API.

Additional functionality

The internal implementation of HybridCache should also:

• hold the necessary state to serve concurrent requests for the same key from the same incomplete task, similar to the output-cache implementation
• hold the necessary state to support L1/L2 caching
• optionally, support L1 invalidation by a new optional invalidation API

Note that it is this additional state for stampede and L1/L2 scenarios (and the serializer choice, etc) that makes it impractical to provide this feature simply as extension methods on the existing IDistributedCache.

The new invalidation API is anticipated to be something like:

namespace Microsoft.Extensions.Caching.Distributed;

public interface IDistributedCacheInvalidation : IDistributedCache
{
    event Func<string, ValueTask> CacheKeyInvalidated;
}

(the exact shape of this API is still under discussion)

When this is detected, the event would be subscribed to perform L1 cache invalidation from the backend.

Additional things to be explored for HybridCacheOptions:

• options for L1 / L2 caching; perhaps enabled by default if we have IDistributedCacheInvalidation ?
• eager pre-fetch, i.e. "you've asked for X, and the L1 value is still valid, but only just; I'll give you the L1 value, but I'll kick off a fetch against the backend, so there is not a delay when it expires shortly" (disabled by default, due to concerns over lambdas and captured state mutation)
• compression (disabled by default, for simple compatibility with existing data)
• ...?

Additional modules to be enhanced

To validate the feature set, and to provide the richest experience:

• Microsoft.Extensions.Caching.StackExchangeRedis should gain support for IBufferDistributedCache and IDistributedCacheInvalidation - the latter using the "server-assisted client-side caching" feature in Redis
• Microsoft.Extensions.Caching.SqlServer should gain support for IBufferDistributedCache, if this can be gainful re allocatiuons
• guidance should be offered to the Microsoft.Extensions.Caching.Cosmos owners, and if possible: Alachisoft.NCache.OpenSource.SDK

Open issues

• does the approach sound agreeable?
• naming
• where (in terms of packages) does the shared implementation go? in particular, it may need access to System.Text.Json, and possible an L1 implementation ( which could be System.Runtime.Caching, Microsoft.Extensions.Caching.Memory, this new one, or something else) and possibly compression; maybe a new Microsoft.Extensions.Caching.Distributed ? but if so, should it be in-box with .net, or just NuGet? or somewhere else?
• the exact choice of L1 cache (note: this should be an implementation detail; we don't need L1+L2 for MVP)
• how exactly to configure the serializer
• options for eager pre-fetch TTL and enable/disable L1+L2, via TypedDistributedCacheOptions
• should we add tagging support at this juncture?

[aKzenT]

I know it's difficult to change the existing interface, but depending on the other changes planned in this epic, it would be great if we could have distributed cachr methods that were more performance oriented, working with Span instead of byte[].

A common scenario for using the cache is saving serialized objects or large texts encoded as UTF-8. Requiring byte[] usually means copying this data at least once.

Similar issues exist when reading from the cache which also returns a byte[] and thus does not allow for using rented buffers or similar optimizations.

As the cache is often used several times for each request in any non trivial web application (e.g. session store, query cache, response cache), optimizations here would really pay off.

[mgravell]

Yup. Totally understand, @aKzenT , and that's part of the prototype. Will update with more details of the proposed API as it evolves, but the short version here is:

• serialization will be moved inside the cache layer
• pluggable serialization to be based on ReadOnlySequence-byte and IBufferWriter-byte
• new optional backend (store) API to allow buffer usage, but existing byte[] backends wil continue to work (albeit with less efficiency than they could achieve by implementing the new backend API)

[mgravell]

@aKzenT please see updated body

[Tornhoof]

As for L1+L2 caching, you might want to talk to developers of MS FASTER https://github.com/microsoft/FASTER, which has L1+L2 support, while L2 is not strictly out of process, more like out of main memory (disk based or azure based if I remember correctly).
As from my own experience with MS FASTER, it is not necessarily easy to configure properly, but covers a lot of the functionality for L1/L2.

[mgravell]

@Tornhoof aye, FASTER has come up more than a few times; because of the setup etc required, I doubt we can reasonably make that a default implementation; the most pragmatic solution there might be to provide an IDistributedCache (i.e. L2) implementation that is FASTER-based, and leave L1+L2 disabled; that would 100% be something I'd love to see done, but it doesn't need to be critical-path for this epic

[aKzenT]

@mgravell thank you a lot for the update. I'm seeing a lot of things addressed that I've missed in the past, working on multi-server web apps.

For L1+L2 Caching we have been quite happy with https://github.com/ZiggyCreatures/FusionCache in the past which is also built upon IDistributedCache. I remember that there were some features missing from IDistributedCache that made it hard to implement advanced scenarios in that library. So I would like to invite @jodydonetti to this discussion as well as he can probably best comment on these issues.

One thing I remember that was missing was being able to modify the cache entry options (i.e. life time) of a cache entry without going through a Get/Set cycle. Being able to modify the lifetime allows for some advanced scenarios like invalidating a cache entry from the client (e.g. you received a webhook notifying you about changes in data) or reducing the time to allow things like stale results.

Another thing related to cache invalidation, that is not really possible with the current API in an efficient way, is the removal/invalidation of a group of related cache entries. Let's say you have a cache for pages of a CMS system with each page being one entry. The CMS informs you about changes via a web hook, which invalidates the cache for all pages. Directly refreshing all pages might be expensive, so you would rather refresh them individually on demand. So you want to invalidate all page entries in the cache, but there is no way to get the list of entries from the cache, nor is there a good way to delete the entries. Our solution was to built this functionality ourself using a Redis Set that manages the related keys and then iterating through these keys and removing them one by one. But it felt very hacky as you cannot even use the same Redis Connection that the distributed cache uses, as far as I remember.

[mgravell]

@aKzenT re individual cache invalidation: there is a remove API that is meant to serve that function, but it doesn't allow modify of the options; I'd love to understand the need there further

re group cache invalidations: that sounds a lot like the "tag" feature of output-cache, i.e. you associate entries with zero, one or more tags, and then you can invalidate an entire tag, which nukes everything associated; the problem is: that tracking still needs to go somewhere, and it isn't necessarily an inbuilt feature of the cache backend - it was a PITA to implement reasonably on redis without breaking the cluster distribution, for example (ask me how I know!). It also isn't something that can fit in the existing IDistributedCache backend without significant work. Maybe there is some capacity there if we simplified to "zero or one" tags, but even then... I'm not sure that is something we can tackle in this epic, but I'm open to being wrong there!

[mgravell]

Re FusionCache: that isn't one I've seen before, but glancing at the homepage, I see that the fundamental design is pretty similar (some differences, but: very comparable). There is a common theme in these things - indeed, a lot of inspiration (not actual code) in the proposal draws on another implementation of the same that we had when I was at Stack Overflow (in that version, we also had some Roslyn analyzers which complained about inappropriate captured / ambient usage - very neat!). My point: lots of approaches converging around the same API shape.

[aKzenT]

@mgravell we had the same experience implementing our invalidation logic for a redis cluster setup. It's really hard to get right. I would not expect the design to provide a complete solution to this issue, but maybe there is some way that would make it possible for other libraries to support that while building on top of IDistributedCache. Maybe @jodydonetti has an idea how that could work.

As for modifying the options, in the case of FusionCache there is the possibility to allow stale entries, which are treated as expired, but still available in case the backend is not available. For these cases there is a separate timeout of how long you want to allow a result being stale. The TTL that is sent to the underlying IDistributedCache is then the initial TTL plus the additional stale timeout. So when you invalidate an entry, but still want to allow stale results, you cannot simply delete the entry. Instead you would want to update the timeout to being equal to the stale timeout. Hope that makes sense.

[mgravell]

Yep, very familiar with the idea - it is conceptually related to the "eager pre-fetch" mentioned above - with two different TTLs with slightly different meanings

[mgravell]

via twitter:

Yes, it does lack proper API there is too much redundant code we are required to write or maintain a library on our end I used the DistributedCache code snippet provided in one of the .NET blog post by you and it is definitely going to be nice to have these features.

This is referring to this repo, which offered some of the highlights of this proposal as extension methods, but without any of the "meat" that drives the additional functionality.

[aKzenT]

One thing I'm wondering is, if the choice to put the generic type parameter on the interface rather than the methods might be limitting in some cases and would require some classes to have to configure and inject multiple IDistributedCache instances. I'm not sure if that is really a problem, but it would be nice to learn, why you went for that choice, which differs from other implementations that I have seen.

[mgravell]

@aKzenT fair question, and it isn't set in stone, but - reasons:

1. to allow the serializer to be injected, although that might also be possible by taking IServiceProvider
2. to allow the callback signature to be inferred in both the stateless and stateful case, although this might also be possible with a <,> method

Tell you what, I'll branch my branch and try it the other way. Although then I need a new name... dammit!

[mgravell]

@aKzenT ^^^ isn't terrible; will add notes above - we can probably go that way; I probably hadn't accounted for the improvements in generic handling in the last few C# versions (especially with delegates); in particular, I didn't have to change the usage code at all - see L107 for entirety of the usage update

I will benchmark the perf of GetService() as it applies here, but to be honest: in any reasonable L1 cache-hit scenario, I would expect the overall performance to be dominated by the deserialize; and in any L2 or backend scenario, we should anticipate the GetService() to be a rounding error compared to the L2/backend work (if it isn't: why are we caching?)

[danielmarbach]

Would it be a option to use change token for the cache key invalidation instead of the event type proposed currently?

[mgravell]

@danielmarbach in reality, I'm not sure that is feasible in this scenario; the example uses shown for that API seem pretty low-volume and low issuance frequency; file config changes etc, but if we start talking about cache: we're going to need as many change tokens as we have cached entries, and crucially: the backend layer would need to know about them; I'm mentally comparing that to how redis change notifications can work, and to do that efficiently: we don't want to store anything extra, especially at all the backend layers. Given that all we actually want/need here is the string, this seems like going a very long way out of shape, to reuse an API for the sake of it. Happy to keep considering alternatives if I'm missing something, though! Please keep it coming, that's the entire point of this!

[aKzenT]

@danielmarbach I believe it should be fairly easy to implement a GetChangeToken(string key) method as an extension method that subscribes to the event and listens to changes to the specific key. The other way arround is harder. That being said, I'm a fan of ChangeTokens and IIRC, MemoryCache uses change tokens to invalidate entries, so there might be some value to provide such an extension method directly in the framework in addition to the event @mgravell .

[normj]

To add to your list of "Current Code Layout" we have an implementation at AWS for DynamoDB as backend. https://github.com/awslabs/aws-dotnet-distributed-cache-provider

[aKzenT]

@aKzenT ^^^ isn't terrible; will add notes above - we can probably go that way; I probably hadn't accounted for the improvements in generic handling in the last few C# versions (especially with delegates); in particular, I didn't have to change the usage code at all - see L107 for entirety of the usage update

I will benchmark the perf of GetService() as it applies here, but to be honest: in any reasonable L1 cache-hit scenario, I would expect the overall performance to be dominated by the deserialize; and in any L2 or backend scenario, we should anticipate the GetService() to be a rounding error compared to the L2/backend work (if it isn't: why are we caching?)

If you want to go that route, is there a particular reason why you want ICacheSerializer to be generic instead of just its methods being generic? I feel like for the basic scenario of System.Text.Json, this can just be a singleton. If someone needs to differentiate serialization by type it should be easy to do using a composite style pattern.

Instead of configuring caching and serialization by type, I would rather have the ability to have named instances that I can configure in addition to a default instance, similar to how HttpClientFactory works. FusionCache allows this by the way, if you are interested in an example.

[aKzenT]

Instead of configuring caching and serialization by type, I would rather have the ability to have named instances that I can configure in addition to a default instance, similar to how HttpClientFactory works. FusionCache allows this by the way, if you are interested in an example.

Keyed Services might make that easy. Although I'm not sure if there is an easy way to register both a keyed cache and a keyed serializer and tell the DI system that it should use the serializer with the same key when injecting into the cache.

[niemyjski]

I never liked the name of it to start with because I might want an in memory, hybrid or distributed cache and the abstraction doesn't change. I also don't like how expiration / expiration strategy is not present (possibly it is with extension methods, I haven't looked in a long time). I also don't feel very strongly about adding generics to the interface because a cache can store all different kinds of shapes. We probably should slim down our default interface, but this is what we've found useful over the past decade: https://github.com/FoundatioFx/Foundatio#caching

[mgravell]

To add to your list of "Current Code Layout" we have an implementation at AWS for DynamoDB as backend. https://github.com/awslabs/aws-dotnet-distributed-cache-provider

Great to know, thanks! The list was never meant to be exhaustive - I'll add and clarify. The key takeaway is: we want to actively avoid breaking any implementations; the new functionality should happily work against the existing backend, with them using the additional / optional extensions if appropriate.

[mgravell]

With a class level TKey version, yes you might want:

protected abstract string FormatKey(TKey key);
protected abstract ValueTask<TValue> GetValueAsync(TKey key, CancellationToken token);

That's not what we're shipping today, but I'm open to there being a more specialized subclass for highly optimized scenarios. The current version targets a more generalized version that is suitable for simple replacement of IDistributedCache usage. Let's land the generalized version, and in the background I'll investigate options for the more specialized scenario.

[mgravell]

Note second API proposal above for invalidation / tag support: #55308

[aKzenT]

@mgravell I just had the chance to review the API proposal. One thing I'm missing is the ability to have more than one (named) HybridCache. The current API proposal allows mostly configuration based on the type that you want to serialize/deserialize. But I would argue that it's beneficial to have different cache instances based on the caller instead. This is also something that FusionCache supports for example and would match how you typically use things like the HttpClientFactory. I.e. instead of configuring HttpClientFactory based on the endpoint you are trying to reach, you typically have a named factory based on the caller. Maybe the new keyed services functionality could be of help here.

The reasoning behind this is that usually when caching is involved, you are using that to improve/enable a specific feature that is often encapsulated in one or more classes. So for example, if you want to cache entities retrieved from your CMS, you might have a class like EntityService that internally uses the cache. Or if you are using the cache for an output cache, you might have some OutputCacheHandler, etc. Each of these classes usually have their own caching requirement and might need to persist dozens types. With the current model I would need to know and configure each of these types individually instead of configuring a HybridCache instance for the specific feature I'm trying to enable. This of course also easily breaks abstractions. E.g. my OutputCacheHandler might use an internal type to cache/store the output together with some metadata. Now I would need to expose this type in order to be able to configure the caching.

For a nice implementation, see FusionCache: https://github.com/ZiggyCreatures/FusionCache/blob/main/docs/NamedCaches.md

[mgravell]

@aKzenT if the main thing we're worrying about there is serialization types: I don't think that is a problem - separate callers should already be able to independently and separately configure that, without needing to see each other's types. Is there a specific example we can talk through?

[jodydonetti]

@mgravell a couple of things that comes to mind, some related to serialization configuration being strongly binded per-type, some related to other things, but all related to how having clearly separated worlds (read: named caches) is frequently used.

Serialization

First off is wanting different serialization configuration for the same type but in different named caches.
Based on what my community told me it's something they actually do, I mean having a named cache serializing with JSON and another one serializing with Protobuf or whatever, even if they sometimes share the instances of the same type.
For this case don't necessarily think high-level types like Person or Product, which may be serialized only in one way, but things like support classes like MyAppSettings with shared app-specific settings for different classes.

A more low-level but common example: it's common not to create a custom type for common types like string, string[], List<string> etc and users tend to frequently save in the cache things like a List<string> with some settings/values/whatever.
And they'll do this for 2 different "worlds" (meaning, what normally are 2 named caches), where the types for one of them are serialized with JSON and types for the other with Protobuf: so now they will have all their "1st named cache" data serialized in Protobuf except for List<string>s which will be serialized in JSON or something like that.

Here's for an even nastier one to consider, which it also happened to me multiple times: the need to manually specify how to (de)serialize a specific type because of how it's shaped, like a IMyBaseObject where I store a type discriminator as an inner string prop.
So I instruct the cache how to (de)serialize this type with, say, Protobuf (in the case of HybridCache via TryCreateSerializer<T>) and from now on I know it'll know how to handle it. All is good.
Then I want to save in the cache something like MyBagOfData which contains a property of type IMyBaseObject or List<IMyBaseObject>: since MyBagOfData has not been registered with a specific serializer, it will use the default one (JSON via System.Text.Json) but will throw when trying to (de)serialize IMyBaseObject because it doesn't know how to handle it.
As I said, nasty, but still something to consider.

Of course for this you may be tempted to simply say "not supported" or "not supported in v1" and call it a day, but is something to at least think about, even just to prepare for the future version when it may eventually be supported.

Different Configurations

A totally different scenario in support for multiple named caches is different configurations.

A classic example is wanting a different DefaultEntryOptions per named cache.

Another one is different sub-components, like different distributed caches or different tag-based invalidation mechanisms (what in FusionCache is the backplane).

This can be good for different reasons, from smaller to bigger:

• data partitioning: think same Redis instance but different Redis databases (even though this is not possible in cluster mode, it's something I saw used)
• better performance: think different Redis instances to better scale out horizontally
• different technologies: think one cache using Redis and another using Memcached or whatever
• expectations: like mentioned by @aKzenT there are expectations commonly coming from HTTP named clients and friends, where you can have the same results manually handling the low level message handler etc but it's expected to have a better way

Tag-based invalidations and general namespacing

Another example is, since it seems to be part of HybridCache v1, tag-based invalidation.
A user caches Orders tagged "Italy", Foods tagged "Italy" and Persons tagged "Italy" (give some rope here for the example 😅).
Now, if the cache is effectively only one shared mega-cache, a tag-invalidating for "Italy" will wipe any cache entry tagged with "Italy".

Of course an answer may be "just namespace your tags too, like you namespace you cache keys", for example via the common way of using a prefix, but then users will have to manually prefix everything and... that's not great.

Lingua Franca and Integrations

For a FusionCache-HybridCache integration I don't see big problems related to not having multiple named caches in v1. The way I'm imagining the integration would be is something like this:

services.AddFusionCache()
  .WithOptions(...)
  .WithSerializer(...)
  .WithDistributedCache(...)
  .AsHybridCache(...) // THE MAGIC HAPPENS HERE
;

where it will register in DI an HybridCache adapter implementation that underneath uses FusionCache, so that users may setup features like fail-safe, soft timeouts, etc and have them work even when used as an HybridCache.

In this case users, even when directly using multiple named FusionCache instances in their apps, can still setup one of them for HybridCache usages.

For example this will be useful if (as I hope/think it will be the case) HybridCache will become the shared abstraction or lingua franca for different higher level components like output cache, etc (not to add even more pressure here for you fellas 😬, just trying to reasonably look forward).

So something like this:

services.AddFusionCache("MyHybridCacheUsedForOutputCacheOrWhatever")
  .WithOptions(...)
  .WithSerializer(...)
  .WithDistributedCache(...)
  .AsHybridCache(...) // THE MAGIC HAPPENS HERE
;

Does this make sense?

A lot to digest

Now, I know it's a lot to digest and a lot to ask for HybridCache v1 (FusionCache for example did not have multiple named caches for some time), but at least knowing this upfront may allow you and your team to design for a future addition and not have to make either breaking changes down the road or have a badly designed api in the future because you didn't see these things coming.

Hope this helps.

ps: if I forgot something I hope @aKzenT can chip in with the missing pieces!

[aKzenT]

Thank you @jodydonetti . This sums up perfectly what I was thinking. Serialization, namespacing and default options were my main points as well and you articulated them better than I could have and added some additional aspects I did not think of, but agree 100%.

As another edge case, if I develop a library with a class that uses caching internally using MyPrivateData class, how would I configure its serialization behavior as a library consumer without requiring my MyPrivateData to be exposed publically? I might get away with exposing it only internally and then providing DI extension methods to configure that, but that basically prohibits the use of any other DI frameworks.

[jodydonetti]

As another edge case, if I develop a library with a class that uses caching internally using MyPrivateData class, how would I configure its serialization behavior as a library consumer without requiring my MyPrivateData to be exposed publically? I might get away with exposing it only internally and then providing DI extension methods to configure that, but that basically prohibits the use of any other DI frameworks.

I'm not understanding this part: if your library uses internally a cache, why should it be configurable from the outside?
Can you provide a concrete yet concise example?

[mgravell]

Re the DI topic: it'll already want the expected DI APIs to lazily resolve the serializers as it sees the need. But within that: your component can still call AddHybridCache etc and configure serialization - everything should still work correctly, but I'll adds tests for that scenario.

I need to take a moment to digest the multiple cache thing.

[aKzenT]

The HybridCache needs to know how to serialize the types that my library uses. So this means at some point the caller has to directly or indirectly configure the HybridCache to correctly handle (serialize) them.

So imagine I have a DataLoader class that uses HybridCache using a construction parameter and requires serialization of an internal DataCacheEntry. I need to register a matching serializer for that class so that HybridCache knows how to serialize it.

I'm assuming that usually this would happen through an extension method, so that I don't need to be aware of the details as a caller. What I was wondering is, how that scenario would play out if using another DI or when constructing the class manually.

Having said that, after writing everything down, I think the solution here would be for the library to provide a public serializer factory. I missed that before, so ignore my comment.

[mlstubblefield]

If it hadn't been mentioned (didn't see it), I'd change the API such that the base interface is a "multi get" (set based operations).

IE: it takes in keys as a list and then when some of them aren't found, it should run a function that gets the values for those keys and kicks back a dictionary.

Then throw an extension method on top of that to do the single get/add.

That way my fallback can go to the DB and grab N items efficiently.

[jodydonetti]

IE: it takes in keys as a list and then when some of them aren't found, it should run a function that gets the values for those keys and kicks back a dictionary.

I'm not part of the team, or Microsoft in general, but having "been there done that" in the past I'd like to share a couple of issues.

The main one is the fact that when working with one cache entry the input for the cache is the cache key, and not the db item id, whereas the input for the factory (called by the cache) is the item id, and not the cache key.

For example (oversimplified):

var id = 123;
cache.GetOrSet<Product>(
  $"product/{id}", // INPUT FOR THE CACHE: CACHE KEY
  _ => GetFromDb(id) // INPUT FOR THE FACTORY: ITEM ID
);

So the core problem here is one of mapping between the 2 things, cache key and item id.

By then going "multi", you should consider the input for the cache being something like a list of cache keys, then the cache should use that, see what is missing, and then pass the cache keys for the missing entries to the factory, which in turn should decompose the cache keys to extract the item ids.

There are basically 2 paths here that can be followed:

• make the cache learn about how to manipulate the mapping between item ids <-> cache keys (eg: with 2 lambdas, one for id -> key and the other for key -> id)
• have the consumer (eg: you) do it yourself by parsing/splitting the cache keys to get back the item ids

and both ways are generally not that nice.

(just as a note: personally, I'm working on something that would solve this, in what would be imho a correct way, but it takes a different approach and I need a little bit more time for that).

Opinions?

[mgravell]

On the key vs id; this also crosses over into the alloc-free key discussion that I had on the (new cache proposal thing that I've forgotten the name of). If we had *that*, then yes there's some possible future additional piece that is basically a TKey, with a TKey-to-string function somewhere (perhaps even implied via attribute or naming), but I don't think that piece is a "today" thing.

…

On Sat, 25 May 2024, 10:58 Jody Donetti, ***@***.***> wrote: IE: it takes in keys as a list and then when some of them aren't found, it should run a function that gets the values for those keys and kicks back a dictionary. I'm not part of the team, or Microsoft in general, but having *"been there done that"* in the past I'd like to share a couple of issues. The main one is the fact that when working with one cache entry the input for the cache is the *cache key*, and not the db *item id*, whereas the input for the factory (called by the cache) is the *item id*, and not the *cache key*. For example (oversimplified): var id = 123; cache.GetOrSet<Product>( $"product/{id}", // INPUT FOR THE CACHE: CACHE KEY _ => GetFromDb(id) // INPUT FOR THE FACTORY: ITEM ID); So the core problem here is one of mapping between the 2 things, *cache key* and *item id*. By then going "multi", you should consider the input for the cache being something like a list of cache keys, then the cache should use that, see what is missing, and then pass the cache keys for the missing entries to the factory, which in turn should decompose the cache keys to extract the item ids. There are basically 2 paths here that can be followed: - make the cache learn about how to manipulate the mapping between item ids <-> cache keys (eg: with 2 lambdas, one for id -> key and the other for key -> id) - have the consumer (eg: you) do it yourself by parsing/splitting the cache keys to get back the item ids and both ways are generally not that nice. (just as a note: personally, I'm working on something that would solve this, in what would be imho a *correct* way, but it takes a different approach and a I'd need a little bit more time for that). Opinions? — Reply to this email directly, view it on GitHub <#53255 (comment)> or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAEHMHHDYJVSD3JTY3QXSTZEBODZBFKMF2HI4TJMJ2XIZLTSWBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVIYTEOBUGE3TKMBTGSSG4YLNMWUWQYLTL5WGCYTFNSBKK5TBNR2WLKRVGU3TENBTGEZTAM5ENZQW2ZNJNBQXGX3MMFRGK3FMON2WE2TFMN2F65DZOBS2YSLTON2WKQ3PNVWWK3TUUZ2G64DJMNZZJAVEOR4XAZNKOJSXA33TNF2G64TZUV3GC3DVMWUDCNZWGIYDGNBXQKSHI6LQMWSWS43TOVS2K5TBNR2WLKRSGA3TGMRXGE4TMMECUR2HS4DFUVWGCYTFNSSXMYLMOVS2UMJSHA2DCNZVGAZTJAVEOR4XAZNFNRQWEZLMUV3GC3DVMWVDKNJXGI2DGMJTGAZ2O5DSNFTWOZLSUZRXEZLBORSQ> . You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .

[jrgcubano]

I tend to use extensions like the following example in some scenarios like the one described by @mlstubblefield and @jodydonetti. But I only use it with in-memory caches... Does this flow make sense?

public static class MemoryCacheExtensions
{
    public static async Task<IReadOnlyCollection<TResult>> GetOrAddToCache<TResolverArg, TResult>(
        this IMemoryCache cache,
        IReadOnlyCollection<string> ids,
        Func<TResult, string> itemIdSelector,
        Func<string, TResolverArg, string> cacheKeyResolver,
        Func<IReadOnlyCollection<string>, TResolverArg, Task<IReadOnlyCollection<TResult>>> dataResolver,
        TResolverArg resolverArg,
        TimeSpan expiryInMinutes)
    {
        var (fromCache, toRetrieve) = cache.GetDataByIdsFromCache<TResolverArg, TResult>(ids, cacheKeyResolver, resolverArg);
        if (toRetrieve.Count == 0)
            return fromCache;

        var fromIo = await dataResolver(toRetrieve, resolverArg);
        if (fromIo == null)
            return fromCache;

        cache.AddToCache(fromIo, itemIdSelector, cacheKeyResolver, resolverArg, expiryInMinutes);

        foreach (var item in fromIo)
        {
            fromCache.Add(item);
        }

        return fromCache;
    }

    static void AddToCache<TResolverArg, TResult>(
        this IMemoryCache cache,
        IEnumerable<TResult> fromIo,
        Func<TResult, string> itemIdSelector,
        Func<string, TResolverArg, string> cacheKeyResolver,
        TResolverArg resolverArg,
        TimeSpan expiryInMinutes)
    {
        foreach (var item in fromIo)
        {
            cache.Set(
                key: cacheKeyResolver(itemIdSelector(item), resolverArg),
                value: item,
                expiryInMinutes);
        }
    }

    static (List<TData> data, List<string> remainder) GetDataByIdsFromCache<TResolverArg, TData>(
        this IMemoryCache cache,
        IEnumerable<string> ids,
        Func<string, TResolverArg, string> cacheKeyResolver,
        TResolverArg resolverArg)
    {
        var fromCache = new List<TData>();
        var toRetrieve = new List<string>();

        foreach (var id in ids)
        {
            if (cache.TryGetValue(cacheKeyResolver(id, resolverArg), out var data))
            {
                fromCache.Add((TData)data);
                continue;
            }

            toRetrieve.Add(id);
        }

        return (fromCache, toRetrieve);
    }
}

Example:

...
var resolverArg = (
   repository: _repository,
   cachePrefix: 'product', 
   cancellationToken);

var entities = await _memoryCache.GetOrAddToCache(
	ids: query.Ids,
    itemIdSelector: static item => item.Id,
    cacheKeyResolver: static (id, rArg) => $"{rArg.cachePrefix}:{rArg.product.Id}"
    dataResolver: static (toRetrieve, rArg) => FetchFromDb(rArg, toRetrieve, rArg.cancellationToken),
    resolverArg,
    5);

For a distributed version with L1 and L2, it would be ideal to be able to fetch and set all the keys in a single operation...

[jodydonetti]

Preview 4 Notes

Hi @mgravell and team, I played with the preview 4 bits and here are my notes.
They are a mix of 2 things:

• general notes on HybridCache itself, eg: using it as a normal user
• notes about implementing it with FusionCache, eg: using HybridCache as a sort of lingua franca, a shared abstraction that 3rd party packages may depend on (see: OutputCache, etc) while still being able to use different implementations that may have more/different features or scenarios covered

NOTE: I will not touch on things that I know will already change, like RemoveByKey becoming Remove, etc.

At First Glance

I see that there's a SetAsync method but still not a GetAsync method, currently: my first thought (been there, done that etc) is because you may not still be comfortable with exposing the internal "entry" class? If that is the case I would suggest a TryGetAsync method instead, which would still be able to clearly differentiate between null and "no value was there". It requires the definition of a kinda generic "maybe monad" type (in FusionCache it is MaybeValue<T> for example) and considering it's a MS thing maybe would prefer to wait for an "official" general thing. Or maybe I just got it wrong, I don't know 😅

Anyway I'll wait for preview 5 for that.

Regarding the HybridCacheEntryOptions, I see that the way you seem to have made it is that:

• if null is passed, it will use the DefaultEntryOptions as a fallback, if defined, or a somewhat even global-er fallback if even that is not defined. I understand the rationale, but I would suggest for the DefaultEntryOptions on the cache to be defined forcibly instead, as it would remove a lot of issues down the road (just my 2 cents ofc)
• any null property inside of a passed entry options object will fall back to that property's default value on the default entry oprions (or related fallback, see above). Again, I see the rationale, but it can be too limiting: for example in case of props where null is a valid value, and you'll not be able to differentiate between null meaning "fall back to the default" and "I really want null here". Again, just my 2 cents, but my suggestion is to think about this really really carefully, and try to apply Second Order Thinking on this, but like, really hard

As pointed out by some in the video by @Elfocrash , the auto-magic usage of any registered IDistributedCache service in the DI container will:

1. surprise some (because it may have been registered for some other reasons, including the default MemoryDistributedCache one registered by aspnet core automatically)
2. will not be controllable by the user. I would suggest taking the same path as FusionCache (or a similar one anyway) where the user needs to explicitly specify to use one, even if the registered one (in FusionCache there's a method in the builder called WithRegisteredDistributedCache() for example).

Something else that's not clear is what LocalCacheExpiration does or how it works: in the xml docs it's stated that "Cache duration in local cache; when retrieving a cached value from an external cache store, this value will be used to calculate the local cache expiration, not exceeding the remaining overall cache lifetime" and I don't understand if it will be somewhat "overwritten" by some internal logic when getting data from L2, or what. In my tests (again, with preview 4, so things may have already changed...) I saw that with only L1, by setting LocalCacheExpiration I get the expiration I want, whereas by setting Expiration and not setting LocalCacheExpiration I was having surprises. Maybe this is related to one of my points about, meaning in this case it's not possible, by setting LocalCacheExpiration to null, to differentiate between "I don't want to specialize the local cache expiration, just use the normal Expiration" and "for LocalCacheExpiration I want to fall back to a default value". This is subtle but, I'm warning you, very important. Like, very very.

Finally I see that the "by tag" logic is there, all the time: I get that it is desired (and I mean, I really really get it, it would be an awesome feature, and I'm playing with it in FusionCache for some time now already) but, since it is reasonably tied to the distributed backend used, I'm wondering about 2 things:

1. how would existing (and future) implementations of IDistributedCache communicate to HybridCache that they support tagging, or not?
2. what should they do if they do not support that (eg: any one of current SqlServer, CosmosDB, SQLite, etc)? Throw when tags has values? Ignore it? Are there guidelines?
3. how can users of HybridCache check that the current instance actually supports tagging? Is there a public readonly bool prop or something? For example in FusionCache I'm not exposing the underlying IDistributedCache instance, but I'm exposing a bool HasDistributedCache prop to maybe do feature detection, so a similar rational can be applied here
4. how would HybridCache itself knows what to do? Since it's not tied to a specific cache backend (eg: Redis, Memcached, etc) and since there's no concept of a backplane or similar like in FusionCache, what are your thoughts about this? If you can share, I'd like to know

Honestly, I can see the difficulty in creating an API surface area that allows that but also allows to check for that, but still, I'd like to point that out in this first preview.

HybridCache implemented on top of FusionCache

I've been able to reasonably implement very quickly a FusionCache-based adapter adapter for HybridCache, so kudos for how easy it has been.

Right now in the example project I'm able to register the normal HybridCache with this:

// ADD NORMAL HYBRIDCACHE
services.AddHybridCache(o =>
{
	o.DefaultEntryOptions = new HybridCacheEntryOptions
	{
		LocalCacheExpiration = defaultDuration
	};
});

and a FusionCache-based one with this:

// ADD FUSIONCACHE, ALSO USABLE AS HYBRIDCACHE
services.AddFusionCache()
  .AsHybridCache() // MAGIC
  .WithDefaultEntryOptions(o =>
  {
    o.Duration = defaultDuration;
  });

The "magic" part is basically registering a FusionCache-based adapter as an HybridCache service, and the rest of the code is like this:

// THIS WILL WORK BOTH WITH THE DEFAULT HybridCache IMPLEMENTATION
// AND THE FusionCache BASED ONE
var cache = serviceProvider.GetRequiredService<HybridCache>();

Console.WriteLine($"TYPE: {cache.GetType().FullName}");

const string key = "test key";

// TAGS: NOT CURRENTLY SUPPORTED
//await cache.SetAsync("fsdfsdfds", "01", tags: ["", ""]);

// SET
await cache.SetAsync(key, "01");
var foo = await cache.GetOrCreateAsync<string>(
	key,
	async _ => "02"
);
Console.WriteLine($"FOO: {foo} (SHOULD BE 01)");

// REMOVE
await cache.RemoveKeyAsync(key);
foo = await cache.GetOrCreateAsync<string>(
	key,
	async _ => "03"
);
Console.WriteLine($"FOO: {foo} (SHOULD BE 03)");

// NO-OP
foo = await cache.GetOrCreateAsync<string>(
	key,
	async _ => "04"
);
Console.WriteLine($"FOO: {foo} (SHOULD BE 03)");

// LET IT EXPIRE
Console.WriteLine("WAITING FOR EXPIRATION...");
await Task.Delay(defaultDuration);

// SET
foo = await cache.GetOrCreateAsync<string>(
	key,
	async _ => "05"
);
Console.WriteLine($"FOO: {foo} (SHOULD BE 05)");

Running this with both the registrations in turn gives me this first:

TYPE: Microsoft.Extensions.Caching.Hybrid.Internal.DefaultHybridCache
FOO: 01 (SHOULD BE 01)
FOO: 03 (SHOULD BE 03)
FOO: 03 (SHOULD BE 03)
WAITING FOR EXPIRATION...
FOO: 05 (SHOULD BE 05)

and this second:

TYPE: ZiggyCreatures.Caching.Fusion.MicrosoftHybridCache.FusionCacheAdapterForHybridCache
FOO: 01 (SHOULD BE 01)
FOO: 03 (SHOULD BE 03)
FOO: 03 (SHOULD BE 03)
WAITING FOR EXPIRATION...
FOO: 05 (SHOULD BE 05)

Of course right now I'm talking without tagging support, for which I'm currently throwing an exception (because of the reasons I'll highlight soon in the other issue).

Apart from that I think it may be useful to have a method (either protected virtual or public virtual) that allows to "resolve" a single HybridCacheEntryOptions object passed in (including null) to what it will be used in the end, meaning fallbacks and similar.

Currently I'm applying some logic here, simulating what you have described in your specs, but having some cache instance method where you can pass an HybridCacheEntryOptions object (again including null) that will give you back a "normalized" one with all the defaults applied based on whatever logic you'll decide to apply is fundamental, otherwise 3rd party implementers like me will have to manually implement the same logic, again and again.
Of course the problems highlighted above about different meaning for null for props still aply.

Another thing I noticed is that currently it's possible to override the GetOrCreate method that accepts a state, but not the other one without state: from my understanding, this is because the one without state basically calls the other with state but with a logical "null" one.
Again, I understand the rationale, but for libs that do not currently supports explicit state in the methods this means a double closure allocation, because the one with explicit state would call the one without explicit state which would call the one with explicit state. If there is not a hard reason to disallow the override I would suggest allowing it, even though probably not many will do it (but it would still be possible).
PS: I'm working on adding support for explicit state overrides for FusionCache, so I don't know if I will need this, I'm thinking more generally.

I see that the namespace/packages in preview 4 are "wrong" long term: if you are wondering about types forwardings and similar, my suggestion is to just "destroy everything" in the next preview. This is exactly what previews are for, and (again, personally) I'd like to update the packages, see that I'm not able to compile anymore, and know exactly what needs to be changed to allow it to compile again.
It definitely feels better and, in the long run, it pays dividends (again, just my 2 cents).

For now I think it's all, thanks for sharing the update.

Hope this helps.

[mgravell]

Great feedback. I will ponder!

Minor note: there is no additional closure allocation: we pass the stateless callback as the TState, and invoke it from there via a static lambda. This trick is also available to anyone wrapping the library.

[jodydonetti]

Minor note: there is no additional closure allocation: we pass the stateless callback as the TState, and invoke it from there via a static lambda. This trick is also available to anyone wrapping the library.

Oh, that's neat, I didn't notice it!

[HaikAsatryan]

If my application has 2 instances and 1st instance updates record in L1 + L2, will 2nd instance L1 get the updated record as well with some pub/sub magic?

[michael-wolfenden]

Hi @mgravell and team,

LazyCache supports setting the expiry of the cache entry based on the data being cached.

For example:

cache.GetOrAdd("token", async entry => {
    var token = await GetAccessToken();
    // can set expiry date using a DateTimeOffset (or a TimeSpan with RelativeToNow variant)
    entry.AbsoluteExpiration = token.Expires;
    return token;
});

Does HybridCache support this scenario?

[mgravell]

@michael-wolfenden no, the API as implemented doesn't support that option; such options are passed in as arguments, so are fixed at the time of call.

[michael-wolfenden]

Is the team considered adding similar functionality? The api as is stands now doesn't allow the cache time to be set dynamically based on the data being returned which is a common scenario when caching data from a third party where they control the data's lifetime.

[mgravell]

Output-cache already has some inbuilt functionality similar to stampede protection, which I believe (I can check on Monday) is enabled by default. I don't know without checking whether YARP uses output cache. Can you clarify exactly what you are looking for here, so I can check more?

…

On Sat, 29 Jun 2024, 20:48 Tal Wald, ***@***.***> wrote: Hi @mgravell <https://github.com/mgravell>! We are looking to add YARP as a reverse proxy and want the out of the box stampede protection that HybridCache provides. What is the equivalent of UseOutputCache for HybridCache? — Reply to this email directly, view it on GitHub <#53255 (comment)> or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAEHMAAC4NON44IVNRH3FTZJ4FSVBFKMF2HI4TJMJ2XIZLTSWBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVIYTEOBUGE3TKMBTGSSG4YLNMWUWQYLTL5WGCYTFNSBKK5TBNR2WLKRVGU3TENBTGEZTAM5ENZQW2ZNJNBQXGX3MMFRGK3FMON2WE2TFMN2F65DZOBS2YSLTON2WKQ3PNVWWK3TUUZ2G64DJMNZZJAVEOR4XAZNKOJSXA33TNF2G64TZUV3GC3DVMWUDCNZWGIYDGNBXQKSHI6LQMWSWS43TOVS2K5TBNR2WLKRSGA3TGMRXGE4TMMECUR2HS4DFUVWGCYTFNSSXMYLMOVS2UMJSHA2DCNZVGAZTJAVEOR4XAZNFNRQWEZLMUV3GC3DVMWVDKNJXGI2DGMJTGAZ2O5DSNFTWOZLSUZRXEZLBORSQ> . You are receiving this email because you were mentioned. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .

[vvadyak]

@mgravell and @jodydonetti, I believe combining your efforts will result in a best-in-class cache abstraction

[lindeberg]

Are hit/miss metrics available? How?

[mgravell]

Currently: not; working on it via .net metrics

[stevejgordon]

@mgravell I quickly looked at using HybridCache in a sample app as I was interested in the tiered support and low allocation APIs. Forgive me if it's a very naive question, but the lack of a GetAsync method stumped me for my scenario. I see that it was at one point in the proposal. In the code I'm playing with, I want to access the value from the cache (without creating it). If it's present, I'll handle the case where it's not, but I don't want that to trigger the caching of the result automatically.

In a related scenario, is there a pattern for when the factory can't produce the data (external service is offline), so all we have is some default value we fall back to, such as a static Empty instance or null? For example, if I run this:

var result = await _hybridCache.GetOrCreateAsync("something", _ => ValueTask.FromResult<string?>(null));
result = await _hybridCache.GetOrCreateAsync("something", _ => ValueTask.FromResult<string?>(null));

The factory in the second call won't be invoked as null has been cached.

Ideally, I need a way to signal that the creation failed and not store anything under the key. If there were a way to do this, that might then provide a workaround to the first problem, as I can do GetOrCreate but signal that I don't want anything actually stored in the cache when the entry does not exist.

Another scenario I've yet to explore is where my "expensive" async calls to external systems that I use to build up some data that I'd like to cache, also provide some extra info I need later in my method but that I don't want cached. I'm guessing, but have yet to try, that using the state might be a way to do that but I'm curious if it's a pattern you expect to see used?

[mgravell]

Will do example of the first tomorrow.

On the latter: if you need that value, what do you expect to use on the anticipated case when the value does come from cache? You say you need it, but if it isn't stored: where does it come from?

I guess philosophically it should usually be preferable to have things work the same whether or not the callback was invoked this time, but yes you can always mutate ambient state in either the TState sense or via closures.

Note that there's also a category of cases where the callback was invoked for the active result, but not by you - i.e. a stampede scenario where you're merely an observer.

[stevejgordon]

@mgravell, thanks. This morning, with a fresh eye, I just discovered the flags that seemed to give me the control I needed to avoid the set.

var result = await _hybridCache.GetOrCreateAsync("something", _ => ValueTask.FromResult<string?>(null), 
    new() { Flags = HybridCacheEntryFlags.DisableLocalCacheWrite | HybridCacheEntryFlags.DisableDistributedCacheWrite });

result = await _hybridCache.GetOrCreateAsync("something", _ => ValueTask.FromResult<string?>(null));

The main reason I was looking at this was akin to the FusionCache adaptive caching, where, based on the outcome of the factory, I may want not to cache (or choose a very short cache time) when the external service couldn't give me the value I needed. In that case, I may present an error to the end user (for the current request); for a subsequent request, it tries again to get a value. Using a short empty cached value might be helpful for DDos or general outages to save hitting a struggling backend for a while. However, I may also get that behaviour from Polly and the HTTP resiliency stuff.

If a null/empty value is cached for the configured expiration time, say, 30 mins, that's not ideal. That said, if the cache returns empty, I can still trigger a fetch of the data and manually update the cache. So I think, I see a way to achieve this.