[FEATURE] Limit Memory Size

[smthbh]

Problem

There isn't a way to set a memory limit for the cache. Without this, it's possible that memory usage can grow without bound and crash the system.

Solution

Ideally, fusion cache would have an option to set the size of memory usage and prevent growing beyond this limit.

Additional context

Microsoft's docs on memory caching size limits are here: https://learn.microsoft.com/en-us/aspnet/core/performance/caching/memory?view=aspnetcore-7.0#use-setsize-size-and-sizelimit-to-limit-cache-size

[seantleonard]

I've seen generic guidance from those Microsoft Docs and StackOverflow that determining cache size is up to the developer creating the cache entries. There are historic GitHub issues on .net's repo by the maintainers saying they moved away from directly calculating memory size of cache entries due to inaccurance/complexity:
dotnet/runtime#48567 (comment)

I don't think this matters. The whole point of this implementation is to be a lightweight concurrent dictionary with expiration that doesn't try to respond to memory pressure and start kicking out entries. That's the only reason I see to be concerned about having a single cache. The for it to have a global process wide view of the state. That is already not the case today with .NET Core. Applications have different caches with different policies. I think this is no different.

A cache entry's size is currently a unitless/arbitrary number supplied when adding entries to the cache. I can, for example, try to estimate determining the size of a cache entry by calculating the size of the string (char count * bytes [and null terminator]) and supply that with the 'size' in the cache entry options for FusionCache. An example of this size estimation can be found in Asp.Net's own implementation of ResponseCacheMiddleware's CacheEntryHelpers https://github.com/dotnet/aspnetcore/blob/main/src/Middleware/ResponseCaching/src/CacheEntry/CacheEntryHelpers.cs

One question I was about to open up an issue for, directly related to this, is how to provide a size value for the result returned by the factory method provided to fusionCache.GetOrSet()
And also any guidance for how FusionCache responds to memory pressure or why we shouldn't be concerned with it.

[jasenf]

I'm not quite sure if FusionCache does this, but for most in-memory cache's they are simply storing a reference to the object, they aren't wasting time doing any serialization, so there is no way to estimate object size.

If you are serializing to some kind of string or byte array, then this could be accomplished easily, but you would never really want that for the in-memory cache.

Best you could do is add a parameter and let the implementor send over the estimated size of an object. Maybe support a default extension method like .GetSizeForFusionCache() that the cache looks for and if the object implements it let it calculate itself.

[jodydonetti]

Hi @smthbh and thanks for using FusionCache!

If you are talking about the size of the memory cache (either because you are not using the distributed cache or because you are interested only about that part), it is already possible: since FusionCache works on top of IMemoryCache and IDistributedCache, you can configure them however you want.

Normally, when creating a FusionCache instance, if you don't pass anything a new MemoryCache instance will be created for you, to be used as the 1st level (memory): you can however specify your own instance of MemoryCache configured however you want. If you are using the builder you can simply use WithMemoryCache(...) and pass it your instance, and in turn create it with the MemoryCacheOptions you prefer, including the SizeLimit. Then, when working with FusionCache, you can specify a Size for each entry just like you already can do with MemoryCache, and all will work.

Hope this helps!

[jodydonetti]

Hi @seantleonard

A cache entry's size [...] and supply that with the 'size' in the cache entry options for FusionCache.

Exactly!

One question I was about to open up an issue for, directly related to this, is how to provide a size value for the result returned by the factory method provided to fusionCache.GetOrSet()

Good question, and the answer is Adaptive Caching, which is fancy name for when you change some of the entry's options inside of the factory, while it is running.

Let me know if this has helped you.

[jodydonetti]

Hi @jasenf

I'm not quite sure if FusionCache does this, but for most in-memory cache's they are simply storing a reference to the object, they aren't wasting time doing any serialization, so there is no way to estimate object size.

Yep, totally: serialization only happens when talking to the 2nd level (via the IDistributedCache interface). When the distributed level is not involved, no serialization at all.

If you are serializing to some kind of string or byte array, then this could be accomplished easily, but you would never really want that for the in-memory cache.

Agree.

[smthbh]

Hi @smthbh and thanks for using FusionCache!

If you are talking about the size of the memory cache (either because you are not using the distributed cache or because you are interested only about that part), it is already possible: since FusionCache works on top of IMemoryCache and IDistributedCache, you can configure them however you want.

Normally, when creating a FusionCache instance, if you don't pass anything a new MemoryCache instance will be created for you, to be used as the 1st level (memory): you can however specify your own instance of MemoryCache configured however you want. If you are using the builder you can simply use WithMemoryCache(...) and pass it your instance, and in turn create it with the MemoryCacheOptions you prefer, including the SizeLimit. Then, when working with FusionCache, you can specify a Size for each entry just like you already can do with MemoryCache, and all will work.

Hope this helps!

That makes sense, thanks!