HybridCache : implement the tag expiration feature #5785

mgravell · 2025-01-09T17:47:54Z

(note: this is a re-do of #5748 / #5781 which went "a bit wrong" when rebasing)

Implement tag expiration in HybridCache

The HybridCache system has a "tags" feature; rather than building a bespoke per-backend secondary index (like how the redis OutputCache backend works), here we move everything into the core library:

in the DefaultHybridCache, we maintain a map of tags (string, note * means everything) to expirations, and we track the creation time against each CacheItem
when fetching items from L1 (in-proc) cache, we check the item's creation data against the cache's wildcard and per-tag expirations; if the creation date is earlier than any of these: it is treated as disallowed
to allow persistence of invalidation between sessions when a L2 (backend cache) is in play, we additionally store the timeout data via additional simple values in the cache
we also pre-fetch any missing tag data from L2 as-needed, and enforce the previous expirations
to facilitate this, and allow reliability that the data we are parsing is the data requested: the data stored in L2 is now embedded inside a payload that includes the creation timestamp, duration, key, tags, etc as well as the payload
in the rare event that the L2 data contains additional tags to those advertised by the caller: these additional tags are also fetched and enforced

Because the tag fetches are async, the data in the lookup is not (timestamp), it is Task<(timestamp)>

Outstanding:

additional logging
- add log when L2 data is rejected for (reason)
- add metric of the number of tag expirations being tracked in L1
resolve netfx test failure (actually: niche timing problem in tag invalidation)
address option member inheritance (or not - per @jodydonetti comments here)

Microsoft Reviewers: Open in CodeFlow

jodydonetti · 2025-01-09T17:54:26Z

Note

Reposting my comment here from there for ease of discussion.

Hi Marc, not strictly related to tagging, but I'm looking at some code change in the PR and I spotted this xml comment regarding HybridCacheEntryOptions:

If options are specified at the individual call level, the non-null values are merged
(with the per-call options being used in preference to the global options). If no value is
specified for a given option (globally or per-call), the implementation can choose a reasonable default.

When we touched on this subject some time ago I remember we agreed on the fact that inheritance (from per-call up to the DefaultEntryOptions) would have to be done on an all-or-nothing approach, and not on a per-single-option approach.
What I mean is that "if null is passed as an HybridCacheEntryOptions param" than it would fall back to the DefaultEntryOptions, but single options inside that object (like Expiration, LocalCacheExpiration) should not have inheritance of values.

All of this was because of the impossibility to express "undefined" or, to better say it, the impossibility to differentiate between null meaning "I'm saying null because I want a local fallback" (for example from LocalCacheExpiration to Expiration) and null meaning "I'm saying null because I want a global fallback" (for example to the same option in the DefaultEntryOptions").

A practical example of a problem this may create: in the DefaultEntryOptions I set Expiration to 5min and LocalCacheExpiration to 10s to refresh L1 more frequently. Then in a certain call I pass a specific HybridCacheEntryOptions instance where I set Expiration to 1h and leave LocalCacheExpiration to null, thinking it will be the same as Expiration, when in reality it will become 10s. Because of this, there will be for me no way to keep the 2 options in-sync at every call, unless I keep specifying the same value for both Expiration and LocalCacheExpiration at every call.

Can you clarify what is the current rationale here?

I'm asking for 2 reasons: first I'm interested in general, and second because as you know I'm creating a 3rd party implementation, and I'd like the observable external behaviour to be the same for both implementations.

Thanks!

ps: if you like I can open a different issue for this, just let me know.
pps: will post more about the tagging implementation as soon as I'll finish reading the massive code changes.

dotnet-comment-bot · 2025-01-09T18:22:05Z

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.Extensions.Caching.Hybrid	Line	86	84.14 🔻
Microsoft.Extensions.Caching.Hybrid	Branch	86	84.49 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Extensions.AI	88	89

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=912260&view=codecoverage-tab

jodydonetti · 2025-01-09T23:33:30Z

An extra point to think about, again related the entry options inheritance thing, is how this method (marked internal, and maybe it will not be used in the end) would be affected.

BrennanConroy · 2025-01-09T23:51:00Z

to allow persistence of invalidation between sessions when a L2 (backend cache) is in play, we additionally store the timeout data via additional simple values in the cache

Shouldn't we be concerned about machine clocks not being synchronized? i.e. Machine 1 says it's 10:05:03 and adds a timeout for 10:10:03, but Machine 2 says it's 10:12:05 and when it reads the cache item it will determine it to be invalid.

jodydonetti · 2025-01-10T00:43:59Z

Shouldn't we be concerned about machine clocks not being synchronized? i.e. Machine 1 says it's 10:05:03 and adds a timeout for 10:10:03, but Machine 2 says it's 10:12:05 and when it reads the cache item it will determine it to be invalid.

Eh, welcome to the world of clock shifting/drifting/skewing & friends: it's not something that's solvable via software, at least not at this level (to the best of my knowledge).

Just to make an example: typically Microsoft/Meta/Google/AWS have their own private atomic clocks around the world to avoid this problem (... as much as possible), usually dedicated to specific services (like Google Spanner with TrueTime).

Some interesting reads:

dotnet-comment-bot · 2025-01-10T16:43:31Z

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.Extensions.Caching.Hybrid	Line	86	78.59 🔻
Microsoft.Extensions.Caching.Hybrid	Branch	86	82.65 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Extensions.AI	88	89

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=913450&view=codecoverage-tab

mgravell · 2025-01-10T17:35:52Z

Clock skew: ultimately as stated, not easily solvable. But in the context of cache servers: we don't need super complexity or accuracy: IMO if any layer is unhappy re expiration, that's good enough to consider it invalid. As a side-effect, it also makes the logic time-testable.

dotnet-comment-bot · 2025-01-28T11:27:26Z

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.Extensions.Caching.Hybrid	Line	86	78.59 🔻
Microsoft.Extensions.Caching.Hybrid	Branch	86	82.65 🔻
Microsoft.Gen.MetadataExtractor	Line	98	57.35 🔻
Microsoft.Gen.MetadataExtractor	Branch	98	62.5 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Extensions.AI	88	89
Microsoft.Extensions.AI.OpenAI	77	78

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=931777&view=codecoverage-tab

…into marc/hc-tags3

dotnet-comment-bot · 2025-01-28T13:26:47Z

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.Extensions.Caching.Hybrid	Line	86	84.35 🔻
Microsoft.Extensions.Caching.Hybrid	Branch	86	84 🔻
Microsoft.Gen.MetadataExtractor	Line	98	57.35 🔻
Microsoft.Gen.MetadataExtractor	Branch	98	62.5 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Extensions.AI.OpenAI	77	78
Microsoft.Extensions.AI	88	89

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=931922&view=codecoverage-tab

…into marc/hc-tags3

dotnet-comment-bot · 2025-01-29T17:21:28Z

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.Extensions.AI.Ollama	Line	80	77.96 🔻
Microsoft.Extensions.Caching.Hybrid	Line	86	78.19 🔻
Microsoft.Extensions.Caching.Hybrid	Branch	86	82.83 🔻
Microsoft.Gen.MetadataExtractor	Line	98	57.35 🔻
Microsoft.Gen.MetadataExtractor	Branch	98	62.5 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Extensions.AI.OpenAI	77	78

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=933691&view=codecoverage-tab

jodydonetti · 2025-01-29T20:11:53Z

Hi Marc, I'm looking at this and wow, those are a lot of tests 💪

Just to doublecheck so we are on the same page: the idea is still this?

ps: the amount of work you are doing on HC is spectacular, kudos.

…s work)

dotnet-comment-bot · 2025-01-30T14:51:19Z

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.Extensions.AI.Ollama	Line	80	78.25 🔻
Microsoft.Extensions.Caching.Hybrid	Line	86	79.42 🔻
Microsoft.Extensions.Caching.Hybrid	Branch	86	83.84 🔻
Microsoft.Gen.MetadataExtractor	Line	98	57.35 🔻
Microsoft.Gen.MetadataExtractor	Branch	98	62.5 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Extensions.AI.OpenAI	77	78
Microsoft.Extensions.AI	88	89
Microsoft.Extensions.AI.Abstractions	83	84

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=934881&view=codecoverage-tab

dotnet-comment-bot · 2025-01-30T16:29:09Z

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.Extensions.Caching.Hybrid	Line	86	84.72 🔻
Microsoft.Extensions.Caching.Hybrid	Branch	86	84.34 🔻
Microsoft.Extensions.AI.Ollama	Line	80	78.25 🔻
Microsoft.Gen.MetadataExtractor	Line	98	57.35 🔻
Microsoft.Gen.MetadataExtractor	Branch	98	62.5 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Extensions.AI.OpenAI	77	78
Microsoft.Extensions.AI.Abstractions	83	84
Microsoft.Extensions.AI	88	89

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=934939&view=codecoverage-tab

src/Libraries/Microsoft.Extensions.Caching.Hybrid/Internal/TagSet.cs

dotnet-comment-bot · 2025-01-30T17:29:12Z

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.Extensions.AI.Ollama	Line	80	78.25 🔻
Microsoft.Gen.MetadataExtractor	Line	98	57.35 🔻
Microsoft.Gen.MetadataExtractor	Branch	98	62.5 🔻
Microsoft.Extensions.Caching.Hybrid	Line	86	85.48 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Extensions.AI.OpenAI	77	78
Microsoft.Extensions.AI	88	89
Microsoft.Extensions.AI.Abstractions	83	84

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=935018&view=codecoverage-tab

…Set.cs Co-authored-by: Sébastien Ros <[email protected]>

…into marc/hc-tags3

dotnet-comment-bot · 2025-01-30T19:29:04Z

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.Gen.MetadataExtractor	Line	98	57.35 🔻
Microsoft.Gen.MetadataExtractor	Branch	98	62.5 🔻
Microsoft.Extensions.AI.Ollama	Line	80	78.25 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Extensions.AI.Abstractions	83	84
Microsoft.Extensions.AI	88	89
Microsoft.Extensions.AI.OpenAI	77	78

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=935225&view=codecoverage-tab

dotnet-comment-bot · 2025-02-03T16:24:01Z

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.Extensions.Caching.Hybrid	Line	86	82 🔻
Microsoft.Extensions.AI.Ollama	Line	80	78.25 🔻
Microsoft.Gen.MetadataExtractor	Line	98	57.35 🔻
Microsoft.Gen.MetadataExtractor	Branch	98	62.5 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Extensions.AI.OpenAI	77	78
Microsoft.Extensions.AI	88	89
Microsoft.Extensions.AI.Abstractions	83	84

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=938365&view=codecoverage-tab

dotnet-comment-bot · 2025-02-03T17:28:05Z

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.Extensions.AI.Ollama	Line	80	78.25 🔻
Microsoft.Gen.MetadataExtractor	Line	98	57.35 🔻
Microsoft.Gen.MetadataExtractor	Branch	98	62.5 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Extensions.AI.Abstractions	83	84
Microsoft.Extensions.Caching.Hybrid	86	87
Microsoft.Extensions.AI.OpenAI	77	78
Microsoft.Extensions.AI	88	89

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=938418&view=codecoverage-tab

DeagleGross

Approved; but would be happy if someone else takes a look (PR is long living). Below asked a couple of specific code questions; overall purpose of PR is kind of understandable, but I dont have any background with HybridCache

DeagleGross · 2025-02-04T14:01:47Z

src/Libraries/Microsoft.Extensions.Caching.Hybrid/Internal/BufferChunk.cs

@@ -58,7 +62,7 @@ public byte[] ToArray()
        }

        var copy = new byte[length];
-        Buffer.BlockCopy(Array!, 0, copy, 0, length);
+        Buffer.BlockCopy(OversizedArray!, Offset, copy, 0, length);


not sure if it is important, but Span.CopyTo is faster than Buffer.BlockCopy. See proof

nice; I will integrate that next cycle

DeagleGross · 2025-02-04T14:09:37Z

src/Libraries/Microsoft.Extensions.Caching.Hybrid/Internal/DefaultHybridCache.L2.cs

+        var length = HybridCachePayload.Write(oversized, key, cacheItem.CreationTimestamp, GetL2AbsoluteExpirationRelativeToNow(options),
+            HybridCachePayload.PayloadFlags.None, cacheItem.Tags, payload.AsSequence());
+
+        await SetDirectL2Async(key, new(oversized, 0, length, true), GetL2DistributedCacheOptions(options), token).ConfigureAwait(false);


is ConfigureAwait(false) needed for lower FXs?

for environments that have a sync-context, which usually means lower FXs; however, as library code: we don't know about the sync-context state, so: best to be explicit

I believe the plan to allow a global setting for this is still being discussed; I will check whether that finally shipped!

if this were "aspnetcore code, only targeting netcurrent", then yes: we could probably get away without specifying this; however, HC a: can be used in non-aspnetcore contexts (WinForms, etc), and b: supports netstandard, netfx, etc

DeagleGross · 2025-02-04T14:15:48Z

src/Libraries/Microsoft.Extensions.Caching.Hybrid/Internal/DefaultHybridCache.L2.cs

+
+        byte[] oversized = ArrayPool<byte>.Shared.Rent(sizeof(long));
+        BinaryPrimitives.WriteInt64LittleEndian(oversized, timestamp);
+        var pending = SetDirectL2Async(TagKeyPrefix + tag, new BufferChunk(oversized, 0, sizeof(long), false), _tagInvalidationEntryOptions, token);


is there any overhead calling await SetDirectL2Async right here? If it is an already finished, isn't await doing nothing more than just getting a result?

(i guess the async-state-machine will still be generated, yep?)

yes, this is to avoid the state machine overhead in the happy path; that isn't terribad, but: it is sometimes desirable to hand-crank it

DeagleGross · 2025-02-04T15:36:07Z

src/Libraries/Microsoft.Extensions.Caching.Hybrid/Internal/DefaultHybridCache.cs

@@ -20,8 +21,11 @@ namespace Microsoft.Extensions.Caching.Hybrid.Internal;
 /// <summary>
 /// The inbuilt implementation of <see cref="HybridCache"/>, as registered via <see cref="HybridCacheServiceExtensions.AddHybridCache(IServiceCollection)"/>.
 /// </summary>
+[SkipLocalsInit]


why SkipLocalsInit exists on specific types only? Not on module?

I'd be fine with either, but I don't really need it globally; least surprises, etc

rebase hc-tags work from dev (easier to re-branch due to drift)

5fd212d

mgravell requested review from a team as code owners January 9, 2025 17:47

dotnet-policy-service bot assigned mgravell Jan 9, 2025

mgravell added 4 commits January 10, 2025 16:08

detect malformed unicode in tags/keys

9abc346

avoid try/finally in unicode validation step

886a413

make build happy

d8cc8e2

happier

c3cfd2f

evgenyfedorov2 added the area-caching-hybrid label Jan 17, 2025

jodydonetti mentioned this pull request Jan 24, 2025

[FEATURE] 🏷️ Tagging ZiggyCreatures/FusionCache#319

Closed

mgravell added 2 commits January 28, 2025 10:52

Merge branch 'main' into marc/hc-tags3

1b99172

Merge branch 'main' into marc/hc-tags3

0478a5d

mgravell added 2 commits January 28, 2025 12:52

normalize and test how per-entry vs global options are inherited

9a5e4dc

Merge branch 'marc/hc-tags3' of https://github.com/mgravell/extensions …

5d72360

…into marc/hc-tags3

mgravell added 7 commits January 29, 2025 16:32

add loggine of rejected data

6a278fa

event log for tag invalidations

654173b

make the CI overlord happy

47a4123

rebase hc-tags work from dev (easier to re-branch due to drift)

c60048e

detect malformed unicode in tags/keys

6664698

avoid try/finally in unicode validation step

75defc2

make build happy

ba433e5

mgravell added 5 commits January 29, 2025 16:48

happier

138946a

normalize and test how per-entry vs global options are inherited

f78e41d

add loggine of rejected data

466b369

event log for tag invalidations

803003f

make the CI overlord happy

1b89029

mgravell force-pushed the marc/hc-tags3 branch from 47a4123 to 1b89029 Compare January 29, 2025 16:48

Merge branch 'marc/hc-tags3' of https://github.com/mgravell/extensions …

33279db

…into marc/hc-tags3

add event-source into more tests (proves logging-related code branche…

b761e50

…s work)

mgravell added 2 commits January 30, 2025 15:29

add inbuilt type serializer test

11c7efd

add license header

16eff9e

improving code coverage

437e2fa

sebastienros reviewed Jan 30, 2025

View reviewed changes

mgravell and others added 5 commits January 30, 2025 18:42

incorporate PR feedback re thread-static array

a5b816e

make the robots happy

e470ef7

Update src/Libraries/Microsoft.Extensions.Caching.Hybrid/Internal/Tag…

a20dc13

…Set.cs Co-authored-by: Sébastien Ros <[email protected]>

tag-based invalidate: pass multiple tags, to ensure code paths

98dfbd6

Merge branch 'marc/hc-tags3' of https://github.com/mgravell/extensions …

8b5155e

…into marc/hc-tags3

Merge branch 'main' into marc/hc-tags3

6c7870b

more coverage

07e7a39

DeagleGross approved these changes Feb 4, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HybridCache : implement the tag expiration feature #5785

HybridCache : implement the tag expiration feature #5785

mgravell commented Jan 9, 2025 •

edited

Loading

jodydonetti commented Jan 9, 2025 •

edited

Loading

dotnet-comment-bot commented Jan 9, 2025

jodydonetti commented Jan 9, 2025

BrennanConroy commented Jan 9, 2025

jodydonetti commented Jan 10, 2025 •

edited

Loading

dotnet-comment-bot commented Jan 10, 2025

mgravell commented Jan 10, 2025 via email

dotnet-comment-bot commented Jan 28, 2025

dotnet-comment-bot commented Jan 28, 2025

dotnet-comment-bot commented Jan 29, 2025

jodydonetti commented Jan 29, 2025 •

edited

Loading

dotnet-comment-bot commented Jan 30, 2025

dotnet-comment-bot commented Jan 30, 2025

dotnet-comment-bot commented Jan 30, 2025

dotnet-comment-bot commented Jan 30, 2025

dotnet-comment-bot commented Feb 3, 2025

dotnet-comment-bot commented Feb 3, 2025

DeagleGross left a comment

DeagleGross Feb 4, 2025

mgravell Feb 4, 2025

DeagleGross Feb 4, 2025

mgravell Feb 4, 2025

mgravell Feb 4, 2025

DeagleGross Feb 4, 2025

DeagleGross Feb 4, 2025

mgravell Feb 4, 2025

DeagleGross Feb 4, 2025

mgravell Feb 4, 2025

HybridCache : implement the tag expiration feature #5785

Are you sure you want to change the base?

HybridCache : implement the tag expiration feature #5785

Conversation

mgravell commented Jan 9, 2025 • edited Loading

Microsoft Reviewers: Open in CodeFlow

jodydonetti commented Jan 9, 2025 • edited Loading

dotnet-comment-bot commented Jan 9, 2025

jodydonetti commented Jan 9, 2025

BrennanConroy commented Jan 9, 2025

jodydonetti commented Jan 10, 2025 • edited Loading

dotnet-comment-bot commented Jan 10, 2025

mgravell commented Jan 10, 2025 via email

dotnet-comment-bot commented Jan 28, 2025

dotnet-comment-bot commented Jan 28, 2025

dotnet-comment-bot commented Jan 29, 2025

jodydonetti commented Jan 29, 2025 • edited Loading

dotnet-comment-bot commented Jan 30, 2025

dotnet-comment-bot commented Jan 30, 2025

dotnet-comment-bot commented Jan 30, 2025

dotnet-comment-bot commented Jan 30, 2025

dotnet-comment-bot commented Feb 3, 2025

dotnet-comment-bot commented Feb 3, 2025

DeagleGross left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mgravell commented Jan 9, 2025 •

edited

Loading

jodydonetti commented Jan 9, 2025 •

edited

Loading

jodydonetti commented Jan 10, 2025 •

edited

Loading

jodydonetti commented Jan 29, 2025 •

edited

Loading