mlcontext.Data.Cache is not releasing memory #6915

System Information (please complete the following information):

• OS & Version: Windows 10]
• ML.NET Version:ML.NET 3.0.0
• .NET Version: .NET 8

Describe the bug
When using mlcontext.Data.Cache on idataview the memory is not released even if GC is called and null is set for the idataview and mlContext.

When not using mlcontext.Data.Cache, the memory is yes released after setting null to idataview and calling GC.

To Reproduce
Load data into idataview, DONT call cache, create model with lgbm (fit, transform test set), destroy idataviews, check memory consumption
Load data into idataview, DO call cache, create model with lgbm(fit, transform test set), destroy idataviews, check memory consumption
See the difference

Expected behavior
All memory allocated by an object which no longer can actually be used should be released
If there is a specific way to release the memory please instruct me on how to do that.
I must use mlcontext.Data.Cache since its improves performance by 50%