Creating memory efficient reactive IObservable<IChangeset<T>> based on Realm database query

[kyurkchyan]

Realm Intro

I am working on multiple applications that utilize the Realm database. It's an amazing object-relational database with reactive capabilities. Essentially, any query and object are live, and you can subscribe to object and query changes to update the UI reactively.

Realm's .NET SDK is a practical tool that implements INotifyPropertyChanged for individual objects and INotifyCollectionChanged events for the queries. This means you can confidently bind the realm objects and collections directly to the UI controls supporting those interfaces, such as WPF and .NET MAUI.

Direct binding to realm results

I've created a sample project: RxRealm, demonstrating all of the use cases mentioned here.

For instance, imagine you have an entity called Product. You can get the products from the Realm database like this

IEnumerable<Product> products = realm.All<Product>();

// Assumes .NET MAUI collection view
collectionView.ItemsSource = products;

Take a look ProductsPage and ProductsViewModel in the sample project for an example of this kind of implementation.

Having just those 2 lines above will give you the following features:

1. You can scroll through the list of all products
2. Whenever a product is added or removed, the collection view will be updated automatically as the collection returned by the Realm database implements INotifyCollectionChanged, and it works as if you have bound the items to an ObservableCollection

The beauty of the Realm database is that you can have 1 million products in it and still use the above 2 lines of code without any pagination or worries about memory and performance. The actual data will only be fetched from the database when accessed. So, you can scroll through those 1 million products without worrying about lazy loading or performance issues. Everything will be very nice and smooth.

Transforming model to a view model

However, your application will eventually become complicated. Instead of directly using the Product entity, you need to create a ProductViewModel with some additional logic, or you may need to combine multiple collection streams into one. There are multiple ways you can achieve this.

Approach 1: direct selection

The naive way of doing this would be something like this:

IEnumerable<ProductViewModel> products = realm.All<Product>().Select(p => new ProductViewModel(p));

The above approach will not work, as Realm's LINQ implementation doesn't support transformations through the Select operator and you will receive an exception if you try doing something like above

Approach 2: converting to realm collection and using selection

The next naive approach would be doing something like this:

IEnumerable<ProductViewModel> products = realm.All<Product>().AsRealmCollection().Select(p => new ProductViewModel(p));

Even though AsRealmCollection returns a collection that implements INotifyCollectionChanged and has the same lazy-loading and memory-efficient characteristics as the plain realm.All<Product>(), as soon as you do Select you convert that collection to an IEnumerable instead. This means that as soon as you assign the result of Select to a collection view ItemsSource, it will materialize the enumerable and create a list for all of the objects inside. Thus, you've lost not only the lazy-loading aspect of it (you will create millions of objects if the collection has millions of objects), but also the reactiveness - you won't receive change notifications from newly created IEnumerable.

I have a benchmark called LoadProductsFromRealm_WithSelection inside the sample project and here's what you'll end up having when using this approach

Method	Mean	Error	StdDev	Gen0	Gen1	Gen2	Allocated
LoadProductsFromRealm_WithSelection	6.296 s	0.9147 s	0.2375 s	233000.0000	94000.0000	23000.0000	1.24 GB

As you can see we allocate 1.24GB of memory for 1 million products with this approach.

Approach 3: Dynamic data

The next best thing is using DynamicData. The code will look something like this

realm.All<Product>().AsRealmCollection()
                                       .AsObservableChangeSet()
                                       .Filter(item => item.IsValid)
                                       .Transform(p => new ProductViewModel(p));

Take a look at the benchmark called LoadProductsWith_WithDynamicDataTransformation

Method	Mean	Error	StdDev	Median	Gen0	Gen1	Gen2	Allocated
LoadProductsWith_WithDynamicDataTransformation	13.45 s	26.37 s	6.848 s	10.28 s	244000.0000	86000.0000	2000.0000	1.57 GB

As you can see, the memory consumption is 1.57GB, which is horrible. The only benefit of this solution compared to the previous one is that the result of this can be bound to an ObservableCollection and the products collection will remain reactive and responsive to additions and deletions.

Approach 4: Dynamic data virtualization

We need to instantiate ProductViewModel only when the data is needed in the view. The DynamicData virtualization framework can help here.

Subject<VirtualRequest> pagination = new();
        var products = _productsService.Realm.All<Product>().AsRealmCollection()
                                       .AsObservableChangeSet()
                                       .Virtualise(pagination)
                                       .Transform(p => new ProductViewModel(p));
        products.Subscribe(c => Console.WriteLine($"________________ Products count changed: {c.TotalChanges}"))
                .DisposeWith(_disposables);
        pagination.OnNext(new VirtualRequest(0, 50));

We will only create ProductViewModel whenever a new page is fetched from the original list. Then, in the UI, we will send another request to load the next page whenever we reach the end of the collection view. As the UI will be bound to the observable collection, it will update automatically.

This approach is implemented in the sample project's VirtualizedProductsPage & VirtualizedProductsViewModel.

Take a look at benchmark LoadProductsWith_Virtualization_Transformation

Method	Mean	Error	StdDev	Gen0	Gen1	Gen2	Allocated
LoadProductsWith_Virtualization_Transformation	1.971 s	0.6475 s	0.1682 s	56000.0000	21000.0000	2000.0000	360.15 MB

As you can see, the time and memory consumption are much less than in the previous approaches. However, they're still substantial. You can also notice this performance hit if you launch the application and try to switch to the "Virtualized" tab - the application will require significant time to become responsive and show the page until the dynamic data completes loading the data.

This happens because AsObservableChangeSet will still instantiate a collection under the hood to track the changes. If we have 1 million products in the database, a collection with 1 million items will be instantiated, resulting in significant memory consumption.

This approach also destroys the reactivity. When something changes inside the realm database and you use virtualization, the changes are not applied to the paginated list.

Approach 5: Using SourceList and manually implementing pagination

With this approach, we implement the pagination manually. We can use SourceList and then, every time we want to load the next page of the data, we fetch it manually from the Realm database and insert it into the SourceList, which then reports the results back to the view. To make this abstraction easier to consume, I've implemented a class called PaginatedResults<T> that abstracts away the logic of loading the data, tracking the pages and reporting on the count and whether there is more data or no.

This approach is implemented in PaginatedProductsPage and PaginatedProductsViewModel.

If we take a look at benchmarks LoadFirstPageOfProducts and Load10PagesOfProducts

Method	Mean	Error	StdDev	Median	Allocated
LoadFirstPageOfProducts	33.66 ms	229.50 ms	59.60 ms	6.701 ms	34.32 KB
Load10PagesOfProducts	52.71 ms	54.15 ms	14.06 ms	47.546 ms	207.36 KB

Everything looks much better both in terms of processing time and memory consumption.

This approach is better than any of the approaches mentioned above, which I am currently using in my app.
However, this approach comes with a compromise - it's not reactive and won't update the list when the source data in the database changes.

The challenge of implementing a reactive and a memory-efficient solution

The solution that I want to achieve is having an implementation of IObservable<IChangeset<T>> which is tailored towards the specifics of the realm database and has the following characteristics

1. Is reactive and updates the changes when they happen in the database
2. Is memory efficient - loads only the data that's needed. Probably, through the built-in Dynamic data virtualization
3. Database changes should trigger changes inside the collection only if that data is present in the UI. This will ensure that we don't have unnecessary subscriptions to the internal change streams unless necessary.

Essentially, we should have a "window" that overlooks the source realm data. Data should be instantiated only within that window, changes should be reported only within that "window".

I imagine that I will need to create something like SourceList called RealmSourceList and implement the necessary interface methods in a way to satisfy the above mentioned constraints. However, after looking into the implementation of the SourceList I couldn't figure out how exactly I would do that.

I have seen people asking this same question in the ReactiveUI slack channel, but no one found a real solution. I decided to put an effort to create this article/discussion both to share my efforts solving this issue and also finally find the real, elegant, reactive and memory efficient solution that perfectly integrates into the DynamicData functional pipelines.

[JakenVeina]

Realm looks like a very intriguing project, I'm sorry I'd never heard of it before. Thanks for bring it up.

By my reckoning, the version of .AsObservableChangeSet() that you're using in approaches 3 and 4 is...

public static IObservable<IChangeSet<TObject>> AsObservableChangeSet<TObject>(
    this    IEnumerable<TObject>    source,
            bool                    completable = false);

So yeah, that's going to naively force the entire Realm collection to materialize, permanently in approach 3, and temporarily in approach 4.

By my reckoning, what you want is this method, found on DynamicData.Binding.ObservableCollectionEx...

public static IObservable<IChangeSet<T>> ToObservableChangeSet<TCollection, T>(this TCollection source)
    where TCollection : INotifyCollectionChanged, IEnumerable<T>
    where T : notnull;

This lines up VERY well with what Realm states their concept of a collection is...

public interface IRealmCollection<out T>
    : IReadOnlyList<T>,
        IReadOnlyCollection<T>,
        IEnumerable<T>,
        IEnumerable,
        INotifyCollectionChanged,
        INotifyPropertyChanged
{ }

I.E. it's an object that allows you to iterate over a set of items (IEnumerable<T>), and publishes notifications about when those items are added, moved, removed, or replaced (INotifyCollectionChanged). And as this operator suggests, that's really all that DynamicData needs, in order to build a change stream. the IEnumerable<T> aspect is needed just to build the initial changeset, and INotifyCollectionChanged is used to derive all future changesets.

Really, it surprises me that Realm doesn't implement their own concept of a transform operator. Perhaps they plan to in the future, and if they ever did, I think I would recommend going with that. But until such time, DynamicData definitely can fill the gap, and that .ToObservableChangeSet() operator should handle the interop. With this operator, you can keep all the paging and "virtualization" handled by Realm, and handle transforms, binding, and maybe sorting, with Dynamic Data. I wouldn't say adding a RealmSourceList<> or RealmSourceCache<> would be a good idea, unless you just REALLY like Dynamic Data's APIs on those classes, more than the equivalents in Realm.

I do note that there doesn't appear to be a Cache equivalent of the .ToObservableChangeSet() operator. Generally speaking, Cache-land in DynamicData almost always outperforms its equivalents in List-land, so that could be a new operator we add support for. Then again, if Realm is handling paging already, the data sizes that DynamicData would see may be small enough that the performance difference is insignificant. It's also pretty easy to convert to a Cache stream right after the .ToObservableChangeSet() operator, with the .AddKey() operator, it would just likely be more efficient to have an operator that keys items coming out of Realm immediately.

As far as adding support for Realm specifically, we could add a DynamicData.Realm extension package, but I don't actually see what value added there would be. From the use-cases you describe, .AsRealmCollection().ToObservableChangeSet() seems to handle all the functionality that one might need. If there's other Realm APIs you think DynamicData could benefit from consuming directly, I'd be happy to see them.

[kyurkchyan]

Hey @JakenVeina

Thanks for getting back to me with this extensive answer. Realm is an amazing database. I am also amazed how few people know about it; it outperforms Sqlite, is much easier to use, and, most amazingly, has real-time sync capabilities through MongoDB AppServices service. It's definitely something you can try!

Unfortunately, Realm doesn't seem to have any immediate plan to support the transformation (aka LINQ Select operator) any time soon as mentioned here. It also doesn't have a real pagination API that would allow doing skip/take that preserves the INotifyCollectionChanged interface.

Regarding the usage of the ToObservableChangeSet() extension method instead of AsObservableChangeSet. I have tried them both. Unfortunately, it has the same performance characteristics as other approaches.

[Benchmark]
    public void LoadProductsWith_Virtualization_Transformation_ToObservableChangeSet()
    {
        Subject<VirtualRequest> pagination = new();
        var products = _productsService.Realm.All<Product>().AsRealmCollection()
                                       .ToObservableChangeSet<IRealmCollection<Product>, Product>()
                                       .Virtualise(pagination)
                                       .Transform(p => new ProductViewModel(p));
        products.Subscribe(c => Console.WriteLine($"________________ Products count changed: {c.TotalChanges}"))
                .DisposeWith(_disposables);
        pagination.OnNext(new VirtualRequest(0, 50));
    }

When done on a large set of products, (1 million to be exact), this is what the benchmarks look like:

Method	Mean	Error	StdDev	Gen0	Gen1	Gen2	Allocated
LoadProductsWith_Virtualization_Transformation_ToObservableChangeSet	1.422 s	0.6000 s	0.1558 s	44000.0000	15000.0000	1000.0000	290.74 MB

It results in 290MB of usage and tone of garbage collection when doing just 25 item virtualization. So, this method even though has some virtualization - it doesn't transform the objects outside of the virtualization, it does create an inner list wrapping the initial IEnumerable<T> which essentially materializes the realm collection.

If we look inside the ToObservableChangeSet implementation, we will see a line like this.

 public static IObservable<IChangeSet<T>> ToObservableChangeSet<TCollection, T>(this TCollection source)
        where TCollection : INotifyCollectionChanged, IEnumerable<T>
        where T : notnull
    {
                ............
                var data = new ChangeAwareList<T>(source);
                ............

And here's the ChangeAwareList constructor

public ChangeAwareList(IEnumerable<T> items)
    {
        items.ThrowArgumentNullExceptionIfNull(nameof(items));

        var list = items.ToList();

        _innerList = new List<T>(list);

        if (_innerList.Count > 0)
        {
            _changes.Add(new Change<T>(ListChangeReason.AddRange, list));
        }
    }

Essentially, it does a ToList materializing the realm collection causing that large memory footprint on large datasets.

Frankly, I will never have a query that returns 1 million records in my application. I have used my benchmarks to exaggerate reality and understand how DynamicData extensions behave. However, I do have queries that return thousands of records, and the performance hit becomes noticeable, especially on less powerful mobile devices. So, I need to find a better solution to the problem that will have virtualization and reactive change notifications built in.

[JakenVeina]

If Realm itself doesn't support any pagination, I don't see how Dynamic Data could be made to interop with it, for large datasets. I also don't see how Realm is terribly useful, at all, then. If you can only bind an entire table at once, into a view, what's the point of using Realm? Just to capture incremental changes to the table after the binding is initialized? You stated "The actual data will only be fetched from the database when accessed." initially, but I don't see how this is possible if Realm doesn't support some kind of paging. That's really what paging is, it's saying "I don't need the entire dataset all at once, just a portion of it".

Maybe there's an API in Realm at a lower-level than IRealmCollection<> that DD could hook onto. I should definitely take some time, when I can, to experiment with Realm, hands-on.

[kyurkchyan]

@JakenVeina , it totally makes sense. Realm relies on the fact that they provide a lazily evaluated IEnumerable<T> which also happens to implement INotifyCollectionChanged. And I hoped that DynamicData could integrate with these interfaces, preserving lazy evaluation and collection change events. But I realize that it's not possible without materializing the returned results. And thus, realm should have actual pagination built-in.

Thanks for looking into this

[JakenVeina]

Lazily-evaluated IEnumerable<T>s, like enumerating them is thread-blocking, while the database is queried? Eesh, that's a rather big no-no in .NET-land. I figured it would be more like the IEnumerable<T> just initially enumerates as empty, and just fires CollectionChanged as items are loaded. Structurally, that'd make it rather simple to support paging, you'd just need to support the paging params as mutable inputs to the queries.