Methods to support common scenarios

[SuperJMN]

I find myself writing methods like this too often:

changeSet
   .ToCollection()
   .Select(col => col.Any());

changeSet
   .ToCollection()
   .Select(col => col.Count());

changeSet
   .ToCollection()
   .Select(col => col.Any(condition));

Is there any better (more idiomatically) do things like this?

[JakenVeina]

Not built-in, but yeah, each of those could deeeeeeeefinitely be done better. Each one of those appears to perform a full clone of the existing collection upon each received ChangeSet. That wouldn't be too terrible if the collections were System.Collections.Immutable, internally, but they're not. For IChangeSet<T>, the internal collection is just List<T>.

For these specific scenarios, you could optimize pretty heavily.

public static IObservable<int> Count(this IObservable<IChangeSet<T>> source)
    => source.Scan(
        source:         source,
        seed:           0,
        accumulator:    static (count, changeSet) =>
        {
            var nextCount = count;
            foreach(var change in changeSet)
            {
                if (change.Reason == ChangeReason.Add)
                    ++nextCount;
                else if (change.Reason == ChangeReason.Remove)
                    --nextCount;
            }
            return nextCount;
        });

public static IObservable<int> Count(this IObservable<IChangeSet<T>> source)
    => source.Count().Select(count => count != 0);

The last one, .Any(predicate) can be accomplished with just a combination of .Filter() and .Any(), which should be reasonably performant. A more direct implementation that just caches matched items and counts might be a bit more optimal, but it's probably not worth it.

[RolandPheasant]

An optimized version of Count() is available under DynamicData.Aggregation namespace, and Any would probably be a good addition to the aggregation namespace. I'd welcome a PR.

@JakenVeina Before I released Dynamic Data, I experimented with every conceivable data structure and one of the worst performing (memory and speed considered) ones was the immutable collections. For all round performance, the standard list proved to be much superior to all else, mainly because it can be pre-sized and that operations when the index if known are super fast,

[JakenVeina]

DynamicData.Aggregation

Damn. Somehow I missed that namespace when I was researching all the extension methods, earlier today.

So, ImmutableList<T> beat out List<T> for memory usage? That surprises me, for sure. The memory usage of an Add/Insert ought to be O(log n), right? For a tree structure that occasionally needs to re-balance? While it'd be O(n) for allocating a new List<T> every time. I can definitely buy that the speed performance would be better with List<T>.

Neither holds a candle to just keeping a count, though ;).

Would it be worth making a dedicated .Any() operator, or just make it an alias for .Count != 0 like I did? Or were you referring to an .Any(predicate) implementation? If so, same question, should it just be .Filter().Any() or a dedicated implementation?

[RolandPheasant]

I don't want to get into details about some measurements about all around performance profile which I did a decade ago, but I too was surprised by the memory characteristics of the immutable list and genuinely horrified by it's slowness. Regardless however, it was unusable within Dynamic Data. Just to remind myself I just read this and the measurements seem to back up my findings. See SomeMicrosoftArticle.

Regarding Count(), the optimized aggregations in DynamicData.Aggregation do not store the underlying collection at all and are therefore more efficient. This means they cannot reactive to object mutations, which is the reason for using backed up internal storage elsewhere. Version 1 of Dynamic Data (which I never released) did not even account for object mutations and life then was simpler and happier and required very little state tracking !

Regarding the Any operator, I don't think it would make much difference whether Count!=0, or a dedicated implementation was introduced as it would only be a few lines of code anyway. For example, see CountEx

[JakenVeina]

See SomeMicrosoftArticle.

I was looking at that article yesterday. I particularly like how Figure 3 is titled "Amount of Time and Memory Used" and then gives numbers that are unlabeled as to whether they indicate time or memory usage. And then Figure 4 appears to be simply broken.

See CountEx.

Might as well just make .Any() an alias for .IsNotEmpty() then.

Isn't there a separate .Aliases namespace, as well? Would these new extensions belong there?

[dwcullop]

I find myself writing methods like this too often:

changeSet
   .ToCollection()
   .Select(col => col.Any());

I would do like this instead:

IObservable<bool> hasAny = changeSet.ToObservableCache().CountChanged.Select(n => n > 0);

changeSet
   .ToCollection()
   .Select(col => col.Count());

This is even more simple, but the same idea:

IObservable<int> collectionSize = changeSet.ToObservableCache().CountChanged;

changeSet
   .ToCollection()
   .Select(col => col.Any(condition));

Same as the first, but filter the changeset BEFORE it comes in.

Is there any better (more idiomatically) do things like this?

IObservable<bool> hasAny = changeSet.Filter(condition).ToObservableCache().CountChanged.Select(n => n > 0);

This approach still uses an intermediate collection, like ToCollection, the difference is, you don't have to re-create the collection each time it changes. Only the deltas get applied and then the events fire from that. More efficient, especially for large collections.

[SuperJMN]

OK, I've found this scenario quite common:

obs.Filter(x => x.IsMatch)
.Count()
.Select(x => x > 0);

The problem is that it doesn't push 0 when the filtered colleciton is empty. How to proceed?

[dwcullop]

First, Filter has an optional boolean to control whether or not empty changesets are suppressed or not. Pass in false for that and you'll see the empty ones, but they won't help your problem.

Count is not the right operator. That's a regular Rx operator that emits the number of values it has seen when the observable completes. It won't know anything about ChangeSets, so you'll just get the number of changesets that were observed, which isn't the number of items in the changesets.

There isn't an operator like that for ChangeSets, so you'll have to use a temporary cache. I would hide it in a custom operator. Here are some examples. One that will give you the count like you wanted and another that just gives you a bool that fires when the number of entries go over zero (or back down to zero).

public static class ObservableChangeSetEx
{
    public static IObservable<int> CountChanged<T, TKey>(this IObservable<IChangeSet<T, TKey>> source)
        where T : notnull
        where TKey : notnull =>  Observable.Create<int>(observer =>
    {
        var localCache = new IntermediateCache<T, TKey>(source);
        return new CompositeDisposable(localCache, localCache.CountChanged.SubscribeSafe(observer)); 
    });

    public static IObservable<bool> HasEntries<T, TKey>(this IObservable<IChangeSet<T, TKey>> source)
        where T : notnull
        where TKey : notnull =>
        Observable.Defer(() => source.CountChanged().Select(count => count > 0).DistinctUntilChanged());
}

Then you can use it like these few examples:

using var cleanup = obs.Filter(x => x.IsMatch).HasEntries()
     .Do(hasValue => SetHasValues(hasValue))
     .Subscribe();

using var cleanup2 = obs.Filter(x => x.IsMatch).CountChanged()
     .Do(count => SetCount(count))
     .Subscribe();

[JakenVeina]

Count is not the right operator. That's a regular Rx operator...

There isn't an operator like that for ChangeSets, so you'll have to use a temporary cache.

There is very much an operator, and it's called Count. It's just not in the core DynamicData namespace, it's in DynamicData.Aggregation.

DynamicData/src/DynamicData/Aggregation/CountEx.cs

Line 21 in 04e2ff8

public static IObservable<int> Count<TObject, TKey>(this IObservable<IChangeSet<TObject, TKey>> source)

The bit about Filter and the suppressEmptyChangesets flag is the answer here.

[dwcullop]

Ah ha! DynamicData.Aggregation TIL!

I looked for it in the IDE and didn't show up, but I didn't have that namespace listed. Without it, it just finds the regular Rx operator of the same name.

But with it, it finds it and it works similar to the custom one I wrote above (except it doesn't fire an initial 0 count). However, neither one need to deal with empty changesets to work correctly... At least not that I can see.

[JakenVeina]

I looked for it in the IDE and didn't show up, but I didn't have that namespace listed. Without it, it just finds the regular Rx operator of the same name.

I went through the same thing, a while back.

There's definitely a discoverability/documentation deficiency, here.