Redis overall performance improvements

[rehia]

While heavily using this awesome lib with Redis, I have a couple of performance improvements in mind.
I'd like to share them here, to open the discussion on it.
Obviously, this will lead to some PRs.

• the way events are retrieved by scanning all the keys with a match pattern forces to get all the keys before any mget, because scan does not guarantee the order. maybe using a sorted set to store keys could help scanning them more efficiently, and even stream events (I know, node-eventstore does not support streaming... yet!)

• snapshots are stored forever, although they can be useful for some time only. It could be useful to be able to clear the oldest, or having a rolling strategy which only keeps the nth latest.

I'll add other improvements in this thread. This is an open discussion.
So don't hesitate to make suggestions or remarks.

[adrai]

Nice ideas...
would mean when changing the redis implementation it will not be backwards compatible (just keep in mind)
For the rolling snapshot I wish that feature would be working for the other db implementation too ;-)

[rehia]

I still think about this issue.
And I had performance issues, even after using snapshots (I had about 40K snapshots).
It took a very long time (3 sec ?) loading the last one.

The main concern I see are still the ones mentioned in the issue description.

• Find a way to better index snapshots and events to scan a large number of them. This could be done even backwards compatible, by just creating new indexes next to the current ones.
• Remove old snapshots, definitely, with a default strategy of keeping them all, to be backwards compatible as well.

Don't know yet when I'll have time for this :/
Just applying quick fixes on my data for the moment...