Feature: Memory/Indxdb Hybrid

[nickchomey]

Is your feature request related to a problem?

The indxdb backend is vastly slower (5-10x, across various queries) than the in-memory backend. (And in-memory is much slower than using a single node RocksDB on the server). My (and surely most) web app require persistence, but the slow performance is somewhat prohibitive...

Describe the solution

Perhaps it might be possible to use some sort of hybrid of the memory and indxdb backends, such that, at the very least, reads are done from memory while writes get passed through to indxdb.

RxDB has a "Memory Mapped" mechanism for this, which apparently improves performance by orders of magnitude.

Alternative methods

na

SurrealDB version

na

Contact Details

No response

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

[nickchomey]

I just set up RxDB's basic Dexie Storage adapter to compare performance with SUrrealdb WASM with the indxdb backend. I was surprised to see that they were fairly comparable on the basic queries that I was running. Then I realized that I hadn't added an index to the query filter field in RxDB. When I did so, it became about 5x faster than Surreal. (And, again, Surreal indxdb without any indexes is another 5x slower than with indexes)

If I switch to Surrealdb's mem backend, the performance is about on par with persisted RxDB!

So, it seems that Surreal's indxdb doesnt actually make use of indexeddb's indexes - it just stores a lot of ArrayBuffers, which presumably contain its own indexes. So, in order to use Surreal's (slower) indexes, it has to just do normal non-indexed data querying/retrieval. Given this, it doesnt seem particularly possible to leverage IndexedDB's native indexes. This makes a hybrid "memory mapped" mechanism all the more important.