Performance with many operations

[radarfox]

Hello Jason,

we are planning to migrate our app storage adapter from "js-data-localstorage" to "js-data-localforage" because of some bugs in localStorage implementation on some browsers which we would like to support. But the problem is that we ran into some serious performance issues.

Both adapters in current version save these storage items for a resource:

• item with map of all stored ids
• item for each model in the resource

Our average storage now contains around 6000 items with total data around 4MB and the possibility is that the amount of items will be raising even more in time. We create/destroy all those data together as user logs in/out of our app (with createMany and destroyMany).

So I made some storage performance testing on target Android v5 tablet (Samsung Galaxy Tab 4 10.1) and here are the results:

• IndexedDB 5000 items x 5kB => 151s (!!!)
• IndexedDB 5 items x 5000kB => 2,5s
• WebSQL 5000 items x 5kB => 111s (!!!)
• WebSQL 5 items x 5000kB => 4s
• localStorage 1000 items x 5kB => 9s
• localStorage 10 items x 500kB => instant

Tests on other devices, like iOS and desktop browsers came basically the same. The time that operations take, drastically rises on number of items, not on their data size.

The conclusion is that the current approach with many storage items is not acceptable under this storage utilization. None of our clients will wait 2 extra minutes just for offline The problem is in both localstorage and localforage adapters, but is much more affecting the localforage adapter in IndexedDB or WebSQL mode.

So what we should do to get rid of this problem?

I'm thinking of rewriting the whole adapter functionality so there is always 1 storage item for 1 resource. This item would basically contain the same data as current "ids map item" where ids would be substituted by the model data. Here is the example:

Key: some-base-path/resource
Value: {"foo":{"id":"foo","value":true},"bar":{"id":"bar","value":false}}

I'm aware of that performance with single model changes will suffer, because of a need to JSON stringify/parse the whole resource data into the single storage item. But the extra time should be much lower and acceptable as it is solving 2 minutes of waiting on "xxxMany" operations.

Maybe the best solution would be to add something like storageStrategy configuration, so you can choose on each resource if you want to use single or many storage items in adapter operations.

As I'm writing this post, I have another idea for storage organization. Maybe it should work "hybrid" and allow both single and many approach at the same time. For example, for a situation where I had storageStrategy: 'many' on existing resource and then I switch to storageStrategy: 'single' and I don't want to loose the old data. I might look something like:

Key: some-base-path/resource
Value: {"foo":1,"bar":{"id":"bar","value":false}}

Key: some-base-path/resource/foo
Value: {"id":"foo","value":true}

So what do you think of that all? Hope I haven't flooded you with information, but I was trying to explain you all the important ideas and causes.

This issue is very important for our application and we hope you can help us solve it. Would be nice, if you contribute with some code, or at least by letting us implement it and then merge into your adapter production version, so we don't lose further features and fixes. Implementation of this (or any other working) solution is inevitable for us.

Thanks for your interest and developing these superb plugins ;)

[radarfox]

Almost 5 months passed without any reply. So this project is not maintained anymore?

[jmdobry]

It is, it's just lower priority than the 20 other projects I maintain.

[radarfox]

I understand your situation. I reacted based on your "Looking for contributors!" appeal on homepage. I thought you will be interested in our help with developing this project. Maybe I hoped for quick evaluation or a few hints based on your knowledge of jsdata ecosystem. Well if you are not interested, then we will develop this feature on our own.

[jmdobry]

Your storageStrategy idea looks like it could work, I'd be happy to review your pull requests.

[mobidev111]

@radarfox are you still working on this? I would be interested in this - and revamping the localforage adapter to work with js-data v3. so we could join forces..