Using typed queries becomes extremely slow on conversion

[Rydra]

I have the following query:

var transactions = await oclient
                .For<Transaction>(Transaccion.CrmTableName)
                .FindEntriesAsync(annotations).ToList();

while this works and maps the properties every retrieved Transaction entry properly (I'm using JsonProperty attributes on every member of the Transaction class), when the number of entries scales up to the thousands, it is extremely slow (for example, for 5000 records it's taking minutes).

Doing a small JSON.Net trick for conversion has lead me to the same result in just a matter of seconds, like this:

var aux = await oclient
                .For(Transaccion.CrmTableName)
                .FindEntriesAsync(annotations);

var transactions = JsonConvert.DeserializeObject<List<Transaccion>>(JsonConvert.SerializeObject(aux));

What's the problem down there? Why is the regular conversion of entities so slow?

Thank you.

[object]

Hi,
Is it just ToList() which is different? Then the problem lies in collection transformation. I can't think right now about something that migh affect it, if you have a public feed I can investigate what's going on. Otherwise since the library source is available, you can check it our yourself.

[Rydra]

I think ToList just materializes every element (due to lazy loading). You can add a ToList() to the second piece of code and still works very quickly in comparison.

After checking the source code on how Simple Odata Client converts from a Dictionary<string, object> to a concrete instance, it seems it relies a lot of reflection and other stuff for datatype conversions, which may make it so slow (just guessing). My other guess is that serializing and deserializing from JSON is a blazing fast operation.

I should profile it further to ensure my statements, but those are my two cents.

[object]

Thanks for the observation. Yes it must be it. But why ToDictionary isn't called in the second scenario? I am bit puzzled how JSON serialization may change it. Will appreciate more information.

[Rydra]

The return type of the second FindEntriesAsync is Enumerable<Dictionary<string, object>>. It's pointless to call in there ToDictionary.

When I call the Json serialize function over that list of dictionaries, internally iterates over the enumerable (therefore no need to call toList to materialize the results) , it turns the result into a series of key/pair values like "member": "value" (as it should be, given that it is a Dictionary). With that format, Json.Net is very efficient on deserializing it on every other type (like the Transaction class in my case).

[object]

Thank you, I will investigate it further.

[Amberg]

Any news on this?
ToList() on a typed query takes very very long. Especially on a Android device.

The "slow" callstack ist often something like:

0x87 in System.RuntimeType.GetBaseType C#
0x21 in System.RuntimeType.IsSubclassOf C#
0x31 in System.RuntimeType.IsValueTypeImpl C#
0x1 in System.Type.get_IsValueType C#
0x6 in Simple.OData.Client.Extensions.TypeExtensions.IsValue C#
0x1 in Simple.OData.Client.Extensions.DictionaryExtensions.IsCompoundType C#
0x13 in Simple.OData.Client.Extensions.DictionaryExtensions.ConvertSingle C#
0x97 in Simple.OData.Client.Extensions.DictionaryExtensions.ConvertCollection C#
0x13 in Simple.OData.Client.Extensions.DictionaryExtensions.ConvertValue C#
0xAC in Simple.OData.Client.Extensions.DictionaryExtensions.ToObject C#
0xBE in Simple.OData.Client.Extensions.DictionaryExtensions.ToObject<SharedDto.PublicApi.InstitutionDtoV3> C#
0x8D in Simple.OData.Client.FluentClientBase<SharedDto.PublicApi.InstitutionDtoV3,Simple.OData.Client.IBoundClient<SharedDto.PublicApi.InstitutionDtoV3>>.ConvertResult C#
0xD in Simple.OData.Client.FluentClientBase<SharedDto.PublicApi.InstitutionDtoV3,Simple.OData.Client.IBoundClient<SharedDto.PublicApi.InstitutionDtoV3>>. C#
0x12 in System.Linq.Utilities.<>c__DisplayClass2_0<Simple.OData.Client.AnnotatedEntry,System.Collections.Generic.IDictionary<string,object>,SharedDto.PublicApi. C#
0x2A in System.Linq.Enumerable.SelectListIterator<Simple.OData.Client.AnnotatedEntry,SharedDto.PublicApi.InstitutionDtoV3>.ToList C#
0x20 in System.Linq.Enumerable.ToList<SharedDto.PublicApi.InstitutionDtoV3> C#

[phatcher]

@Amberg What version are you using, the reflection performance issues were addressed by 5.6+

[MartinWegner]

I ran into this problem today. The initial behavior that @Rydra described is still present in version 6.0.1. His fix with JsonConvert works too.

If anybody runs into problems with nullvalues. You can fix this by using JsonConvert.DeserializeObject<List<T>>(JsonConvert.SerializeObject(res), new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore });