[Bug]: Investigate performance degradation between 2.3.0 and 2.4.0 version

[bbarani]

Describe the bug

Performance test results showed increase of indexing latency in 2.4.0 version compared to 2.3.0 version as listed here. We need to identify the root cause if the degradation ( i.e. Is the degradation caused due to perf tool setup OR related to OpenSearch core changes)

To reproduce

The results are published in 2.4.0 release issue

Expected behavior

No response

Screenshots

If applicable, add screenshots to help explain your problem.

Host / Environment

No response

Additional context

No response

Relevant log output

No response

[anasalkouz]

I can see the latency increased on 2.4 on 10/21/2022. @adnapibar may be Rabi if we can see what was merged on that date.

[adnapibar]

Doesn't look like any significant change

[adnapibar]

Do we know why there is a significant difference in the p99 between the perf tests ran today vs 11/10?
e.g. OpenSearch 2.3.0 vs 2.4.0 Linux (With security): 1.66% (11/10) vs -42.42% (today)

[andrross]

The NYC taxis dataset does define a couple geo_point data types. This change to add support for GeoJSON was backported to the 2.x branch on October 21, so it lines up with the regression. @heemin32 is there any chance this change resulted in a performance regression during indexing?

The throughput graph shows the change in performance a little more clearly, in my opinion:

[navneet1v]

The change in question defines a new way of inputting the geo point. It doesn't make any change in the indexing.

Will be doing deep dive on this.

[heemin32]

The change include creation of XContent for array format so that it can be extendible by plugin correctly. This might have caused the index delay for point data in array format.

opensearch-project/OpenSearch@a282d39

[heemin32]

NYC taxis dataset contains geopoint in array format. I suspect the degradation is from the change that I made in AbstractPointGeometryFieldMapper.java.

Instead of reverting the entire commit, I am going to revert the change in AbstractPointGeometryFieldMapper.java which shouldn't impact on existing functionality.

[bbarani]

Closing this issue as we found the root cause of the degradation. You can view more info on this PR