Change the default search sort tiebreaker to `_shard_doc` for PIT search #4378

LantaoJin · 2025-09-25T06:49:51Z

Description

Change the default search sort tiebreaker of PIT search requests to _shard_doc after opensearch-project/OpenSearch#18924 being merged.

Main changes:

Remove sort field _doc in pushDownFilter(). (_doc is a default hits ordering if no specific orders are set)
Explicitly add _shard_doc as sort tiebreaker of PIT search (~~not 100% sure the current implementation of _shard_doc is an implicit sort in PIT search, see~~ question).
Keep the legacy engine unchanged.

Related Issues

Resolves #3064

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Lantao Jin <[email protected]>

LantaoJin · 2025-09-26T06:08:34Z

Is it a must have for 3.3.0?

dai-chen

Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?

LantaoJin · 2025-09-28T03:55:09Z

Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?

I think all PIT search requests should add _shard_doc as tiebreaker, or the _score will used as implicit sort field in response. But this PR doesn't check all code base, it just changes the all places which use _doc + _id as tiebreaker to _shard_doc.

I checked PR of _shard_doc, seems only PIT search requests can sort by _shard_doc. Non-PIT search requests sort by _shard_doc throws exception. I will refactor this PR to force v2 and v3 only. The v1 (legacy) code is out of my knowledge, I won't touch them in this PR.

Signed-off-by: Lantao Jin <[email protected]>

LantaoJin · 2025-09-30T05:08:55Z

opensearch-project/OpenSearch#18924 merged just now. So I tested in local that all IT are passed except GeoIpFunctionsIT mentioned here.

So when #4036 merged and the OpenSearch 3.3.0-SNAPSHOT published with latest code. The CI of this PR should be passed.
cc @penghuo @Swiddis

LantaoJin · 2025-09-30T05:13:44Z

docs/user/optimization/optimization.rst

            "name": "OpenSearchIndexScan",
            "description": {
-              "request": "OpenSearchQueryRequest(indexName=accounts, sourceBuilder={\"from\":0,\"size\":10000,\"timeout\":\"1m\",\"query\":{\"bool\":{\"filter\":[{\"range\":{\"age\":{\"from\":null,\"to\":20,\"include_lower\":true,\"include_upper\":false,\"boost\":1.0}}},{\"range\":{\"age\":{\"from\":10,\"to\":null,\"include_lower\":false,\"include_upper\":true,\"boost\":1.0}}}],\"adjust_pure_negative\":true,\"boost\":1.0}},\"_source\":{\"includes\":[\"age\"],\"excludes\":[]},\"sort\":[{\"_doc\":{\"order\":\"asc\"}}]}, searchDone=false)"
+              "request": "OpenSearchQueryRequest(indexName=accounts, sourceBuilder={\"from\":0,\"size\":10000,\"timeout\":\"1m\",\"query\":{\"bool\":{\"filter\":[{\"range\":{\"age\":{\"from\":null,\"to\":20,\"include_lower\":true,\"include_upper\":false,\"boost\":1.0}}},{\"range\":{\"age\":{\"from\":10,\"to\":null,\"include_lower\":false,\"include_upper\":true,\"boost\":1.0}}}],\"adjust_pure_negative\":true,\"boost\":1.0}},\"_source\":{\"includes\":[\"age\"],\"excludes\":[]}}, searchDone=false)"


the almost of changes in rst, json and yaml files are moving sort _doc, even in non-pit search request, the sort _doc is not necessary.

LantaoJin · 2025-09-30T05:15:27Z

opensearch/src/main/java/org/opensearch/sql/opensearch/request/OpenSearchQueryRequest.java

+      // https://github.com/opensearch-project/OpenSearch/pull/18924#issuecomment-3342365950
+      if (this.sourceBuilder.sorts() == null
+          || this.sourceBuilder.sorts().stream().noneMatch(ShardDocSortBuilder.class::isInstance)) {
+        this.sourceBuilder.sort(SortBuilders.shardDocSort());


This line is the primary change of this PR.
Currently, the sort tiebreaker _shard_doc is not implicit added in PIT search request. So we explicit sort with SortBuilders.shardDocSort().

Swiddis · 2025-09-30T22:46:26Z

#4420 still fails tests after merge

LantaoJin · 2025-10-01T01:08:13Z

#4420 still fails tests after merge

Seems #4420 contains some changes that are not in this PR. Let me update from upstream and run CI again.

Signed-off-by: Lantao Jin <[email protected]>

RyanL1997 · 2025-10-01T01:39:28Z

https://github.com/opensearch-project/sql/actions/runs/18148240816/job/51654012596?pr=4378#step:6:481

CalciteNoPushdownIT > org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions FAILED
REPRODUCE WITH: ./gradlew ':integ-test:integTest' --tests 'org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions' -Dtests.seed=FD9C48B5A496F6A0 -Dtests.security.manager=false -Dtests.locale=ff-SN -Dtests.timezone=America/Santo_Domingo -Druntime.java=21
    java.lang.AssertionError: expected:<0> but was:<1>
        at __randomizedtesting.SeedInfo.seed([FD9C48B5A496F6A0:5AB6F276A0DA7BAD]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at org.opensearch.sql.util.MatcherUtils.verifyNumOfRows(MatcherUtils.java:188)
        at org.opensearch.sql.ppl.SearchCommandIT.testSearchWithComplexChainedExpressions(SearchCommandIT.java:1316)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:1583)

RyanL1997 · 2025-10-01T01:53:58Z

this is the same failure on #4344

Signed-off-by: Lantao Jin <[email protected]>

LantaoJin · 2025-10-01T02:06:47Z

https://github.com/opensearch-project/sql/actions/runs/18148240816/job/51654012596?pr=4378#step:6:481

CalciteNoPushdownIT > org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions FAILED
REPRODUCE WITH: ./gradlew ':integ-test:integTest' --tests 'org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions' -Dtests.seed=FD9C48B5A496F6A0 -Dtests.security.manager=false -Dtests.locale=ff-SN -Dtests.timezone=America/Santo_Domingo -Druntime.java=21
    java.lang.AssertionError: expected:<0> but was:<1>
        at __randomizedtesting.SeedInfo.seed([FD9C48B5A496F6A0:5AB6F276A0DA7BAD]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at org.opensearch.sql.util.MatcherUtils.verifyNumOfRows(MatcherUtils.java:188)
        at org.opensearch.sql.ppl.SearchCommandIT.testSearchWithComplexChainedExpressions(SearchCommandIT.java:1316)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:1583)

@RyanL1997 @Swiddis , confirmed with @yuancu , seems a test bug triggered at Oct 1st. To unblock our release, I have ignored it in the last commit and @yuancu will look into it offline.

LantaoJin · 2025-10-01T02:19:31Z

Reminder: Do not backport it to 2.19-dev since it calls an API only provided in OS core 3.3.0.

penghuo · 2025-10-01T15:49:25Z

Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?

I think all PIT search requests should add _shard_doc as tiebreaker, or the _score will used as implicit sort field in response. But this PR doesn't check all code base, it just changes the all places which use _doc + _id as tiebreaker to _shard_doc.

I checked PR of _shard_doc, seems only PIT search requests can sort by _shard_doc. Non-PIT search requests sort by _shard_doc throws exception. I will refactor this PR to force v2 and v3 only. The v1 (legacy) code is out of my knowledge, I won't touch them in this PR.

For search=index query, What is performance difference sort[_doc, _id], vs sort[_shard_doc] ?
Should we use _shard_docs as tiebreaker only?

Change the default search sorting to _shard_doc

8151247

Signed-off-by: Lantao Jin <[email protected]>

LantaoJin added v3.3.0 enhancement New feature or request labels Sep 25, 2025

dai-chen reviewed Sep 26, 2025

View reviewed changes

LantaoJin added 2 commits September 28, 2025 14:42

refactor

c43d0c3

Signed-off-by: Lantao Jin <[email protected]>

typo

fe0efac

Signed-off-by: Lantao Jin <[email protected]>

LantaoJin changed the title ~~Change the default search sorting to _shard_doc~~ Change the default search sort tiebreaker to _shard_doc for PIT search Sep 28, 2025

LantaoJin added 3 commits September 28, 2025 18:06

Fix ITs

d8140cf

Signed-off-by: Lantao Jin <[email protected]>

Merge remote-tracking branch 'upstream/main' into issues/3064

e885030

Signed-off-by: Lantao Jin <[email protected]>

Fix explain IT

7fe7015

Signed-off-by: Lantao Jin <[email protected]>

LantaoJin marked this pull request as ready for review September 29, 2025 11:26

LantaoJin requested review from ps48, kavithacm, derek-ho, joshuali925, YANG-DB, mengweieric, vamsimanohar, Swiddis, penghuo, seankao-az, MaxKsyunz, Yury-Fridlyand, anirudha, forestmvey, acarbonetto, GumpacG and ykmr1224 as code owners September 29, 2025 11:26

LantaoJin requested review from qianheng-aws, yuancu and RyanL1997 as code owners September 29, 2025 11:26

LantaoJin added 2 commits September 29, 2025 22:34

Merge remote-tracking branch 'upstream/main' into issues/3064

ec436fe

Merge remote-tracking branch 'upstream/main' into issues/3064

cc532c0

Signed-off-by: Lantao Jin <[email protected]>

LantaoJin commented Sep 30, 2025

View reviewed changes

Swiddis previously approved these changes Sep 30, 2025

View reviewed changes

Swiddis mentioned this pull request Sep 30, 2025

Change the default search sorting to _shard_doc #4420

Closed

8 tasks

Merge remote-tracking branch 'upstream/main' into issues/3064

494d964

Fix conflicts

6f4ba2b

Signed-off-by: Lantao Jin <[email protected]>

LantaoJin dismissed Swiddis’s stale review via 6f4ba2b October 1, 2025 01:24

RyanL1997 mentioned this pull request Oct 1, 2025

[Enhancement] Add error handling for known limitation of sql JOIN #4344

Merged

8 tasks

Swiddis previously approved these changes Oct 1, 2025

View reviewed changes

RyanL1997 previously approved these changes Oct 1, 2025

View reviewed changes

Ignore the incorrect IT

0d93c14

Signed-off-by: Lantao Jin <[email protected]>

LantaoJin dismissed stale reviews from RyanL1997 and Swiddis via 0d93c14 October 1, 2025 01:59

Swiddis approved these changes Oct 1, 2025

View reviewed changes

LantaoJin enabled auto-merge (squash) October 1, 2025 02:28

yuancu approved these changes Oct 1, 2025

View reviewed changes

LantaoJin merged commit 3e95147 into opensearch-project:main Oct 1, 2025
33 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Change the default search sort tiebreaker to `_shard_doc` for PIT search #4378

Change the default search sort tiebreaker to `_shard_doc` for PIT search #4378

Uh oh!

LantaoJin commented Sep 25, 2025 •

edited

Loading

Uh oh!

LantaoJin commented Sep 26, 2025

Uh oh!

dai-chen left a comment

Uh oh!

LantaoJin commented Sep 28, 2025 •

edited

Loading

Uh oh!

LantaoJin commented Sep 30, 2025

Uh oh!

LantaoJin Sep 30, 2025

Uh oh!

LantaoJin Sep 30, 2025 •

edited

Loading

Uh oh!

Swiddis commented Sep 30, 2025

Uh oh!

LantaoJin commented Oct 1, 2025

Uh oh!

RyanL1997 commented Oct 1, 2025

Uh oh!

RyanL1997 commented Oct 1, 2025

Uh oh!

LantaoJin commented Oct 1, 2025

Uh oh!

LantaoJin commented Oct 1, 2025 •

edited

Loading

Uh oh!

Uh oh!

penghuo commented Oct 1, 2025 •

edited

Loading

Uh oh!

Uh oh!

Change the default search sort tiebreaker to _shard_doc for PIT search #4378

Change the default search sort tiebreaker to _shard_doc for PIT search #4378

Uh oh!

Conversation

LantaoJin commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

LantaoJin commented Sep 26, 2025

Uh oh!

dai-chen left a comment

Choose a reason for hiding this comment

Uh oh!

LantaoJin commented Sep 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LantaoJin commented Sep 30, 2025

Uh oh!

LantaoJin Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

LantaoJin Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Swiddis commented Sep 30, 2025

Uh oh!

LantaoJin commented Oct 1, 2025

Uh oh!

RyanL1997 commented Oct 1, 2025

Uh oh!

RyanL1997 commented Oct 1, 2025

Uh oh!

LantaoJin commented Oct 1, 2025

Uh oh!

LantaoJin commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

penghuo commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Change the default search sort tiebreaker to `_shard_doc` for PIT search #4378

Change the default search sort tiebreaker to `_shard_doc` for PIT search #4378

LantaoJin commented Sep 25, 2025 •

edited

Loading

LantaoJin commented Sep 28, 2025 •

edited

Loading

LantaoJin Sep 30, 2025 •

edited

Loading

LantaoJin commented Oct 1, 2025 •

edited

Loading

penghuo commented Oct 1, 2025 •

edited

Loading