Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add _routing to SQL includes list (#277) #1771

Merged
merged 10 commits into from
Jul 11, 2023

Conversation

acarbonetto
Copy link
Collaborator

Description

Add the _routing reserved word as a metafield accessible as an identifier from any opensearch index.

Example:

select _id, _index, _routing, int0 from calcs limit 5

Example response:

        [
            "1",
            "calcs",
            "[xoyIAOdqQsS74x2yKSgiEQ][calcs][0]",
            1
        ],

Issues Resolved

Partially resolves: sql#1478

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: acarbonetto <[email protected]>
Yury-Fridlyand
Yury-Fridlyand previously approved these changes Jun 23, 2023
Copy link
Collaborator

@Yury-Fridlyand Yury-Fridlyand left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need an IT to prove that filtering by _routing works fine, for example:

select *, _id from <index> where _routing = 'a'
select *, _id from <index> where _routing = 'b'

The test should propagate the data properly into different shards and get their names using REST API.

@codecov
Copy link

codecov bot commented Jun 23, 2023

Codecov Report

Merging #1771 (1c0b35d) into main (fa840e0) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##               main    #1771   +/-   ##
=========================================
  Coverage     97.32%   97.32%           
- Complexity     4458     4459    +1     
=========================================
  Files           388      388           
  Lines         11050    11052    +2     
  Branches        790      791    +1     
=========================================
+ Hits          10754    10756    +2     
  Misses          289      289           
  Partials          7        7           
Flag Coverage Δ
sql-engine 97.32% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...search/sql/opensearch/storage/OpenSearchIndex.java 100.00% <ø> (ø)
...ch/sql/opensearch/response/OpenSearchResponse.java 100.00% <100.00%> (ø)

@acarbonetto
Copy link
Collaborator Author

I think we need an IT to prove that filtering by _routing works fine, for example:

select *, _id from <index> where _routing = 'a'
select *, _id from <index> where _routing = 'b'

The test should propagate the data properly into different shards and get their names using REST API.

Right now routing shard names are randomly assigned, so I cannot filter by shard name. Yet.
In the following PR, I will be able to assign documents to specific shards, and will be able to filter by routing shard using exactly what you were suggesting.

Note: this is also why a doctest with _routing doesn't work, because the documents are assigned to random shards.

@Yury-Fridlyand
Copy link
Collaborator

Yes, but you can get shard names with REST API or CAT API

@acarbonetto acarbonetto changed the title Integ metafields routing Add _routing to SQL includes list (#277) Jun 23, 2023
Signed-off-by: acarbonetto <[email protected]>
Signed-off-by: acarbonetto <[email protected]>
dai-chen
dai-chen previously approved these changes Jun 26, 2023
Copy link
Collaborator

@dai-chen dai-chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes!

@acarbonetto acarbonetto dismissed stale reviews from dai-chen and Yury-Fridlyand via ede3956 June 27, 2023 20:58
@acarbonetto acarbonetto added the v2.9.0 v2.9.0 label Jul 4, 2023
@MaxKsyunz MaxKsyunz merged commit 876a9c8 into opensearch-project:main Jul 11, 2023
13 of 15 checks passed
@MaxKsyunz MaxKsyunz deleted the integ-metafields-routing branch July 11, 2023 06:32
opensearch-trigger-bot bot pushed a commit that referenced this pull request Jul 11, 2023
* Add _routing to SQL includes list

Signed-off-by: Andrew Carbonetto <[email protected]>

* Update IT index

Signed-off-by: Andrew Carbonetto <[email protected]>

* Update doctest

Signed-off-by: acarbonetto <[email protected]>

* Add WHERE clause IT tests

Signed-off-by: acarbonetto <[email protected]>

* Fix IT test

Signed-off-by: acarbonetto <[email protected]>

* Update documentation for _routing

Signed-off-by: acarbonetto <[email protected]>

---------

Signed-off-by: Andrew Carbonetto <[email protected]>
Signed-off-by: acarbonetto <[email protected]>
(cherry picked from commit 876a9c8)
penghuo pushed a commit that referenced this pull request Jul 11, 2023
* Add _routing to SQL includes list

Signed-off-by: Andrew Carbonetto <[email protected]>

* Update IT index

Signed-off-by: Andrew Carbonetto <[email protected]>

* Update doctest

Signed-off-by: acarbonetto <[email protected]>

* Add WHERE clause IT tests

Signed-off-by: acarbonetto <[email protected]>

* Fix IT test

Signed-off-by: acarbonetto <[email protected]>

* Update documentation for _routing

Signed-off-by: acarbonetto <[email protected]>

---------

Signed-off-by: Andrew Carbonetto <[email protected]>
Signed-off-by: acarbonetto <[email protected]>
(cherry picked from commit 876a9c8)

Co-authored-by: Andrew Carbonetto <[email protected]>
opensearch-trigger-bot bot pushed a commit that referenced this pull request Jul 11, 2023
* Add _routing to SQL includes list

Signed-off-by: Andrew Carbonetto <[email protected]>

* Update IT index

Signed-off-by: Andrew Carbonetto <[email protected]>

* Update doctest

Signed-off-by: acarbonetto <[email protected]>

* Add WHERE clause IT tests

Signed-off-by: acarbonetto <[email protected]>

* Fix IT test

Signed-off-by: acarbonetto <[email protected]>

* Update documentation for _routing

Signed-off-by: acarbonetto <[email protected]>

---------

Signed-off-by: Andrew Carbonetto <[email protected]>
Signed-off-by: acarbonetto <[email protected]>
(cherry picked from commit 876a9c8)
Yury-Fridlyand pushed a commit that referenced this pull request Jul 11, 2023
* Add _routing to SQL includes list

Signed-off-by: Andrew Carbonetto <[email protected]>

* Update IT index

Signed-off-by: Andrew Carbonetto <[email protected]>

* Update doctest

Signed-off-by: acarbonetto <[email protected]>

* Add WHERE clause IT tests

Signed-off-by: acarbonetto <[email protected]>

* Fix IT test

Signed-off-by: acarbonetto <[email protected]>

* Update documentation for _routing

Signed-off-by: acarbonetto <[email protected]>

---------

Signed-off-by: Andrew Carbonetto <[email protected]>
Signed-off-by: acarbonetto <[email protected]>
(cherry picked from commit 876a9c8)

Co-authored-by: Andrew Carbonetto <[email protected]>
MitchellGale pushed a commit to Bit-Quill/opensearch-project-sql that referenced this pull request Jul 11, 2023
* Add _routing to SQL includes list

Signed-off-by: Andrew Carbonetto <[email protected]>

* Update IT index

Signed-off-by: Andrew Carbonetto <[email protected]>

* Update doctest

Signed-off-by: acarbonetto <[email protected]>

* Add WHERE clause IT tests

Signed-off-by: acarbonetto <[email protected]>

* Fix IT test

Signed-off-by: acarbonetto <[email protected]>

* Update documentation for _routing

Signed-off-by: acarbonetto <[email protected]>

---------

Signed-off-by: Andrew Carbonetto <[email protected]>
Signed-off-by: acarbonetto <[email protected]>
Signed-off-by: Mitchell Gale <[email protected]>
MitchellGale pushed a commit to Bit-Quill/opensearch-project-sql that referenced this pull request Jul 11, 2023
* Add _routing to SQL includes list

Signed-off-by: Andrew Carbonetto <[email protected]>

* Update IT index

Signed-off-by: Andrew Carbonetto <[email protected]>

* Update doctest

Signed-off-by: acarbonetto <[email protected]>

* Add WHERE clause IT tests

Signed-off-by: acarbonetto <[email protected]>

* Fix IT test

Signed-off-by: acarbonetto <[email protected]>

* Update documentation for _routing

Signed-off-by: acarbonetto <[email protected]>

---------

Signed-off-by: Andrew Carbonetto <[email protected]>
Signed-off-by: acarbonetto <[email protected]>
Signed-off-by: Mitchell Gale <[email protected]>
MitchellGale pushed a commit to Bit-Quill/opensearch-project-sql that referenced this pull request Jul 20, 2023
* Add _routing to SQL includes list

Signed-off-by: Andrew Carbonetto <[email protected]>

* Update IT index

Signed-off-by: Andrew Carbonetto <[email protected]>

* Update doctest

Signed-off-by: acarbonetto <[email protected]>

* Add WHERE clause IT tests

Signed-off-by: acarbonetto <[email protected]>

* Fix IT test

Signed-off-by: acarbonetto <[email protected]>

* Update documentation for _routing

Signed-off-by: acarbonetto <[email protected]>

---------

Signed-off-by: Andrew Carbonetto <[email protected]>
Signed-off-by: acarbonetto <[email protected]>
Signed-off-by: Mitchell Gale <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow individual shards to be targeted during query execution [FEATURE]
7 participants