Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Backport 2.x] Add spark connector #1809

Merged
merged 4 commits into from
Jul 11, 2023
Merged

Conversation

opensearch-trigger-bot[bot]
Copy link
Contributor

Backport a816a58 from #1780

* Create Spark Connector

Signed-off-by: Vamsi Manohar <[email protected]>

* Add spark client and engine

Signed-off-by: Rupal Mahajan <[email protected]>

* Remove vars

Signed-off-by: Rupal Mahajan <[email protected]>

* Spark connector draft

Signed-off-by: Rupal Mahajan <[email protected]>

* nit

Signed-off-by: Rupal Mahajan <[email protected]>

* Fix checkstyle errors

Signed-off-by: Rupal Mahajan <[email protected]>

* nit

Signed-off-by: Rupal Mahajan <[email protected]>

* Fix license header

Signed-off-by: Rupal Mahajan <[email protected]>

* Add spark storage test

Signed-off-by: Rupal Mahajan <[email protected]>

* Update comments

Signed-off-by: Rupal Mahajan <[email protected]>

* Fix checkstyle in comments

Signed-off-by: Rupal Mahajan <[email protected]>

* Update tests

Signed-off-by: Rupal Mahajan <[email protected]>

* Address PR comments

Signed-off-by: Rupal Mahajan <[email protected]>

* Refactor class name

Signed-off-by: Rupal Mahajan <[email protected]>

* Address PR comment

Signed-off-by: Rupal Mahajan <[email protected]>

---------

Signed-off-by: Vamsi Manohar <[email protected]>
Signed-off-by: Rupal Mahajan <[email protected]>
Co-authored-by: Vamsi Manohar <[email protected]>
(cherry picked from commit a816a58)
@codecov
Copy link

codecov bot commented Jul 5, 2023

Codecov Report

Merging #1809 (bbe929d) into 2.x (776a76c) will increase coverage by 0.01%.
The diff coverage is 100.00%.

❗ Current head bbe929d differs from pull request most recent head c4aebdd. Consider uploading reports for the commit c4aebdd to get more accurate results

@@             Coverage Diff              @@
##                2.x    #1809      +/-   ##
============================================
+ Coverage     97.35%   97.37%   +0.01%     
- Complexity     4443     4475      +32     
============================================
  Files           388      394       +6     
  Lines         10997    11065      +68     
  Branches        784      789       +5     
============================================
+ Hits          10706    10774      +68     
  Misses          284      284              
  Partials          7        7              
Flag Coverage Δ
sql-engine 97.37% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...implementation/SparkSqlFunctionImplementation.java 100.00% <100.00%> (ø)
...ctions/resolver/SparkSqlTableFunctionResolver.java 100.00% <100.00%> (ø)
...nctions/scan/SparkSqlFunctionTableScanBuilder.java 100.00% <100.00%> (ø)
...ensearch/sql/spark/storage/SparkStorageEngine.java 100.00% <100.00%> (ø)
...nsearch/sql/spark/storage/SparkStorageFactory.java 100.00% <100.00%> (ø)
...a/org/opensearch/sql/spark/storage/SparkTable.java 100.00% <100.00%> (ø)

@penghuo penghuo merged commit a3d9bfc into 2.x Jul 11, 2023
19 of 20 checks passed
@github-actions github-actions bot deleted the backport/backport-1780-to-2.x branch July 11, 2023 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants