Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Drop spark31x shims [databricks] #11159

Merged
merged 9 commits into from
Jul 12, 2024
Merged

Drop spark31x shims [databricks] #11159

merged 9 commits into from
Jul 12, 2024

Commits on Jul 9, 2024

  1. Remove spark31x json lines and shim files 

    1, Remove spark31x json lines from the source code

2, Remove the files those only for spark31x shims

3, Move the files for spark31x and spark32x+ shims into sql-plugin/src/main/spark320 folder

Signed-off-by: Tim Liu <[email protected]>
    @NvTimLiu
    NvTimLiu committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    bdaa7f7 View commit details
    Browse the repository at this point in the history

  2. Drop spark31x shims in the build scripts and pom files 

    Signed-off-by: Tim Liu <[email protected]>
    @NvTimLiu
    NvTimLiu committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    79e95c1 View commit details
    Browse the repository at this point in the history

  3. Restore the accidentally deleted file: OrcStatisticShim.scala 

        tests/src/test/spark311/scala/com/nvidia/spark/rapids/shims/OrcStatisticShim.scala
     -->
    tests/src/test/spark321cdh/scala/com/nvidia/spark/rapids/shims/OrcStatisticShim.scala

check if we chan merge this file into?

    tests/src/test/spark320/scala/com/nvidia/spark/rapids/shims/OrcStatisticShim.scala
Signed-off-by: Tim Liu <[email protected]>
    @NvTimLiu
    NvTimLiu committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    eaa4871 View commit details
    Browse the repository at this point in the history

  4. Update Copyright to 2024 

    Signed-off-by: Tim Liu <[email protected]>
    @NvTimLiu
    NvTimLiu committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    ecae7c5 View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2024

  1. Remove the 31x in ShimLoader.scala according to the review comments 

    Signed-off-by: Tim Liu <[email protected]>
    @NvTimLiu
    NvTimLiu committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    e6dd67f View commit details
    Browse the repository at this point in the history

  2. Update the file scala2.13/pom.xml 

    Signed-off-by: Tim Liu <[email protected]>
    @NvTimLiu
    NvTimLiu committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    ae1b138 View commit details
    Browse the repository at this point in the history

  3. Drop 3.1.x shims in docs, source code and build scripts 

        Change the default shim to spark320 from spark311
    in the shims in docs, source code and build scripts

Signed-off-by: Tim Liu <[email protected]>
    @NvTimLiu
    NvTimLiu committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    878af85 View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2024

  1. Updating the docs for the dropping 31x shims 

    Signed-off-by: Tim Liu <[email protected]>
    @NvTimLiu
    NvTimLiu committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    38ed9ae View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2024

  1. Clean up unused and duplicated 'org/roaringbitmap' folder 

    To fix: NVIDIA#11175

Clean up unused and duplicated 'org/roaringbitmap' in the spark320 shim folder to walk around for the JACOCO error 'different class with same name', after we drop 31x shims and change the default shim to spark320

Signed-off-by: Tim Liu <[email protected]>
    @NvTimLiu
    NvTimLiu committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    cf859d1 View commit details
    Browse the repository at this point in the history