Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DO NOT REVIEW] Run ucx shuffle smoke tests under scala 2.13 #11105

Draft
wants to merge 3 commits into
base: branch-24.12
Choose a base branch
from
Draft

[DO NOT REVIEW] Run ucx shuffle smoke tests under scala 2.13 #11105

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7336da2
Run ucx shuffle smoke tests under scala 2.13
abellina Jun 28, 2024
401bdb8
Force spark 3.3.0 for scala 2.13
abellina Jun 28, 2024
5fa9758
Cleanup spark-premerge-build slightly
abellina Jun 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions jenkins/spark-premerge-build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,9 @@ ci_scala213() {

# Download a Scala 2.13 version of Spark
prepare_spark 3.3.0 2.13
# Match the shim version to the Spark we are using for integration tests
# in order to pick the correct shuffle manager in our shuffle smoke tests
export SHUFFLE_SPARK_SHIM=spark330

# build Scala 2.13 versions
for version in "${SPARK_SHIM_VERSIONS_PREMERGE_SCALA213[@]}"
Expand All @@ -203,6 +206,7 @@ ci_scala213() {

$MVN_CMD -U -B $MVN_URM_MIRROR clean package $MVN_BUILD_ARGS -DskipTests=true
cd .. # Run integration tests in the project root dir to leverage test cases and resource files

export TEST_TAGS="not premerge_ci_1"
export TEST_TYPE="pre-commit"
# SPARK_HOME (and related) must be set to a Spark built with Scala 2.13
Expand All @@ -214,6 +218,8 @@ ci_scala213() {
# export 'LC_ALL' to set locale with UTF-8 so regular expressions are enabled
SPARK_HOME=$SPARK_HOME PYTHONPATH=$PYTHONPATH \
LC_ALL="en_US.UTF-8" TEST="regexp_test.py" ./integration_tests/run_pyspark_from_build.sh

rapids_shuffle_smoke_test
}

prepare_spark() {
Expand Down