Add Databricks 14.3 Support [DO NOT MERGE] #11467
Conversation
razajafri
changed the title
Signed-off-by: Raza Jafri <[email protected]>
| elif spark_version.startswith('3.5'): | ||
| spark_prefix = '----ws_3_5' | ||
| mvn_prefix = '--mvn' |
There was a problem hiding this comment.
nit: looks like a pretty stable pattern emerged for 3.3, 3.4, 3.5
jenkins/databricks/install_deps.py
Outdated
| @@ -125,15 +128,15 @@ def define_deps(spark_version, scala_version): | |||
| Artifact('com.fasterxml.jackson.core', 'jackson-annotations', | |||
| f'{prefix_ws_sp_mvn_hadoop}--com.fasterxml.jackson.core--jackson-annotations--com.fasterxml.jackson.core__jackson-annotations__*.jar'), | |||
| Artifact('org.apache.spark', f'spark-avro_{scala_version}', | |||
| f'{spark_prefix}--vendor--avro--avro-*.jar'), | |||
| f'{prefix_ws_sp_mvn_hadoop}--org.apache.avro--avro--org.apache.avro*.jar'), | |||
There was a problem hiding this comment.
is it really an unconditional change
gerashegalov
changed the title
|
build |
Signed-off-by: timl <[email protected]>
| @@ -0,0 +1,25 @@ | |||
| /* | |||
| * Copyright (c) 2023, NVIDIA CORPORATION. | |||
There was a problem hiding this comment.
Copyright 2024 for this folder and other files
|
@razajafri Added a commit into your PR to support the pre-merge job against Databricks-14.3 runtime As we're setting up nightly build and test CICD for the new branch-24.12, will first support DB-14.3 pre-merge as long as the branch-24.12 pipelines are ready before Chinese National Day holiday. We'll support the nightly build/test for DB14.3 after holiday, thanks! |
razajafri
requested review from
jlowe,
revans2,
tgravescs and
GaryShen2008
as code owners
There was a problem hiding this comment.
spark-rapids-private.version should point to 24.12.0-SNAPSHOT
Line 726 in 2036f16
Did you up-merge ?
| @@ -91,7 +91,7 @@ pipeline { | |||
| // 'name' and 'value' only supprt literal string in the declarative Jenkins | |||
| // Refer to Jenkins issue https://issues.jenkins.io/browse/JENKINS-62127 | |||
| name 'DB_RUNTIME' | |||
| values '11.3', '12.2', '13.3' | |||
| values '11.3', '12.2', '13.3', '14.3' | |||
There was a problem hiding this comment.
Will need to drop support for 11.3 Databricks runtime @razajafri chan you help to remove 11.3 here like
values '11.3', '12.2', '13.3', '14.3' --> values '12.2', '13.3', '14.3'
There was a problem hiding this comment.
Removing 11.3 should be a part of a different PR.
This PR is not going to be merged as I am breaking it into smaller PRs
razajafri
changed the title
This is a PR that adds Databricks 14.3 support to the Plugin but this PR shouldn't be merged as it will be broken into smaller PRs.