[SPARK-50946][CORE][TESTS] Add version check for Java 17.0.14 to make `UtilsSuite` test pass #49599

LuciferYang · 2025-01-22T07:09:38Z

What changes were proposed in this pull request?

This pr adds Java version checks and logic adaptations to scenario6 in the test case named "SPARK-35907: createDirectory" within the UtilsSuite, ensuring that it can pass the tests both Java 17.0.13 and Java 17.0.14 as well.

Why are the changes needed?

Java 17.0.14 includes the fix for JDK-8294193: Files.createDirectories throws FileAlreadyExistsException for a symbolic link whose target is an existing directory. This behaves differently from previous expectations in the targeted test scenario, which would cause the "SPARK-35907: createDirectory" test to fail.

However, since setup-java does not always install Java 17.0.14 (sometimes it installs 17.0.13 instead now), the current pr adapts the logic for versions after 17.0.14 and retains the original assertion behavior for versions 17.0.13 and earlier.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass GitHub Actions
manual check

Before

java -version
openjdk version "17.0.14" 2025-01-21 LTS
OpenJDK Runtime Environment Zulu17.56+15-CA (build 17.0.14+7-LTS)
OpenJDK 64-Bit Server VM Zulu17.56+15-CA (build 17.0.14+7-LTS, mixed mode, sharing)

or

java -version
openjdk version "17.0.14" 2025-01-21 LTS
OpenJDK Runtime Environment Corretto-17.0.14.7.1 (build 17.0.14+7-LTS)
OpenJDK 64-Bit Server VM Corretto-17.0.14.7.1 (build 17.0.14+7-LTS, mixed mode, sharing)

"SPARK-35907: createDirectory" will test failed:

[info] - SPARK-35907: createDirectory *** FAILED *** (43 milliseconds)
[info]   Utils.createDirectory(scenario6) was true (UtilsSuite.scala:534)
[info]   org.scalatest.exceptions.TestFailedException:
[info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
[info]   at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
[info]   at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
[info]   at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
[info]   at org.apache.spark.util.UtilsSuite.$anonfun$new$35(UtilsSuite.scala:534)
[info]   at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
[info]   at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
[info]   at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
[info]   at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
[info]   at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:69)
[info]   at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:155)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
[info]   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:227)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
[info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
[info]   at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:69)
[info]   at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
[info]   at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
[info]   at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:69)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
[info]   at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
[info]   at scala.collection.immutable.List.foreach(List.scala:334)
[info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
[info]   at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
[info]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
[info]   at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
[info]   at org.scalatest.Suite.run(Suite.scala:1114)
[info]   at org.scalatest.Suite.run$(Suite.scala:1096)
[info]   at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
[info]   at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
[info]   at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:69)
[info]   at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
[info]   at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info]   at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info]   at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info]   at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info]   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info]   at java.base/java.lang.Thread.run(Thread.java:840)

After

Both 17.0.13 and 17.0.14 will pass the test successfully.

Was this patch authored or co-authored using generative AI tooling?

No

LuciferYang · 2025-01-22T08:59:14Z

cc @HyukjinKwon @dongjoon-hyun FYI

MaxGekk · 2025-01-22T10:21:09Z

core/src/test/scala/org/apache/spark/util/UtilsSuite.scala

@@ -526,10 +526,18 @@ class UtilsSuite extends SparkFunSuite with ResetSystemProperties {

    // The following 3 scenarios are only for the method: createDirectory(File)
    // 6. Symbolic link
+    lazy val (majorVersion: Int, patchVersion: Int) = {
+      // JAVA_RUNTIME_VERSION is like "17.0.14+7-LTS"
+      val splits = SystemUtils.JAVA_RUNTIME_VERSION.split("\\.")


Can't you use some standard methods for parsing, for instance:

scala> Runtime.Version.parse("17.0.14+7-LTS").feature val res3: Int = 17 scala> Runtime.Version.parse("17.0.14+7-LTS").update val res5: Int = 14

Good advice, thank you @MaxGekk ~ updated

MaxGekk · 2025-01-22T15:26:23Z

+1, LGTM. Merging to master/4.0.
Thank you, @LuciferYang.

… `UtilsSuite` test pass ### What changes were proposed in this pull request? This pr adds Java version checks and logic adaptations to `scenario6` in the test case named "SPARK-35907: createDirectory" within the `UtilsSuite`, ensuring that it can pass the tests both Java 17.0.13 and Java 17.0.14 as well. ### Why are the changes needed? Java 17.0.14 includes the fix for JDK-8294193: [`Files.createDirectories throws FileAlreadyExistsException for a symbolic link whose target is an existing directory`](openjdk/jdk17u-dev#2870). This behaves differently from previous expectations in the targeted test scenario, which would cause the "SPARK-35907: createDirectory" test to fail. ![image](https://github.com/user-attachments/assets/21d33f5c-d29a-4def-ba84-41475b231a63) However, since `setup-java` does not always install Java 17.0.14 (sometimes it installs 17.0.13 instead now), the current pr adapts the logic for versions after 17.0.14 and retains the original assertion behavior for versions 17.0.13 and earlier. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GitHub Actions - manual check **Before** ``` java -version openjdk version "17.0.14" 2025-01-21 LTS OpenJDK Runtime Environment Zulu17.56+15-CA (build 17.0.14+7-LTS) OpenJDK 64-Bit Server VM Zulu17.56+15-CA (build 17.0.14+7-LTS, mixed mode, sharing) ``` or ``` java -version openjdk version "17.0.14" 2025-01-21 LTS OpenJDK Runtime Environment Corretto-17.0.14.7.1 (build 17.0.14+7-LTS) OpenJDK 64-Bit Server VM Corretto-17.0.14.7.1 (build 17.0.14+7-LTS, mixed mode, sharing) ``` "SPARK-35907: createDirectory" will test failed: ``` [info] - SPARK-35907: createDirectory *** FAILED *** (43 milliseconds) [info] Utils.createDirectory(scenario6) was true (UtilsSuite.scala:534) [info] org.scalatest.exceptions.TestFailedException: [info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472) [info] at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471) [info] at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231) [info] at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295) [info] at org.apache.spark.util.UtilsSuite.$anonfun$new$35(UtilsSuite.scala:534) [info] at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127) [info] at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282) [info] at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231) [info] at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230) [info] at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:69) [info] at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:155) [info] at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) [info] at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) [info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) [info] at org.scalatest.Transformer.apply(Transformer.scala:22) [info] at org.scalatest.Transformer.apply(Transformer.scala:20) [info] at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226) [info] at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:227) [info] at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224) [info] at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236) [info] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306) [info] at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236) [info] at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218) [info] at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:69) [info] at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234) [info] at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227) [info] at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:69) [info] at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269) [info] at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413) [info] at scala.collection.immutable.List.foreach(List.scala:334) [info] at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401) [info] at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396) [info] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475) [info] at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269) [info] at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268) [info] at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564) [info] at org.scalatest.Suite.run(Suite.scala:1114) [info] at org.scalatest.Suite.run$(Suite.scala:1096) [info] at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564) [info] at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273) [info] at org.scalatest.SuperEngine.runImpl(Engine.scala:535) [info] at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273) [info] at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272) [info] at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:69) [info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213) [info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) [info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) [info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69) [info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321) [info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517) [info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414) [info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [info] at java.base/java.lang.Thread.run(Thread.java:840) ``` **After** Both 17.0.13 and 17.0.14 will pass the test successfully. ### Was this patch authored or co-authored using generative AI tooling? No Closes #49599 from LuciferYang/SPARK-50946. Authored-by: yangjie01 <[email protected]> Signed-off-by: Max Gekk <[email protected]> (cherry picked from commit 54a59b7) Signed-off-by: Max Gekk <[email protected]>

LuciferYang · 2025-01-22T15:59:46Z

Thanks @MaxGekk

dongjoon-hyun · 2025-01-22T16:31:28Z

Thank you, @LuciferYang and @MaxGekk .

… `UtilsSuite` test pass ### What changes were proposed in this pull request? This pr adds Java version checks and logic adaptations to `scenario6` in the test case named "SPARK-35907: createDirectory" within the `UtilsSuite`, ensuring that it can pass the tests both Java 17.0.13 and Java 17.0.14 as well. ### Why are the changes needed? Java 17.0.14 includes the fix for JDK-8294193: [`Files.createDirectories throws FileAlreadyExistsException for a symbolic link whose target is an existing directory`](openjdk/jdk17u-dev#2870). This behaves differently from previous expectations in the targeted test scenario, which would cause the "SPARK-35907: createDirectory" test to fail. ![image](https://github.com/user-attachments/assets/21d33f5c-d29a-4def-ba84-41475b231a63) However, since `setup-java` does not always install Java 17.0.14 (sometimes it installs 17.0.13 instead now), the current pr adapts the logic for versions after 17.0.14 and retains the original assertion behavior for versions 17.0.13 and earlier. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GitHub Actions - manual check **Before** ``` java -version openjdk version "17.0.14" 2025-01-21 LTS OpenJDK Runtime Environment Zulu17.56+15-CA (build 17.0.14+7-LTS) OpenJDK 64-Bit Server VM Zulu17.56+15-CA (build 17.0.14+7-LTS, mixed mode, sharing) ``` or ``` java -version openjdk version "17.0.14" 2025-01-21 LTS OpenJDK Runtime Environment Corretto-17.0.14.7.1 (build 17.0.14+7-LTS) OpenJDK 64-Bit Server VM Corretto-17.0.14.7.1 (build 17.0.14+7-LTS, mixed mode, sharing) ``` "SPARK-35907: createDirectory" will test failed: ``` [info] - SPARK-35907: createDirectory *** FAILED *** (43 milliseconds) [info] Utils.createDirectory(scenario6) was true (UtilsSuite.scala:534) [info] org.scalatest.exceptions.TestFailedException: [info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472) [info] at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471) [info] at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231) [info] at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295) [info] at org.apache.spark.util.UtilsSuite.$anonfun$new$35(UtilsSuite.scala:534) [info] at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127) [info] at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282) [info] at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231) [info] at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230) [info] at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:69) [info] at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:155) [info] at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) [info] at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) [info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) [info] at org.scalatest.Transformer.apply(Transformer.scala:22) [info] at org.scalatest.Transformer.apply(Transformer.scala:20) [info] at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226) [info] at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:227) [info] at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224) [info] at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236) [info] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306) [info] at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236) [info] at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218) [info] at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:69) [info] at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234) [info] at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227) [info] at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:69) [info] at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269) [info] at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413) [info] at scala.collection.immutable.List.foreach(List.scala:334) [info] at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401) [info] at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396) [info] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475) [info] at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269) [info] at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268) [info] at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564) [info] at org.scalatest.Suite.run(Suite.scala:1114) [info] at org.scalatest.Suite.run$(Suite.scala:1096) [info] at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564) [info] at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273) [info] at org.scalatest.SuperEngine.runImpl(Engine.scala:535) [info] at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273) [info] at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272) [info] at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:69) [info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213) [info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) [info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) [info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69) [info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321) [info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517) [info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414) [info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [info] at java.base/java.lang.Thread.run(Thread.java:840) ``` **After** Both 17.0.13 and 17.0.14 will pass the test successfully. ### Was this patch authored or co-authored using generative AI tooling? No Closes #49599 from LuciferYang/SPARK-50946. Authored-by: yangjie01 <[email protected]> Signed-off-by: Max Gekk <[email protected]> (cherry picked from commit 54a59b7) Signed-off-by: Max Gekk <[email protected]> (cherry picked from commit 89ec420) Signed-off-by: Dongjoon Hyun <[email protected]>

dongjoon-hyun · 2025-01-22T16:53:29Z

I backported this to branch-3.5 too.

dongjoon-hyun · 2025-01-22T17:28:06Z

Oops. I tested with Java 17 only when I backport to branch-3.5. Let me check with Java 8.

… `UtilsSuite` test pass ### What changes were proposed in this pull request? This pr adds Java version checks and logic adaptations to `scenario6` in the test case named "SPARK-35907: createDirectory" within the `UtilsSuite`, ensuring that it can pass the tests both Java 17.0.13 and Java 17.0.14 as well. ### Why are the changes needed? Java 17.0.14 includes the fix for JDK-8294193: [`Files.createDirectories throws FileAlreadyExistsException for a symbolic link whose target is an existing directory`](openjdk/jdk17u-dev#2870). This behaves differently from previous expectations in the targeted test scenario, which would cause the "SPARK-35907: createDirectory" test to fail. ![image](https://github.com/user-attachments/assets/21d33f5c-d29a-4def-ba84-41475b231a63) However, since `setup-java` does not always install Java 17.0.14 (sometimes it installs 17.0.13 instead now), the current pr adapts the logic for versions after 17.0.14 and retains the original assertion behavior for versions 17.0.13 and earlier. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GitHub Actions - manual check **Before** ``` java -version openjdk version "17.0.14" 2025-01-21 LTS OpenJDK Runtime Environment Zulu17.56+15-CA (build 17.0.14+7-LTS) OpenJDK 64-Bit Server VM Zulu17.56+15-CA (build 17.0.14+7-LTS, mixed mode, sharing) ``` or ``` java -version openjdk version "17.0.14" 2025-01-21 LTS OpenJDK Runtime Environment Corretto-17.0.14.7.1 (build 17.0.14+7-LTS) OpenJDK 64-Bit Server VM Corretto-17.0.14.7.1 (build 17.0.14+7-LTS, mixed mode, sharing) ``` "SPARK-35907: createDirectory" will test failed: ``` [info] - SPARK-35907: createDirectory *** FAILED *** (43 milliseconds) [info] Utils.createDirectory(scenario6) was true (UtilsSuite.scala:534) [info] org.scalatest.exceptions.TestFailedException: [info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472) [info] at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471) [info] at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231) [info] at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295) [info] at org.apache.spark.util.UtilsSuite.$anonfun$new$35(UtilsSuite.scala:534) [info] at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127) [info] at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282) [info] at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231) [info] at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230) [info] at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:69) [info] at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:155) [info] at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) [info] at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) [info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) [info] at org.scalatest.Transformer.apply(Transformer.scala:22) [info] at org.scalatest.Transformer.apply(Transformer.scala:20) [info] at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226) [info] at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:227) [info] at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224) [info] at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236) [info] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306) [info] at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236) [info] at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218) [info] at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:69) [info] at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234) [info] at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227) [info] at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:69) [info] at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269) [info] at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413) [info] at scala.collection.immutable.List.foreach(List.scala:334) [info] at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401) [info] at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396) [info] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475) [info] at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269) [info] at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268) [info] at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564) [info] at org.scalatest.Suite.run(Suite.scala:1114) [info] at org.scalatest.Suite.run$(Suite.scala:1096) [info] at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564) [info] at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273) [info] at org.scalatest.SuperEngine.runImpl(Engine.scala:535) [info] at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273) [info] at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272) [info] at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:69) [info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213) [info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) [info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) [info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69) [info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321) [info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517) [info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414) [info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [info] at java.base/java.lang.Thread.run(Thread.java:840) ``` **After** Both 17.0.13 and 17.0.14 will pass the test successfully. ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#49599 from LuciferYang/SPARK-50946. Authored-by: yangjie01 <[email protected]> Signed-off-by: Max Gekk <[email protected]> (cherry picked from commit 54a59b7) Signed-off-by: Max Gekk <[email protected]> (cherry picked from commit 89ec420) Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit bd2b8e5) Signed-off-by: Dongjoon Hyun <[email protected]>

dongjoon-hyun · 2025-01-22T17:49:31Z

I made a backporting PR to branch-3.5 by replacing Runtime.Version (Java 9+)

[SPARK-50946][CORE][TESTS][3.5] Add version check for Java 17.0.14 to make UtilsSuite test pass #49606

init

3a23dc2

github-actions bot added the CORE label Jan 22, 2025

LuciferYang changed the title ~~[SPARK-50946][CORE][TESTS] Add patch version check for Java 17.0.14 to make UtilsSuite test pass~~ [SPARK-50946][CORE][TESTS] Add version check for Zulu 17.0.14 to make UtilsSuite test pass Jan 22, 2025

LuciferYang marked this pull request as draft January 22, 2025 08:05

check 17

cbd7549

LuciferYang changed the title ~~[SPARK-50946][CORE][TESTS] Add version check for Zulu 17.0.14 to make UtilsSuite test pass~~ [SPARK-50946][CORE][TESTS] Add version check for Java 17.0.14 to make UtilsSuite test pass Jan 22, 2025

LuciferYang marked this pull request as ready for review January 22, 2025 08:58

MaxGekk reviewed Jan 22, 2025

View reviewed changes

address comments

84d8c57

MaxGekk approved these changes Jan 22, 2025

View reviewed changes

MaxGekk closed this in 54a59b7 Jan 22, 2025

LuciferYang mentioned this pull request Jan 22, 2025

[SPARK-50909][PYTHON] Setup faulthandler in PythonPlannerRunners #49592

Closed

dongjoon-hyun mentioned this pull request Jan 22, 2025

[SPARK-50918][ML][PYTHON][CONNECT] Support Pipeline on connect #49596

Closed

LuciferYang deleted the SPARK-50946 branch January 24, 2025 03:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-50946][CORE][TESTS] Add version check for Java 17.0.14 to make `UtilsSuite` test pass #49599

[SPARK-50946][CORE][TESTS] Add version check for Java 17.0.14 to make `UtilsSuite` test pass #49599

LuciferYang commented Jan 22, 2025 •

edited

Loading

LuciferYang commented Jan 22, 2025 •

edited

Loading

MaxGekk Jan 22, 2025

LuciferYang Jan 22, 2025 •

edited

Loading

MaxGekk commented Jan 22, 2025

LuciferYang commented Jan 22, 2025

dongjoon-hyun commented Jan 22, 2025

dongjoon-hyun commented Jan 22, 2025

dongjoon-hyun commented Jan 22, 2025

dongjoon-hyun commented Jan 22, 2025

[SPARK-50946][CORE][TESTS] Add version check for Java 17.0.14 to make UtilsSuite test pass #49599

[SPARK-50946][CORE][TESTS] Add version check for Java 17.0.14 to make UtilsSuite test pass #49599

Conversation

LuciferYang commented Jan 22, 2025 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

LuciferYang commented Jan 22, 2025 • edited Loading

MaxGekk Jan 22, 2025

Choose a reason for hiding this comment

LuciferYang Jan 22, 2025 • edited Loading

Choose a reason for hiding this comment

MaxGekk commented Jan 22, 2025

LuciferYang commented Jan 22, 2025

dongjoon-hyun commented Jan 22, 2025

dongjoon-hyun commented Jan 22, 2025

dongjoon-hyun commented Jan 22, 2025

dongjoon-hyun commented Jan 22, 2025

[SPARK-50946][CORE][TESTS] Add version check for Java 17.0.14 to make `UtilsSuite` test pass #49599

[SPARK-50946][CORE][TESTS] Add version check for Java 17.0.14 to make `UtilsSuite` test pass #49599

LuciferYang commented Jan 22, 2025 •

edited

Loading

LuciferYang commented Jan 22, 2025 •

edited

Loading

LuciferYang Jan 22, 2025 •

edited

Loading