ci(docker): set legacy IPC format to fix e2e tests

[SajidAlamQB]

Description

Related to: #1170

Fixes CI e2e test failure for kedro-docker with Spark support. The test "Execute docker build and run using spark Dockerfile" was failing when PySpark attempted to convert DataFrames to Pandas using Apache Arrow. The error occurred because Arrow couldn't access sun.misc.Unsafe due to JVM restrictions.

Development notes

Updated kedro_docker/template/Dockerfile.spark to add JVM configuration environment variables
Added --add-opens flags to allow Apache Arrow access:

java.base/java.nio - for buffer operations
java.base/sun.nio.ch - for channel implementations
java.base/jdk.internal.misc - for access to sun.misc.Unsafe

Checklist

• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Updated jsonschema/kedro-catalog-X.XX.json if necessary
• Added a description of this change in the relevant RELEASE.md file
• Added tests to cover my changes
• Received approvals from at least half of the TSC (required for adding a new, non-experimental dataset)

[deepyaman]

Thanks for addressing this long-running issue!

[deepyaman]

Sorry, I actually have a question. 😅

[SajidAlamQB]

Can we get this merged in for now to unblock this, I will create a follow up issue to dig further into this as I don't have capacity at the moment to dig into this further but the current fix does resolve the immediate issue.

We will investigate this further in a follow up issue.