Adding spark plugin to support spark jobs over EMR on EKS

[babourine]

No description provided.

[Copilot]

Pull Request Overview

A new spark plugin has been added to support the submission and execution of Spark jobs over EMR on EKS. Key changes include:

• Introducing a spark plugin handler in plugins/spark/spark.go that wraps the internal spark command logic.
• Implementing the spark job submission flow with support for AWS EMR containers and assume role credentials in internal/pkg/object/command/spark/spark.go.
• Updating go.mod to include necessary AWS SDK dependencies for EMR containers and S3 manager.

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
plugins/spark/spark.go	Added a new plugin handler wrapping the spark command implementation.
internal/pkg/object/command/spark/spark.go	Implements the spark job execution flow including context unmarshalling, AWS client setup, and job polling.
go.mod	Updated module dependencies to include the AWS EMR containers service and S3 manager.

Comments suppressed due to low confidence (1)

plugins/spark/spark.go:1

• Consider renaming the package from 'main' to 'spark' to maintain consistency with the plugin's directory structure and its intended usage.

package main

[Copilot]

Consider adding a nil check for vc.Name before dereferencing it to avoid potential runtime panics in cases where the cluster name may be missing.

Suggested change

	for _, vc := range outputListClusters.VirtualClusters {
	for _, vc := range outputListClusters.VirtualClusters {
	if vc.Name == nil {
	continue
	}