Spark on Kubernetes job fails if service account doesn't have permission to all namespaces.

[ashokkumarrathore]

In a multi tenant Kubernetes cluster, it will not always be the case that livy service account has access to all namespaces. Since Livy currently looks for job in all namespaces, it fails if the permission is not there for some namespace in cluster.

Ideally we should only look for job in the namespace it was submitted to. Relevant code is here, i think :

incubator-livy/server/src/main/scala/org/apache/livy/utils/SparkKubernetesApp.scala

Line 446 in 1f6bd7a

withRetry(kubernetesClient.getApplications().find(_.getApplicationTag.contains(appTag)))

Also this,

incubator-livy/server/src/main/scala/org/apache/livy/utils/SparkKubernetesApp.scala

Line 689 in 1f6bd7a

client.pods.inAnyNamespace

[ashokkumarrathore]

@askhatri @jahstreet fyi

[askhatri]

Thank you @ashokkumarrathore for the findings. @jahstreet, please review and share your insights.

[ashokkumarrathore]

The potential fix is to provide a namespace argument and use that in GetApplications() from Kubernetes. So, rather than looking for app in any namespace here, it should be namespaced.

The namespace need to be passed from here:

incubator-livy/server/src/main/scala/org/apache/livy/server/batch/BatchSession.scala

Line 109 in 1f6bd7a

SparkApp.create(appTag, None, Option(sparkSubmit), livyConf, Option(s))

There are couple of options on how we do this:

1. Add a namespace argument to SparkApp.create() call. This changes the create() function definition which is generic for both YarnApp and KubernetesApp.
2. To avoid modifying create() call, we can put namespace as part of LivyConf and pass a copy of this rather than actual ref. This will have minimal changes but technically namespace should not be part of LivyConf.

@jahstreet @askhatri Please let me know your thoughts and i can implement it accordingly.

[askhatri]

Hi @ashokkumarrathore,
Thank you for your efforts. I agree with the suggested fix.
CC: @jahstreet , @gyogal

[ashokkumarrathore]

@askhatri @jahstreet Changes for namespace support in Livy for Spark on k8s. #462
Can you guys please take a look?
Also, currently no builds are running for this, can you please approve the workflow to run build/test?