[frontend] Getting Unauthorized error while trying to retrive pod/log information from kubeflow UI

[harsha1163]

Environment

AKS Cluster

• How did you deploy Kubeflow Pipelines (KFP)?

• KFP version: 2.1.0

Steps to reproduce

While trying to check the pod information getting "unauthorized" error. After restart of the ml-pipeline-ui deployment, the authorization error is dissappearing.

Expected result

Should show up the pod info if the pod exists other wise should show pod not found error.

Materials and Reference

---

Impacted by this bug? Give it a 👍.

[vishalmishra369]

hey if this problem has not been taken by anyone can i work on it ?
If yes then please do assign it to me !!.

regards,
vishal

[harsha1163]

@vishalmishra369,

Any idea what could be the reason of unauthorized error?

We have installed kubeflow in an AKS cluster with "Cilium" as network policy. Unable to figure out what is the issue. Is it due to Cilium or some other thing. We have the same setup in AKS with Azure as network policy, there we are not facing any issues.

Any guess will be highly appreciated. Thanks.

[juliusvonkohout]

Please read the whole readme first, especially the istio / cilium part https://github.com/kubeflow/manifests#istio and make sure you are on a clean Kubeflow 1.9.1 install. It works with cilium. Also check your Dex and oauth2-proxy settings.

[harsha1163]

@juliusvonkohout, thanks for pointing to the document. Actually I am using kubeflow bundled with Deploykf. I am installing using deploykf distribution.

I went through the istio/cilium part in the above pointed page, there were two parameters which are mentioned to change in cilium config.

bpf-lb-sock-hostns-only: "true"
cni-exclusive: "false"

I am using "Azure CNI powered by Cilium" and the cilium is installed on the AKS cluster automatically by Azure. Not explicitly installing using helm.

After checking the config, out of two parameters, bpf-lb-sock-hostns-only: "true" --> this parameter is already there as expected.

but cni-exclusive: "false" this parameter is not there and is not part of cilium config. Azure is not giving an option to modify the CNI and suppress Clilum. I suspect that the authorization issue is due to that the cilium is taking precedence over istio.

Is there any other way to resolve this issue?

I can install cilium with out using "Azure CNI powered by Cilium" as network policy, but installing on my own will not get any support from azure if there are any issues. it would be good to have support from azure as we are planning to expand this in production.

[juliusvonkohout]

@juliusvonkohout, thanks for pointing to the document. Actually I am using kubeflow bundled with Deploykf. I am installing using deploykf distribution.

I went through the istio/cilium part in the above pointed page, there were two parameters which are mentioned to change in cilium config.

bpf-lb-sock-hostns-only: "true" cni-exclusive: "false"

I am using "Azure CNI powered by Cilium" and the cilium is installed on the AKS cluster automatically by Azure. Not explicitly installing using helm.

After checking the config, out of two parameters, bpf-lb-sock-hostns-only: "true" --> this parameter is already there as expected.

but cni-exclusive: "false" this parameter is not there and is not part of cilium config. Azure is not giving an option to modify the CNI and suppress Clilum. I suspect that the authorization issue is due to that the cilium is taking precedence over istio.

Is there any other way to resolve this issue?

I can install cilium with out using "Azure CNI powered by Cilium" as network policy, but installing on my own will not get any support from azure if there are any issues. it would be good to have support from azure as we are planning to expand this in production.

Then please contact the provider of deployKF, this is not the issue tracker for deployKF specific problems. If you install from https://github.com/kubeflow/manifests or https://github.com/kubeflow/pipelines directly we might be able to help you, but this topic seems to be very AKS and DeployKF specific.

/close

[google-oss-prow]

@juliusvonkohout: Closing this issue.

In response to this:

@juliusvonkohout, thanks for pointing to the document. Actually I am using kubeflow bundled with Deploykf. I am installing using deploykf distribution.

I went through the istio/cilium part in the above pointed page, there were two parameters which are mentioned to change in cilium config.

bpf-lb-sock-hostns-only: "true" cni-exclusive: "false"

I am using "Azure CNI powered by Cilium" and the cilium is installed on the AKS cluster automatically by Azure. Not explicitly installing using helm.

After checking the config, out of two parameters, bpf-lb-sock-hostns-only: "true" --> this parameter is already there as expected.

but cni-exclusive: "false" this parameter is not there and is not part of cilium config. Azure is not giving an option to modify the CNI and suppress Clilum. I suspect that the authorization issue is due to that the cilium is taking precedence over istio.

Is there any other way to resolve this issue?

I can install cilium with out using "Azure CNI powered by Cilium" as network policy, but installing on my own will not get any support from azure if there are any issues. it would be good to have support from azure as we are planning to expand this in production.

Then please contact the provider of deployKF, this is not the issue tracker for deployKF specific problems. If you install from https://github.com/kubeflow/manifests or https://github.com/kubeflow/pipelines directly we might be able to help you, but this topic seemsto be very AKS and DeployKF specific.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.