kubeflow trainer shouldn't expect living in `kubeflow-system`

[christian-heusel]

What happened?

When following the example at https://www.kubeflow.org/docs/components/trainer/getting-started in an installation from the Kubeflow Community Distribution at some point the following error arises:

from kubeflow.trainer import TrainerClient, KubernetesBackendConfig

client = TrainerClient(backend_config=KubernetesBackendConfig(namespace="example"))

for r in client.list_runtimes():
    print(f"Runtime: {r.name}")

Trainer control-plane version info is not available: unable to read 'kubeflow_trainer_version' from ConfigMap 'kubeflow-trainer-public' in namespace 'kubeflow-system': (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'ede3d47e-0593-40c7-84a7-065651479f8f', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'd8d1b44e-a124-4309-9e68-01915f929a6f', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'ce7ac3cf-9053-4526-b182-8f3346ff2ffe', 'Date': 'Thu, 18 Jun 2026 12:43:21 GMT', 'Content-Length': '427'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"kubeflow-trainer-public\" is forbidden: User \"system:serviceaccount:example:default-editor\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kubeflow-system\": Azure does not have opinion for this user.","reason":"Forbidden","details":{"name":"kubeflow-trainer-public","kind":"configmaps"},"code":403}
Runtime: deepspeed-distributed
Runtime: jax-distributed
Runtime: mlx-distributed
Runtime: torch-distributed
Runtime: torchtune-llama3.2-1b
Runtime: torchtune-llama3.2-3b
Runtime: torchtune-qwen2.5-1.5b
Runtime: xgboost-distributed

This is due to the code snippet here:

sdk/kubeflow/trainer/backends/kubernetes/backend.py

Lines 63 to 85 in c247a8d

	def verify_backend(self) -> None:
	"""Verify that the Trainer control plane exposes version metadata.
	This check only ensures that the public control-plane ConfigMap exists
	and contains a ``kubeflow_trainer_version`` field. It does not
	enforce version compatibility and never raises.
	"""
	system_namespace = os.getenv("KUBEFLOW_SYSTEM_NAMESPACE", "kubeflow-system")
	config_map_name = "kubeflow-trainer-public"
	try:
	_ = self.core_api.read_namespaced_config_map(
	name=config_map_name,
	namespace=system_namespace,
	).data["kubeflow_trainer_version"]
	except Exception as e: # noqa: BLE001
	logger.warning(
	"Trainer control-plane version info is not available: "
	f"unable to read 'kubeflow_trainer_version' from ConfigMap "
	f"'{config_map_name}' in namespace '{system_namespace}': {e}"
	)
	return

What did you expect to happen?

When following the getting started tutorial there is no error.

Environment

Kubernetes version:

$ kubectl version
Client Version: v1.31.6
Kustomize Version: v5.4.2
Server Version: v1.34.2
WARNING: version difference between client (1.31) and server (1.34) exceeds the supported minor version skew of +/-1

Kubeflow Trainer version:

$ kubectl get pods -n kubeflow -l app.kubernetes.io/name=trainer -o jsonpath="{.items[*].spec.containers[*].image}"
ghcr.io/kubeflow/trainer/trainer-controller-manager:v2.2.0

Kubeflow Python SDK version:

$ pip show kubeflow
Name: kubeflow
Version: 0.4.0
Summary: Kubeflow Python SDK to manage ML workloads and to interact with Kubeflow APIs.
Home-page: https://github.com/kubeflow/sdk
Author: 
Author-email: The Kubeflow Authors <kubeflow-discuss@googlegroups.com>
License-Expression: Apache-2.0
Location: /opt/conda/lib/python3.11/site-packages
Requires: kubeflow-katib-api, kubeflow-trainer-api, kubernetes, pydantic
Required-by:

Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

[github-actions]

🎉 Welcome to the Kubeflow SDK! 🎉

Thanks for opening your first issue! We're happy to have you as part of our community 🚀

Here's what happens next:

• Our team will review your issue soon! cc @kubeflow/kubeflow-sdk-team
• If you'd like to contribute to this issue, check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards

Join the community:

• Slack: Join our #kubeflow-ml-experience and #kubeflow-trainer Slack channels
• Meetings: Attend the Kubeflow SDK and ML Experience bi-weekly meetings

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[andreyvelich]

Thanks for creating this @christian-heusel!
As we discussed here, all users should be able to access this ConfigMap: kubeflow/trainer#3083 (comment)
Can you check if the appropriate RoleBinding and Role is created in kubeflow-system namespace?

cc @astefanutti @sameerdattav @tenzen-y @robert-bell

/remove-label needs-triage

[christian-heusel]

The issue is not that the RoleBinding is missing or similar (that works as expected), but that the user has to add a export KUBEFLOW_SYSTEM_NAMESPACE=kubeflow when using the SDK in the context of the kubeflow community distribution where the app ends up in the namespace kubeflow:

https://github.com/kubeflow/trainer/blob/1fee1164a14e3b16969494193cb34681834d325f/manifests/overlays/kubeflow-platform/kustomization.yaml#L3

kustomize build applications/trainer/upstream/overlays/kubeflow-platform/ | grep -A10 -i "kind: Deployment" | grep "namespace"
  namespace: kubeflow

However there is another target in the community distribution that patches this again (but that only works when using their overlay), which I missed (and which should probably be upstreamed):
https://github.com/kubeflow/community-distribution/blob/c46d109729fe4393328b129a1b23a13fbbd26a80/applications/trainer/overlays/kustomization.yaml#L3

In any case, I think it is very much non-obvious why the above error appears and there should atleast be a hint to the user that they can modify this via the environment variable KUBEFLOW_SYSTEM_NAMESPACE without reading the code.

[andreyvelich]

We have a pending issue that we should use kubeflow-system namespace to deploy all Kubeflow control plane:
kubeflow/community-distribution#2937
IIRC, we've already deployed Trainer control plane in that namespace, right @juliusvonkohout ?

[juliusvonkohout]

We also have for example kubeflow-workspaces. I think that kubeflow-trainer, kubeflow-pipelines etc. can be better long-term. But right now it is kubeflow-system for some and for other kubeflow.

[andreyvelich]

Any particular reason we want to separate control plane between different namespaces @kubeflow/wg-notebooks-chairs?
I believe it would be much simpler for platform administrators to manage all system components if they were located in the kubeflow-system namespace.

[juliusvonkohout]

There are up and downsides to it. It allows for better isolation for example and would make the standalone deployments easier. But it is also a preference in the end. We anyway have oauth2-proxy, dex, istio-sytem and many more namespaces.