diff --git a/projects/kubeflow/adopter-interviews/kubeflow-adopter-interview-adopter-1.md b/projects/kubeflow/adopter-interviews/kubeflow-adopter-interview-adopter-1.md new file mode 100644 index 000000000..58073fbc5 --- /dev/null +++ b/projects/kubeflow/adopter-interviews/kubeflow-adopter-interview-adopter-1.md @@ -0,0 +1,139 @@ +# Kubeflow Adopter Interview - Adopter 1 + +**Organization**: Adopter 1 (anonymized) +**Industry**: Financial services / banking +**Interview Date**: June 10, 2026 +**Interviewers (TOC)**: Faseela K + + +## Organization Intro + +Adopter 1 maintains a machine learning platform for internal data scientists where thousands of data scientists, machine learning engineers, and business analysts train and serve machine learning models. The organization has extremely strict regulatory requirements that shape how it evaluates and adopts open-source technology. + + +## 1. How long has your organization used the project? + +Adopter 1 has been using Kubeflow for approximately **4-4.5 years**, including an initial research/evaluation period before going to production. + + +## 2. What were the main motivations to adopt the project and which key features do you use today? + +Before Kubeflow, Adopter 1 was assembling a collection of otherwise disconnected products and effectively rebuilding the same functionality piece by piece. The team decided to stop reinventing the wheel and adopt Kubeflow as a unified ML platform. + +Key features currently in use: **Notebooks, Pipelines, Trainer, Katib** - all components of Kubeflow are used. Adopter 1 is a heavy user of Pipelines specifically, with plans to add ADOPTERS.md entries for other subprojects beyond the existing Pipelines entry. + + +## 3. Compared with other products and projects in this space (proprietary and open), what drew you to the project? + +Two primary differentiators: + +1. **Regulatory compliance and cloud agnosticism**: Adopter 1 operates under extremely strict regulatory requirements that preclude many managed cloud services. Kubeflow's open-source, self-hosted, cloud-agnostic nature (with full deployment control) was a key differentiator. +2. **Full ML lifecycle coverage**: While alternative tools exist for individual components (notebook provisioners, workflow orchestrators, distributed compute abstractions), nothing else consolidates the full model development lifecycle into one ecosystem - no comparable alternative covers the full scope. + + +## 4. What is the current level of usage (pre-production, production) and scale? + +**Level**: Production (both experimentation and production workloads) +**Daily active users**: several thousand (non-prod), with additional production usage +**Workloads**: large numbers of concurrent workflows; business-critical ML models are trained and served on Kubeflow +**Scale**: large distributed compute workloads + + +## 5. What version is used and what is your update cadence with the project? + +Adopter 1 tracks the Kubeflow Community Distribution releases closely and updates **every 1-3 months**. + +**Upgrade experience**: Adopter 1 experienced regressions during the Kubeflow v2 release cycle and contributed many fixes directly upstream. That period is considered an anomaly; the project is now considered to be in much better hands with improved upgrade processes for v3. + + +## 6. Can you walk me through the experience of adopting or integrating Kubeflow? What challenges did you experience? + +Integration with existing infrastructure was manageable. Adopter 1 is very familiar with Istio, so Kubeflow's Istio dependency was never a blocker. + +The primary challenges arose from unique security and regulatory requirements - gaps that upstream Kubeflow did not address out of the box. Addressing those gaps directly led the organization to become significant contributors and maintainers to the project, with multiple notable upstream contributions to Kubeflow Pipelines. + + +## 7. Did you find the information in the repo or the project docs valuable to your implementation? + +Yes. Both the official docs and the generated SDK docs were described as excellent resources: + +- Official docs: https://www.kubeflow.org/docs/components/pipelines/ +- SDK docs: https://kubeflow-pipelines.readthedocs.io/en/sdk-2.16.1/ + +Adopter 1 used these extensively as the basis for creating its own internal documentation for its user base. Some upstream docs may be complex for newcomers, but they were reliable and accurate for the team's implementation needs. + + +## 8. Did you need to engage with the community members or maintainers? + +Yes, extensively. Adopter 1 communicates with maintainers primarily through **CNCF Slack**. Maintainers are described as very responsive. The community is characterized as open, friendly, and welcoming. + +Adopter 1's level of engagement goes well beyond typical end-user interaction: + +- Multiple contributors are active in the project +- Multiple maintainers participate, including at least one in Pipelines +- Active participation in **community calls** +- Presentations at KubeCon + CloudNativeCon and SREcon conferences, including topics covering lessons learned from consumer to contributor, managing upstream forks, and Kubeflow in cloud-native AI +- Adopter 1 has hosted Kubeflow community events + + +## 9. Has your implementation of the project provided measurable value? + +Yes. Some of the organization's highest-value ML models are trained and served via Kubeflow. Specific metrics cannot be shared due to confidentiality, but the team reports: + +- Elimination of the fragmented, manually assembled toolchain that preceded Kubeflow +- Significant productivity gains for thousands of data scientists and ML engineers +- Much of the value generated has been generalized and contributed back upstream + + +## 10. If the project were to be archived, what level of difficulty would your organization experience to remove it? + +High difficulty. No single alternative provides the integrated ML lifecycle coverage that Kubeflow does. Individual components could be replaced (e.g., Raytrain for distributed training), but nothing currently consolidates the full ecosystem. + +Given the scale (thousands of users, high workflow volume, critical production ML models), full removal would be a significant multi-year effort. The organization has existing maintainer investment in the project and would likely step further into a maintainer/steward role rather than abandon the project. + + +## 11. Is there something that holds the project back from reaching its ultimate potential? + +- **Maintainer depth**: Maintainer coverage for Pipelines has improved under the current shared ownership model. This has worked well for the ecosystem, with materially higher contribution velocity and more new contributors than in earlier periods. Expanding the maintainer base across subprojects remains an active priority as the project continues to grow. +- **Helm support**: Helm chart support for deployment would be a meaningful improvement; the organization uses Helm internally and minimizes forks by upstreaming changes where possible. +- **UX for non-engineer users**: A simplified UX targeting data scientists and business analysts (rather than engineers) would broaden adoption and reduce onboarding friction. + + +## 12. In your opinion, what could the project improve? + +1. **More frequent releases**: the current cadence can be too slow for organizations with aggressive security patching requirements; a regulated environment sometimes requires faster vulnerability response than upstream timelines allow, leading to internal forks +2. **Security response process**: a security best practices document for end users would be valuable beyond existing `SECURITY.md` coverage (Kubeflow is designed to be "secure by default"; that design pattern should be better documented) +3. **Maintainer growth**: actively recruiting maintainers across more subprojects would strengthen the ecosystem + + +## 13. What are the overall strengths of the project? + +- **Ecosystem integration**: Kubeflow brings together an otherwise disconnected set of ML tools in a complex problem domain and provides a mostly unified experience +- **Vendor neutrality**: Truly cloud-agnostic and vendor-neutral, a key factor for regulated industries like financial services +- **Community**: Very open, friendly, well-managed, and inclusive; lots of growth opportunities; transparent governance and maintainer ladder +- **Governance**: Well-organized, diverse, and inclusive; the maintainer ladder is transparent and clearly documented +- **Regulatory fit**: Uniquely suited for organizations with strict compliance requirements due to full deployment control + + +## 14. Do you have any future plans regarding the project? More involvement, feature requests, expansion, etc.? + +- **Maintainer growth**: Plans to onboard more contributors, members, and eventually more maintainers across additional Kubeflow subprojects +- **Future KFP releases**: Looking toward contributing to upcoming KFP releases +- **Reference architecture**: Interest in creating a reference architecture for the banking/financial services sector +- **ADOPTERS.md expansion**: Plans to add entries for Trainer, Katib, Notebooks, Spark Operator, Dashboard, and Model Registry in addition to the existing Pipelines entry + + +## Maturity Level Survey + +1. **Do you feel you have a good understanding of the meaning of each CNCF maturity level?** + + Yes. Both interviewees are well-versed in the CNCF maturity framework and the criteria distinguishing Sandbox, Incubating, and Graduated projects. + +2. **Is there information missing regarding the meaning of each different level?** + + The levels are well-understood. The Director noted that Kubeflow's graduation would increase industry credibility, suggesting that the Graduated designation carries meaningful signal for enterprise and regulated-industry adopters evaluating project maturity. + +3. **Do you rely on those levels internally in any way, and if yes how?** + + Yes. CNCF maturity level is a factor in the organization's technology evaluation process. Graduation status signals production readiness, governance health, and long-term sustainability - all of which are particularly important for a regulated financial institution that cannot easily change core infrastructure components. + diff --git a/projects/kubeflow/adopter-interviews/kubeflow-adopter-interview-cern.md b/projects/kubeflow/adopter-interviews/kubeflow-adopter-interview-cern.md new file mode 100644 index 000000000..819b14779 --- /dev/null +++ b/projects/kubeflow/adopter-interviews/kubeflow-adopter-interview-cern.md @@ -0,0 +1,159 @@ +# Kubeflow Adopter Interview - CERN + +**Organization**: CERN (European Organization for Nuclear Research) +**Industry**: Fundamental scientific research / high-energy physics +**Interview Date**: June 8, 2026 +**Interviewers (TOC)**: Faseela K +**Interviewees**: Ricardo Rocha, Raulian Chiorescu (CERN) +**Reference Architecture**: https://architecture.cncf.io/architectures/cern-scientific-computing/ + + +## Organization Intro + +CERN is the European Organization for Nuclear Research, one of the world's largest scientific research centres. It operates the Large Hadron Collider and runs fundamental physics research, employing thousands of scientists and engineers worldwide. CERN runs significant computing infrastructure to process and analyze data from physics experiments, making it a large-scale Kubernetes and cloud-native operator. + + +## 1. How long has your organization used the project? + +CERN began evaluating Kubeflow in 2019-2020. Limited production usage started in 2021, initially focused on notebooks, distributed training and model serving via KServe. Usage expanded in 2024 following an upgrade to Kubeflow 1.9, which added Notebooks and broader KServe adoption to the production workload. + + +## 2. What were the main motivations to adopt the project and which key features do you use today? + +CERN had home-built systems for ML workloads but needed a single solution that covered the majority of use cases. Kubeflow offered that breadth. Key features currently in use: + +- **Notebooks**: interactive access to GPUs for data scientists and ML researchers +- **Model Serving (KServe)**: most popular feature at CERN +- **Distributed Training**: large-scale model training workloads +- **Hyperparameter Optimization (Katib)**: still actively used +- **Pipelines**: ML workflow orchestration, and other workflow automation + + +## 3. Compared with other products and projects in this space (proprietary and open), what drew you to the project? + +Kubeflow was selected because it fills a unique gap in the ecosystem: it bridges the cloud-native (Kubernetes) world with the higher-level ML tooling world. CERN evaluated other options including home-built systems, but Kubeflow's breadth of coverage across the ML lifecycle made it the strongest fit. Its Kubernetes-native design meant integration with CERN's existing infrastructure was relatively easy at the infrastructure level. + + +## 4. What is the current level of usage (pre-production, production) and scale? + +**Level**: Production +**Users**: Hundreds of users on the primary instance; one instance is expected to scale to thousands of users +**GPU Capacity**: Hundreds of GPUs currently; expected to roughly double by end of year +**Reference**: https://architecture.cncf.io/architectures/cern-scientific-computing/ + + +## 5. What version of the project is currently in use and what is your update cadence with the project? + +Currently running **Kubeflow 1.9.2**. CERN is working toward upgrading to 1.11, though the cadence has slowed as the user base has grown. Earlier in the adoption, upgrades happened faster due to a smaller user footprint. Upgrades are sometimes done at the component level (e.g., upgrading KServe independently) rather than as a full platform upgrade. + + +## 6. Can you walk me through the experience of adopting or integrating Kubeflow? What challenges did you experience? + +**Integration** with existing Kubernetes infrastructure was relatively easy because Kubeflow uses standard Kubernetes primitives. + +**Adoption**, however, was not easy at the start: + +- **Installation complexity**: Kubeflow uses Kustomize, not a Helm chart. This is very different from other deployments at CERN. The overlay format changed multiple times in the first two years, delaying upgrades. CERN would strongly prefer a Helm chart; Kustomize-based installation is difficult to teach to newcomers and feels like "a bunch of scripts, not declarative." +- **Istio dependency**: Kubeflow's hard dependency on Istio was a significant challenge. CERN does not otherwise use Istio, and it was unclear why Istio was required rather than a standard ingress. This represented a major learning curve and integration effort. +- **Documentation gaps for service managers**: Upstream documentation is better suited for end users than for the administrators and service managers responsible for operating the platform at scale. + + +## 7. Did you find the information in the repo or the project docs valuable to your implementation? + +From an end-user perspective, CERN has developed its own internal documentation. Users increasingly rely on AI tools to generate configurations, which often fails because the tooling doesn't accurately reflect current Kubeflow docs. Upstream documentation could be improved from a service manager and operator perspective; this was identified as one of the areas where investment would have the most impact. + + +## 8. Did you need to engage with the community members or maintainers? If so, what was the context and outcome? + +Yes, CERN actively engages with the community: + +- **Communication channels used**: CNCF Slack (`#kubeflow-*` channels), online WG meetings, general community call, mailing list +- **KubeCon presence**: CERN volunteers at the Kubeflow booth, conducts live demos, and presents at conferences: + - *KubeCon EU Amsterdam 2026*: Kubeflow as the backbone for GPU access / MLOps + https://kccnceu2026.sched.com/event/3ac486a177945dc278b23b3627edac6b + - *KubeCon EU Amsterdam 2026*: Kubeflow monitoring stack (being contributed upstream) + https://kccnceu2026.sched.com/event/2CW6t/collisions-in-the-dark-illuminating-the-95-of-kubeflow-you-cant-see-amine-lahouel-laura-llinares-cern + (Upstream contribution: https://github.com/kubeflow/manifests/issues/3426) + - *Cloud Native AI + Kubeflow Day Amsterdam 2026*: Running ML challenges on Kubernetes + https://colocatedeventseu2026.sched.com/event/2DY6O/running-ml-challenges-on-kubernetes-part-2-hannes-hansen-paulo-guilherme-pinheiro-pereira-cern + - *Kubeflow Summit London 2025*: Running ML Challenges on Kubeflow + https://colocatedeventseu2025.sched.com/event/8ece7347102293e529d1c764d9971791 + +Community is perceived as easy to interact with and diverse. + + +## 9. Has your implementation of the project provided measurable value? + +Yes, specifically: + +- **Declarative API for ML workloads**: previously very difficult to achieve at CERN's scale +- **Resource sharing** across multiple users and ML workloads on shared GPU infrastructure +- **HPC integration**: Kubeflow now bridges CERN's HPC workloads and cloud-native ML pipelines +- **GPU utilization and cost savings**: better scheduling and multi-user GPU access reduces idle capacity +- **Access democratization**: enabling interactive GPU access for data scientists and ML researchers with no prior Kubernetes knowledge, via the Kubeflow SDK + + +## 10. If the project were to be archived, what level of difficulty would your organization experience to remove it? + +Kubeflow is deeply embedded in CERN's infrastructure. Difficulty would vary by component: + +- **Inference (KServe)**: Relatively easier to migrate; standard OpenAI-compatible APIs mean CERN is not tightly coupled to a specific backend +- **Training, notebooks, and interactive sessions**: Much harder; these are tightly integrated with CERN's GPU allocation model and user workflows +- **Overall**: Given Kubeflow's role as the planned core of CERN's AI strategy for the next 3-5 years, full removal would be a significant multi-year effort. CERN would likely consider stepping into a maintainership and community management role to preserve functionality before accepting full migration to alternatives. + + +## 11. Is there something that holds the project back from reaching its ultimate potential? + +- **Limited vendor ecosystem**: At KubeCon events, CERN primarily sees end users; few vendors or commercial offerings are visibly supporting Kubeflow. Greater vendor visibility would strengthen the project's reputation and adoption by organizations that require commercial support options. +- **Deployment complexity**: Kustomize-based installation adds complexity that can slow down new adopters. CERN has adapted to the existing manifests over time and sees Helm support as the clearest path to lowering the entry barrier. The community is moving in this direction — several subprojects already ship Helm charts, and a GSoC 2026 project is actively working on Helm support for the community distribution. CERN plans to contribute to this effort. Earlier community-driven Helm alternatives (deployKF, treebeard) were also explored but did not reach maturity. +- **Service manager documentation**: Upstream docs are adequate for end users but insufficient for operators managing multi-tenant production deployments. +- **Evolution speed**: Kubeflow has many tightly-coupled components, which means adapting to large AI/ML shifts (e.g., new model paradigms) is inherently slower than for more modular projects. + + +## 12. In your opinion, what could the project improve? + +- **Helm chart for installation**: would dramatically improve adoption by organizations following standard Kubernetes deployment patterns +- **Service manager documentation**: dedicated operator/admin guides are largely missing upstream +- **Security: CVE response across interconnected components**: Kubeflow's interconnected subprojects mean a CVE in a dependency (e.g., KServe) can block the entire platform. A security best practices document for operators would be valuable; the level of isolation between components was poor but is improving over time. +- **Security: inference endpoint multi-tenancy**: currently CERN is affected by an upstream KServe/Knative issue with cross-namespace authorization of inference endpoints: + https://github.com/knative/serving/issues/12533 + (Not yet tracked as a Kubeflow issue, but documented in the manifests repo under "Cross-namespace Access Control" at https://github.com/kubeflow/manifests/blob/master/common/istio/cluster-local-gateway/overlays/m2m-auth/README.md) +- **Pipeline artifact multi-tenancy**: open issue: https://github.com/kubeflow/pipelines/issues/11760 +- **MLMD component maintenance**: The upstream MLMD component is no longer actively maintained and relies on the deprecated `mysql_native_password` authentication method, blocking MySQL backend upgrades at CERN +- **Monitoring stack**: CERN's monitoring stack for Kubeflow is being contributed upstream (https://github.com/kubeflow/manifests/issues/3426) +- **User onboarding via SDK**: The Kubeflow SDK (https://github.com/kubeflow/sdk) is making onboarding easier for data scientists and ML researchers with no Kubernetes background by abstracting Kubernetes concepts into Python; continued investment here would broaden adoption further. + + +## 13. What are the overall strengths of the project? + +- **Ecosystem positioning**: Kubeflow bridges cloud-native Kubernetes and higher-level ML tooling; no direct equivalent covers the same breadth +- **Declarative API for ML workloads**: Enables structured, repeatable ML workflows at scale +- **Multi-user resource sharing**: Enables efficient GPU sharing across teams and user communities +- **Community**: Open, diverse, and accessible; responsive to engagement +- **Breadth of coverage**: Covers the full ML lifecycle from interactive development (Notebooks) through training, HPO (Katib), pipelines, and serving + + +## 14. Do you have any future plans regarding the project? More involvement, feature requests, expansion, etc.? + +- **Current contributions**: 4 CERN team members have contributed upstream, including: + - https://github.com/kubeflow/manifests/pull/3264 + - Monitoring stack contribution (in progress): https://github.com/kubeflow/manifests/issues/3426 +- **Strategic roadmap**: Kubeflow is planned as a core component of CERN's AI strategy for the next 3-5 years, serving as the standard platform for interactive GPU access and ML operations across different scientific communities +- **Scale expansion**: CERN plans to extend Kubeflow-based GPU access beyond the current ML community to thousands of users across other scientific domains +- **Maintainer involvement**: Ricardo and Raulian are open to exploring deeper maintainer involvement in the future as CERN's contribution footprint grows + + +## Maturity Level Survey + +1. **Do you feel you have a good understanding of the meaning of each CNCF maturity level?** + + Yes. Ricardo is a member of the CNCF End User Technical Advisory Board (TAB) and is well-versed in the CNCF project maturity framework, including the criteria and expectations at Sandbox, Incubating, and Graduated levels. + +2. **Is there information missing regarding the meaning of each different level?** + + The definitions are generally well understood. The graduation criteria are clearly documented and the DD process provides transparency into what is evaluated. No significant gaps noted. + +3. **Do you rely on those levels internally in any way, and if yes how?** + + Yes. CERN uses CNCF maturity levels as a signal of project health and long-term viability when evaluating technologies for adoption. Graduation status is a meaningful indicator for infrastructure decisions at CERN - it reflects that a project has demonstrated production readiness, governance maturity, and a sustainable community, all of which are important to an organization running critical scientific infrastructure. + diff --git a/projects/kubeflow/adopter-interviews/kubeflow-adopter-interview-dhl.md b/projects/kubeflow/adopter-interviews/kubeflow-adopter-interview-dhl.md new file mode 100644 index 000000000..8cfb22fc9 --- /dev/null +++ b/projects/kubeflow/adopter-interviews/kubeflow-adopter-interview-dhl.md @@ -0,0 +1,201 @@ +# Kubeflow Adopter Interview - DHL Data & AI + +**Organization**: DHL Data & AI +**Industry**: Logistics / enterprise AI platform +**Interview Date**: June 30, 2026 +**Interviewers (TOC)**: Faseela K, Brandt Keller +**Interviewees**: Julius Von Kohout (Principal Platform Engineer), Dr. Gezim Sejdiu (Chief Data Engineer) + + +## Organization Introduction + +### Can you give us an overview of your organization and what it does? + +DHL Data & AI is the central Data & AI organization within the DHL Group. It provides a portfolio of services including Data Analytics, Operations Research, Machine Learning, AI Platforms & Solutions, Data Engineering, and MLOps Engineering. Its mission is to transform raw data into business value by delivering actionable insights and scalable AI solutions across the DHL Group. + +The team supports the full lifecycle from ideation and experimentation to deployment and operations, and drives group-wide Data & AI governance, technology incubation, enablement, training, and community-building. Solutions span demand forecasting, ETA prediction, network optimization, staff scheduling, customer analytics, finance analytics, GenAI/NLP applications, geocoding, and customs product classification. + +## Motivation + +### Compared with other products in this space (proprietary and open), what drew you to the project? + +Kubeflow stood out for providing a cloud-agnostic ML platform built on Kubernetes. Multiple alternatives were evaluated, including MetaFlow, Databricks, and MLFlow. Kubeflow offered the strongest balance between flexibility, extensibility, scalability, and vendor independence - particularly for multi-cloud deployments where no comparable alternative exists. + +Key factors in the decision: +- Open-source ecosystem and avoidance of vendor lock-in +- Native Kubernetes integration +- Strong support for multi-cloud and hybrid deployments +- Rich ecosystem for experimentation, pipelines, training, and model lifecycle management +- Ability to customize and extend components for enterprise requirements +- Strong alignment with long-term MLOps and platform engineering strategy + +## Usage Scenario + +### How long has your organization used the project? + +DHL Data & AI has been using Kubeflow since 2021 as the foundation of its enterprise Data Science platform. Over this period, the platform evolved from supporting pilot projects to a strategic ML/AI platform serving hundreds of users across the DHL Group. + +### What were the main motivations to adopt the project and which key features do you use today? + +Primary motivations: +1. Standardization of machine learning workflows +2. Scalable model development and deployment +3. Enterprise-grade MLOps capabilities +4. Support for hybrid and multi-cloud architectures +5. Efficient collaboration between data scientists, platform engineers, and business stakeholders + +Currently in active use: +- Kubeflow Notebooks (Notebooks V2 in evaluation) +- Kubeflow Pipelines (KFP V1/V2 in production) +- Kubeflow Dashboard (multi-tenancy workspace management and isolation) +- Kubeflow Spark Operator (managing Apache Spark application lifecycle on Kubernetes) +- GPU-enabled workloads +- Containerized model development +- Experiment tracking and reproducible workflows +- CI/CD integration and MLOps automation +- Resource governance and cost management + +Subprojects on the near-term roadmap: Katib, Trainer, and Hub. + +### What is the current level of usage (pre-production, production) and scale? + +Level: Production + +Scale: +- Several hundred registered platform members and active users per month +- Several hundred onboarded projects and use cases +- Probably several hundred thousand active business end users (drivers, delivery personnel, sorting operations, finance etc.) relying on ML solutions powered by the platform +- Multiple business-critical ML solutions running in production +- Deployments across multi-cloud and hybrid environments + +### What version is used and what is your update cadence with the project? + +Always using the latest Kubeflow Community Distribution. DHL Data & AI uses the community distribution directly. Julius von Kohout is a Community Distribution release maintainer and actively works on the release cadence - in some cases shipping updates the same day as upstream releases. + +### Can you walk me through what your experience was in either adopting it outright or integrating it with your existing services and applications? What challenges did you experience with the project? + +The adoption journey was generally successful. Integration involved existing enterprise identity management systems, security and compliance requirements, multi-cloud infrastructure, monitoring and observability tooling, CI/CD ecosystems, and internal chargeback and governance processes. + +Key challenges from early adoption (with current status): +- **Operational complexity**: Large Kubernetes-based platforms require platform engineering expertise. Has improved substantially with platform maturation. +- **Spark Operator integration (2024)**: Integrating Spark Operator into the community distribution required dedicated work ([community-distribution#2889](https://github.com/kubeflow/community-distribution/pull/2889)). Now resolved and in active production use. +- **Enterprise authentication and authorization (2024)**: OIDC to OAuth2-proxy migration. DHL contributed the solution upstream together with Roche Pharmaceutical, now available to the broader community. +- **Multi-tenancy and enterprise isolation**: Object storage isolation, PSS baseline/restricted by default, network policies, RBAC hardening. +- **Managing upgrades**: Previously complex; now improved with the dedicated [upgrading and extending section](https://github.com/kubeflow/community-distribution#upgrading-and-extending) in the community distribution. + +Current state: operational complexity has improved significantly. The platform is considered security-hardened with the exception of a few remaining multi-tenancy items. + +### Did you find the information in the repository or the project documentation valuable to your implementation? + +Yes. Official documentation, GitHub repositories, community discussions, and implementation examples provided valuable guidance. Documentation has improved notably, with documentation automation now in place. + +A key differentiator of the Community Distribution is that it goes beyond documentation: it provides automation and test-driven development so users can copy verified CI/CD pipelines rather than relying purely on written documentation. + +Particularly useful resources: +- Installation and deployment documentation +- Component architecture overviews +- Kubeflow Pipelines documentation +- Community issue discussions and GitHub Discussions +- Upgrade and migration documentation ([community distribution upgrading section](https://github.com/kubeflow/community-distribution#upgrading-and-extending)) + +### Has your implementation of the project provided measurable value? + +Yes. Key benefits: +- Reduced platform onboarding effort for new projects +- Faster development-to-production cycles +- Improved reproducibility of machine learning workflows +- Standardization of ML development practices across DHL Data & AI and business units +- Multi-cloud deployment flexibility +- Better governance of machine learning solutions +- Reduced operational overhead through automation + +Kubeflow has become a core capability enabling AI and ML at scale across the DHL Group. + +### Do you have any future plans regarding the project? + +- **Hub**: Planned to be exposed to DHL Data & AI customers from the next release onwards. Currently evaluating MLFlow vs Kubeflow's model registry. +- **Katib and Trainer**: On the near-term roadmap for LLM fine-tuning evaluation. +- **Notebooks V2**: Alpha already under analysis. +- **KFP V2 init containers**: KFP V2 currently lacks init container support - an active open item to resolve upstream or contribute. +- **GenAI workloads**: Expanding platform support including LLM-related workloads. +- **KubeCon presentations**: DHL Data & AI teams are already presenting at KubeCon, strengthening community visibility. +- Continued upstream contributions, community engagement, and GSoC mentorship. + +## Perception + +### What is your perception in terms of the project's community and governance? + +- **Community openness**: Very positive. The project welcomes participation from a large set of organizations, especially after it was donated by Google to the CNCF. The community now feels more heterogeneous and healthier than it did 3 years ago, particularly with the Community Distribution initiative. +- **Governance**: Transparent and aligned with the expectations of a mature CNCF ecosystem project. Vendor neutrality is actively enforced - no single-vendor push has been experienced. +- **Community growth potential**: Strong and growing. Market demand for scalable MLOps is increasing. The adopter base has grown and the outlook is confident. +- **Maintainer diversity and ladder**: Contributions from multiple companies and organizations. Kubeflow is active in Google Summer of Code (GSoC), with some interns becoming ongoing contributors. +- **Maintainer response**: Generally positive. Active community meeting participation. + +### How are you participating in the project community? + +DHL Data & AI's participation is deep and structural: +- Julius Von Kohout: Kubeflow Steering Committee (KSC) member, Dashboard repository owner, Pipelines manifests owner, KCD release maintainer, security response team member. +- DHL Data & AI team members are part of the Kubeflow security response team. +- DHL Data & AI runs internal security scanning and contributes fixes upstream. +- Contributed OIDC-to-OAuth2 proxy support (with Roche Pharmaceutical), Spark Operator work, multi-tenancy and security best practices, KFP bugfixes, and architectural guidance for contributors from Google, IBM, RedHat, AWS, and Capital One. +- Active in community meetings and GSoC mentorship. + +### Did you need to engage with the community members or maintainers? + +Yes, and DHL Data & AI's engagement goes beyond typical adopter participation. + +Key engagement examples: +- Contributed OIDC-to-OAuth2 proxy support (with Roche Pharmaceutical) and Spark Operator improvements; both are now available to the broader community. +- Internal security scanning surfaces CVEs; findings are fed upstream and fixed in the community where applicable. +- Active participation in community meetings. +- One area where collaboration could have been more structured: the interactive Spark Operator feature, which could have benefited from earlier community coordination. Raised as a constructive example of where collaboration processes could improve. +- All community interactions described as professional and productive. + +### If the project were to be archived, what level of difficulty would your organization experience? + +DHL Data & AI is a strong Kubeflow maintainer. If other organizations stepped back, DHL Data & AI would be well-positioned to step up. The governance model ensures vendor neutrality, so no single organization's departure would put the project at risk. There is no meaningful multi-cloud alternative to Kubeflow, particularly for regulated sectors. + +## Project Strengths + +### In your opinion, what are the overall strengths of the project? + +- Open-source and vendor-neutral foundation with actively enforced governance +- Strong Kubernetes-native architecture +- MLOps capabilities spanning the full ML lifecycle +- Flexibility and extensibility for enterprise customization +- Multi-cloud and hybrid-cloud support - no comparable alternative exists for multi-cloud deployments +- Large and growing ecosystem +- Ability to scale from experimentation to enterprise production +- Community Distribution initiative strengthening the release and adoption story +- Strong relevance in regulated sectors where vendor neutrality and auditability matter + +## Project Improvements + +### Is there something you feel that holds the project back from reaching its ultimate potential? + +The primary challenge is operational complexity, though it has improved significantly over the years. Deploying and operating the platform at enterprise scale still requires substantial Kubernetes expertise. Reducing this barrier would lower adoption for many organizations. + +Two specific structural gaps: +- **Helm support**: Lack of official Helm chart support adds friction for organizations that have standardized on Helm. Work in progress via GSoC. +- **Standardized user management pipeline**: Each organization currently builds its own user management and onboarding automation. A reusable pattern would reduce duplicated effort across adopters. + +### In your opinion, what could the project improve? + +- Simplified installation and upgrades, specifically Helm support +- More streamlined lifecycle management and release automation +- Stronger observability and monitoring capabilities out-of-the-box +- Standardized user management pipeline + +## Maturity Level Survey + +Q1. Do you feel you have a good understanding of the meaning of each CNCF maturity level? + +Good understanding of CNCF maturity levels and their significance. + +Q2. Is there information missing regarding the meaning of each different level? + +No significant gaps identified. CNCF Graduation is seen as a clear signal of governance maturity and project quality. + +Q3. Do you rely on those levels internally in any way, and if yes how? + +Yes. Graduation provides more confidence in governance and maturity, and is viewed as a signal that a project is a safe long-term investment. The maturity level carries weight when evaluating and advocating for the project internally and externally. diff --git a/projects/kubeflow/adopter-interviews/kubeflow-adopter-interview-nvidia.md b/projects/kubeflow/adopter-interviews/kubeflow-adopter-interview-nvidia.md new file mode 100644 index 000000000..9a4f298b2 --- /dev/null +++ b/projects/kubeflow/adopter-interviews/kubeflow-adopter-interview-nvidia.md @@ -0,0 +1,159 @@ +# Kubeflow Adopter Interview - NVIDIA + +**Organization**: NVIDIA (Run:ai team) +**Industry**: AI/ML platform / GPU computing +**Interview Date**: June 29, 2026 +**Interviewers (TOC)**: Faseela K +**Interviewees**: Ron Kahn (Senior Software Engineer), Ekin Karabulut (Developer Advocate) + + +## Organization Intro + +### Can you give us an overview of your organization and what it does? + +NVIDIA is a large technology company. The interviewees are part of the NVIDIA Run:ai team that builds an AI platform, providing a one-stop shop for AI workloads - from notebook-based model development through distributed training to production deployment. Within this platform, Kubeflow Trainer is used as the core infrastructure for distributed training. Other Kubeflow components are used by other internal NVIDIA teams; the interviewees speak specifically from the perspective of their platform team's usage of Kubeflow Trainer. + +## Motivation + +### Compared with other products in this space (proprietary and open), what drew you to the project? + +At the time of evaluation (2023), no comparable mature project existed. Other ML frameworks (PyTorch-native schedulers, MPI wrappers) solved isolated pieces - single-node training or job submission - but none provided the full combination of distributed training orchestration, Kubernetes-native design, and a developer-facing API that abstracted Kubernetes from ML users. Kubeflow Trainer was the clear fit - no other project matched its combination of maturity, features, and Kubernetes-native design. The CNCF backing was a further confidence signal - it indicated broad organizational adoption and long-term viability. + +## Usage Scenario + +### How long has your organization used the project? + +NVIDIA Run:ai began evaluating Kubeflow Trainer in early 2023, focused on solving distributed training at scale. The first production version of their platform using Kubeflow Trainer shipped approximately 3-4 months into 2023. The project has been in continuous production since then. + +### What were the main motivations to adopt the project and which key features do you use today? + +Two core requirements drove the decision: + +1. Kubernetes-native: The platform is built on Kubernetes, so the solution had to fit naturally into that ecosystem. +2. Familiar to ML users: The abstraction had to hide Kubernetes complexity and let users focus on training semantics, not infrastructure. + +Currently in use: Kubeflow Trainer, specifically for distributed training with PyTorch and MPI frameworks. Other Kubeflow subprojects are used by other internal NVIDIA teams; the interviewees are not the right contact for those. + +### What is the current level of usage (pre-production, production) and scale? + +Level: Production. +Scale: The NVIDIA Run:ai platform serves customers running thousands of GPUs on Kubeflow Trainer for distributed training workloads. Customers host their own clusters; NVIDIA Run:ai ships Trainer as a component of their platform offering, installed when the customer wants distributed training. + +### What version is used and what is your update cadence with the project? + +Currently using Kubeflow Trainer V1 (latest patch version). The NVIDIA Run:ai platform ships monthly, but Kubeflow Trainer itself is updated approximately every 3 months, aligned with the V1 community patch release cycle. + +Migration to V2: Planned but blocked on elastic workload support. NVIDIA Run:ai's users rely on elastic training; until V2 reaches full feature parity on this capability, the V1 to V2 migration is on hold. + +### Can you walk me through what your experience was in either adopting it outright or integrating it with your existing services and applications? What challenges did you experience with the project? + +Integration was smooth overall. The Trainer API was straightforward and the Kubernetes-native design was praised. Three challenges were encountered: + +1. Large-scale configuration guidance: When users brought very large workloads, there was no clear documentation on configuring Trainer for high-scale scenarios. A specific configuration parameter was not exposed in the API at the time. NVIDIA contributed the fix upstream and it was merged. + +2. Job status visibility for non-Kubernetes users: the NVIDIA Run:ai platform abstracts Kubernetes from end users. At the time, Trainer's status exposure was limited. This has been significantly improved in V2. NVIDIA presented this at KubeCon ~1 year ago; maintainers linked the talk to an upstream tracking issue without being asked. + +3. V1-to-V2 feature parity: NVIDIA Run:ai's users depend on elastic workload support, which V2 does not yet fully cover. This is an ongoing operational blocker. There is also no clear documentation showing what V2 features are still missing vs. V1. + +### Did you find the information in the repo or the project docs valuable to your implementation? If so, where did you find the information and what specifically was useful? + +Generally yes. NVIDIA Run:ai's team primarily uses GitHub examples and the kubeflow.org docs site. The examples are practical and sufficient for most needs. Two gaps noted: + +1. V1-to-V2 feature parity documentation is absent - no clear, maintained view of what is still missing in V2 vs. V1. +2. The GitHub repo documentation is sometimes more up-to-date than the official website, creating inconsistency. + +### Has your implementation of the project provided measurable value? Such as reducing manual activities, faster integrations, supported federation/multi-cloud, ease of use, cost savings, etc. + +Yes, though no quantitative metrics were provided. Value described in qualitative and structural terms: + +1. Engineering velocity: Without Kubeflow Trainer, NVIDIA Run:ai would have had to build and maintain a distributed training controller themselves, requiring significantly more engineering investment. +2. Semantic abstraction: Trainer lets users focus on what they are training and why, not how it runs on Kubernetes. +3. Cost avoidance: Reduced engineering spend on infrastructure, reinvested in higher-level platform features. + +"Without the project, we would have to do it ourselves. It would probably take us much more time and reinventing the wheel every time instead of using something so valuable from the community." + +### Do you have any future plans regarding the project? More involvement, feature requests, expansion, etc. + +1. V1-to-V2 migration: Concrete plan once elastic workload parity is delivered in V2. +2. Deeper collaboration: Early-stage ideas for closer collaboration with the Kubeflow community; too early to detail, but the interviewees noted potential for more collaboration that would benefit both teams. +3. Upstream contributions: A few bug fixes contributed historically; the team maintains close relationships with upstream maintainers. + +## Perception + +### What is your perception in terms of the project's: + +- Community openness +- Governance +- Community growth potential +- Maintainer diversity and ladder +- Maintainer response + +Community openness: Very positive. Maintainers are open and approachable. + +Governance: Community-driven; no single organization appears to dominate direction. + +Community growth potential: Strong - NVIDIA sees potential for deeper collaboration. + +Maintainer diversity and ladder: NVIDIA has Kubeflow maintainers in other internal teams. The interviewed team stays in regular contact with one upstream maintainer. + +Maintainer response: Highly responsive. Maintainers proactively linked a KubeCon talk on Trainer status issues to a tracking issue without being asked. + +### How are you participating in the project community? + +Primarily as an observer and occasional contributor. The team attends community meetings from time to time, mostly as listeners, and engages via CNCF Slack and direct maintainer contact. A few bug fixes were contributed upstream historically, and NVIDIA presented at KubeCon on Kubeflow Trainer (~1 year ago). + +### Did you need to engage with the community members or maintainers? If so, what was the context of the engagement, which communication channels did you use and did it reach an acceptable outcome? + +Yes - via CNCF Slack and direct maintainer contact: + +- Contributed a configuration fix upstream (for large-scale workloads); merged successfully. +- Engaged with maintainers on the job status issue; feedback incorporated into V2 design. +- NVIDIA presented an issue and requirement at KubeCon; maintainers linked the talk to an upstream tracking issue and implemented. + +All engagements reached a positive outcome. "They are always making you feel like they are open for new features, they are open for improvement, there is no ego about it." + +### If the project were to be archived, what level of difficulty would your organization experience? + +Kubeflow Trainer is a core component of NVIDIA Run:ai's distributed training platform. Removal would require rebuilding that capability from scratch. The interviewees noted that NVIDIA already has Kubeflow maintainers in other internal teams, so the company would likely step up maintainership rather than accept the project being abandoned. Removal from the platform would be a significant engineering effort. + +## Project Strengths + +### In your opinion, what are the overall strengths of the project? + +- Clear API design: Semantic training abstractions that hide Kubernetes complexity are well-executed. +- Community culture: Maintainers are open, actively incorporate feedback, and are considered one of the project's biggest strengths. +- Vendor-neutral governance: The project feels community-driven day-to-day; no single organization dominates direction. +- CNCF backing: Provided early confidence for NVIDIA Run:ai to adopt and build on the project. +- Kubernetes ecosystem integration: No friction with the GPU layer. Separation of concerns between Trainer and the GPU Operator works well in practice. + +## Project Improvements + +### Is there something you feel that holds the project back from reaching its ultimate potential? + +The primary gap for NVIDIA Run:ai is V1-to-V2 feature parity. The V2 redesign introduced important improvements, but the lack of elastic workload support prevents production users like NVIDIA Run:ai from migrating. The absence of a clear, maintained V1-vs-V2 feature comparison makes it difficult for operators to plan the transition. + +"It was very challenging, very challenging for us to move to a new, complete way to do it without full feature parity." + +No concerns about community direction - the project feels community-driven. + +### In your opinion, what can the project do better? + +- V1-to-V2 migration documentation: A clear feature parity matrix and migration readiness tracker would significantly help platform operators plan upgrades. +- Large-scale configuration guidance: Dedicated documentation for running Trainer at scale is still sparse. +- Elastic workload support in V2: The specific capability blocking NVIDIA Run:ai's V2 migration. + +### Maturity Level Survey + +The following questions survey your understanding of CNCF project maturity levels. Please fill in your answers directly in this document. + +Q1. Do you feel you have a good understanding of the meaning of each CNCF maturity level?: + +Solid understanding of CNCF maturity stages and their criteria. + +Q2. Is there information missing regarding the meaning of each different level? + +No material documentation gaps identified. + +Q3. Do you rely on those levels internally in any way, and if yes how? + +CNCF maturity level is used as one validation signal alongside other selection criteria; weighting varies by team. diff --git a/projects/kubeflow/kubeflow-graduation-dd.md b/projects/kubeflow/kubeflow-graduation-dd.md new file mode 100644 index 000000000..968787b0d --- /dev/null +++ b/projects/kubeflow/kubeflow-graduation-dd.md @@ -0,0 +1,385 @@ +# Kubeflow Graduation Due Diligence + +- Link to [Graduation Application: Kubeflow](https://github.com/cncf/toc/issues/1861) + +## Graduation Evaluation Summary for Kubeflow + +### Criteria Evaluation + +Faseela K and Brandt Keller conducted the due diligence for Kubeflow, which applied for Graduation from Incubating status (accepted 2023-07-23). + +The following criteria implementations are noteworthy: + +- [Kubeflow](https://www.kubeflow.org/) is a modular, Kubernetes-native AI/ML platform. This graduation covers **6 subprojects**: Kubeflow Trainer, Kubeflow Pipelines, Kubeflow Katib, Kubeflow Notebooks, Kubeflow Spark Operator, and Kubeflow Hub. +- Governance is shared across the KSC, KOC, KDC, and the Working Groups that own each subproject, with the 2026 KSC election completed on schedule and representation rules holding. +- The Kubeflow maintainers and community were highly responsive throughout this review, with PRs merged, issues filed and assigned, and feedback addressed quickly. +- All graduation subprojects hold at least an OpenSSF Best Practices Passing badge; Kubeflow Trainer and Kubeflow Katib have achieved Gold. +- The Kubeflow Ecosystem page and process were established during this review to clarify the boundary between core subprojects and ecosystem partners such as KServe, Feast, and Elyra. +- Kubeflow shows broad adoption across research, financial services, logistics, and platform teams. Four adopter interviews were conducted across the graduation scope. +- Contributor health remains strong, with active contributors from a wide range of organizations and positive YoY growth. + +DD Findings Resolved (Blocking/Required): + +- **[kubeflow/community#992](https://github.com/kubeflow/community/issues/992) - Third-party security audit findings**: Ada Logics audit is published; findings are resolved or tracked. KServe findings are out of scope (independent CNCF project). +- **[kubeflow/community#972](https://github.com/kubeflow/community/issues/972) - WG contributor progression docs**: WG Chair requirements and responsibilities documented in [community#989](https://github.com/kubeflow/community/pull/989). +- **[kubeflow/community#988](https://github.com/kubeflow/community/issues/988) - Roadmap gaps**: notebooks ROADMAP added; kubeflow/kubeflow and website subproject roadmap references updated. +- **[kubeflow/community#990](https://github.com/kubeflow/community/issues/990) - Notebooks release process docs**: `RELEASE.md` added in notebooks and linked to detailed release docs. +- **[kubeflow/community#955](https://github.com/kubeflow/community/issues/955) - KSC vendor-neutrality and election status**: 2026 election results published and org-limit language consolidated. +- **[kubeflow/community#961](https://github.com/kubeflow/community/issues/961)/[community#962](https://github.com/kubeflow/community/issues/962) - KServe ecosystem categorization**: ecosystem boundary clarified via [community#963](https://github.com/kubeflow/community/pull/963), [website#4384](https://github.com/kubeflow/website/pull/4384), [community#965](https://github.com/kubeflow/community/pull/965). + +Non-Blocking Recommendations Completed: + +- **[kubeflow/community#964](https://github.com/kubeflow/community/issues/964) - GTR snapshot submission**: completed via [toc#2180](https://github.com/cncf/toc/pull/2180). +- **[kubeflow/community#996](https://github.com/kubeflow/community/pull/996) - Security self-assessment submission**: completed via [toc#2201](https://github.com/cncf/toc/pull/2201). +- **[kubeflow/community#976](https://github.com/kubeflow/community/issues/976) - Add `CODE_OF_CONDUCT.md` files to subproject repos**: All graduation subproject repos now have `CODE_OF_CONDUCT.md`. +- **[kubeflow/community#991](https://github.com/kubeflow/community/issues/991) - Security self-assessment enhancements**: Broken Trainer release link fixed and stale vulnerability-status wording updated via [Fix self-assessment links and stale wording (community#996)](https://github.com/kubeflow/community/pull/996). +- **Community/ADOPTERS.md listed KServe in the "Adopters of Kubeflow Projects" table**: KServe removed from the subprojects adopters table; only the 6 graduation subprojects are listed. +- **[kubeflow/community#967](https://github.com/kubeflow/community/issues/967)/[community#969](https://github.com/kubeflow/community/issues/969) - WG docs ownership and stale links**: outdated WGs removed and WG docs/link updates merged across community and website. + +Adopter Interview Feedback (Non-Blocking, Ongoing): + +- **Helm chart for installation**: deployment ergonomics remain a common request. Trainer and Spark Operator already provide Helm charts, and community-distribution Helm work is actively in progress as part of the KDC roadmap. +- **Service manager / operator documentation**: Upstream docs are adequate for end users but insufficient for operators managing multi-tenant production deployments. +- **Security: inference endpoint multi-tenancy**: Cross-namespace authorization issue in KServe/Knative - upstream: [Cross-namespace inference endpoint auth issue (knative/serving#12533)](https://github.com/knative/serving/issues/12533). +- **Pipeline artifact multi-tenancy**: [Pipeline artifact multi-tenancy (pipelines#11760)](https://github.com/kubeflow/pipelines/issues/11760). +- **MLMD component maintenance**: MLMD is no longer actively maintained and depends on deprecated `mysql_native_password` auth, blocking MySQL backend upgrades. +- **Security CVE response across interconnected components**: CVE in a dependency (e.g., KServe) can block the entire platform; a security best practices document for operators was requested. +- **More frequent releases / faster security patching**: Current release cadence can be too slow for regulated environments with aggressive patching requirements. +- **Maintainer growth, particularly for Pipelines**: Long-term sustainability risk noted; growing the maintainer base across subprojects is an active priority. +- **UX for non-engineer users**: A simplified UX for data scientists and business analysts with no Kubernetes background would broaden adoption. + +### Adoption Evaluation + +Adopter feedback for Kubeflow satisfies the TOC's adoption requirements for Graduation. Public ADOPTERS.md evidence across the graduation-scope subprojects, together with four completed adopter interviews, shows sustained production use in enterprise and research environments. + +The interviews consistently point to lifecycle coverage, Kubernetes-native architecture, multi-cloud flexibility, and active maintainer engagement. + +Adopter-requested enhancements are already captured in the previous section and are tracked as non-blocking follow-up items. + +### Final Assessment + +The TOC has found the project to have satisfied the criteria for Graduation. + +Based on the evidence in this DD, Kubeflow is ready for Graduation. + +The non-blocking items captured in this DD reflect adopter feedback and enhancement opportunities; tracking them via the project's public roadmap would benefit adopter engagement and usability. + +--- + +## Criteria + +## Application Process Principles + +### Suggested + +N/A + +### Required + +- [X] **Give a presentation and engage with the domain specific TAG(s) to increase awareness.** + +Kubeflow has presented to TAG Runtime twice: +- TAG Runtime - January 2021: [https://youtu.be/S6N8ARZZcGs](https://youtu.be/S6N8ARZZcGs) +- TAG Runtime and TOC AI Initiatives - November 2024: [https://youtu.be/u4Mf3Jh8v2E?t=2243](https://youtu.be/u4Mf3Jh8v2E?t=2243) + +The November 2024 presentation covered project overview, 2024 highlights, a live fine-tuning demo, and named graduation as a 2025/2026 goal. No concerns or action items were raised by TAG Runtime. + +- [X] **TAG provides insight/recommendation of the project in the context of the landscape.** + +TAG Runtime provided a positive recommendation for Kubeflow's graduation readiness based on the November 2024 presentation ([https://youtu.be/u4Mf3Jh8v2E?t=2243](https://youtu.be/u4Mf3Jh8v2E?t=2243)) and review of the application materials. + +- [X] **All project metadata and resources are [vendor-neutral](https://contribute.cncf.io/maintainers/community/vendor-neutrality/).** + +The KSC charter enforces a maximum of 1 seat per organization across all 5 KSC seats. Current KSC composition (verified 2026-06-11): Andrey Velichkevich (Apple), Chase Christensen (Wiz), Francisco Arceo (Red Hat), Julius von Kohout (DHL), Mathew Wicks (NVIDIA) - 5 seats, 5 unique organizations. The KOC (Outreach Committee) similarly enforces a maximum of 1 Chair per org. The 2026 election completed on schedule ([2026 KSC election issue (community#950)](https://github.com/kubeflow/community/issues/950)) with the previous Red Hat dual-seat and stale KEP language concerns resolved via [Add 2026 KSC election results (community#956)](https://github.com/kubeflow/community/pull/956) and [Consolidate KSC org-limit rule (community#957)](https://github.com/kubeflow/community/pull/957). + +- [X] **Review and acknowledgement of expectations for sandbox projects and requirements for moving forward through the CNCF Maturity levels.** + - [X] Acknowledged in the graduation application body (September 2025) per CNCF application requirements. + +Completion of this due diligence document, resolution of concerns raised, and presentation for public comment satisfies the Due Diligence Review criterion. + +- [X] **Additional documentation as appropriate for project type, e.g.: installation documentation, end user documentation, reference implementation and/or code samples.** + +All 6 graduation subprojects have dedicated documentation pages on kubeflow.org with installation guides, user guides, operator guides, and reference documentation. The [kubeflow.org docs site](https://www.kubeflow.org/docs/) is actively maintained. The [General Technical Review](https://github.com/kubeflow/community/blob/master/KUBEFLOW-GENERAL-TECHNICAL-REVIEW.md) (GTR) provides an extensive technical reference. Kubeflow Notebooks has an architectural dependency on Kubeflow Central Dashboard and cannot yet be deployed standalone - this is however a part of their roadmap. + +## Governance and Maintainers + +### Suggested + +- [X] **Governance has continuously been iterated upon by the project as a result of their experience applying it, with the governance history demonstrating evolution of maturity alongside the project's maturity evolution.** + +The Kubeflow governance model has evolved alongside the project: from a single Google-donated project to a multi-org structure with the KSC, KOC, KDC, formal WG lifecycle docs, KEPs, and a subproject application process. The history of governance PRs in kubeflow/community demonstrates ongoing iteration. + +### Required + +- [X] **Clear and discoverable project governance documentation.** + +The kubeflow.org [governance page](https://www.kubeflow.org/docs/about/governance/) is the public governance root. In the community repo, governance is documented in the KSC, KOC, and KDC charters, WG governance, and the contributor ladder ([KSC charter](https://github.com/kubeflow/community/blob/master/committee-steering/charter.md), [KOC charter](https://github.com/kubeflow/community/blob/master/committee-outreach/charter.md), [KDC charter](https://github.com/kubeflow/community/blob/master/committee-distribution/charter.md), [wg-governance](https://github.com/kubeflow/community/blob/master/committee-steering/wg-governance.md), [community-membership](https://github.com/kubeflow/community/blob/master/community-membership.md)). + +- [X] **Governance is up to date with actual project activities, including any meetings, elections, leadership, or approval processes.** + +KSC elections and membership are current (2026 election completed). WG governance docs were refreshed to remove defunct WGs and update current meetings, organizers, and links via [community#974](https://github.com/kubeflow/community/pull/974), [community#989](https://github.com/kubeflow/community/pull/989), [community#997](https://github.com/kubeflow/community/pull/997), and [website#4398](https://github.com/kubeflow/website/pull/4398). + +- [X] **Governance clearly documents [vendor-neutral](https://contribute.cncf.io/maintainers/community/vendor-neutrality/) of project direction.** + +The KSC charter documents an explicit 1-seat-per-organization cap for its seats, and the KOC charter similarly caps Chair representation. This keeps governance structurally vendor-neutral even with some technical concentration in contributor activity; the 2026 KSC election confirmed the rule in practice. + +- [X] **Document how the project makes decisions on leadership, contribution acceptance, requests to the CNCF, and changes to governance or project goals.** + +Decision-making is documented across KSC decisions, the KEP process, and the ecosystem join process for external project partnerships ([committee-steering/charter.md](https://github.com/kubeflow/community/blob/master/committee-steering/charter.md), [proposals](https://github.com/kubeflow/community/tree/master/proposals), [ecosystem](https://github.com/kubeflow/community/tree/master/ecosystem)). + +- [X] **Document how role, function-based members, or sub-teams are assigned, onboarded, and removed for specific teams (example: Security Response Committee).** + +The [Kubeflow community page](https://www.kubeflow.org/docs/about/community/) and [Contributing](https://www.kubeflow.org/docs/about/contributing/) page provide the public entry point, while the contributor ladder and WG governance document how Members, Reviewers, Approvers, and WG Chairs are onboarded, expected to contribute, and removed when inactive. The WG chair requirements were clarified in [community#989](https://github.com/kubeflow/community/pull/989). + +- [X] **Document complete list of current maintainers, including names, contact information, domain of responsibility, and affiliation.** + +[MAINTAINERS.md](https://github.com/kubeflow/community/blob/master/MAINTAINERS.md) links to the KSC member list, KOC member list, WG chairs/leads in `wgs.yaml`, and per-subproject OWNERS files. The KSC and KOC lists include names, GitHub handles, organizations, and terms; `wgs.yaml` was refreshed alongside [community#974](https://github.com/kubeflow/community/pull/974). + +- [X] **A number of active maintainers which is appropriate to the size and scope of the project.** + +[MAINTAINERS.md](https://github.com/kubeflow/community/blob/master/MAINTAINERS.md) covers governance-level maintainers (KSC and KOC) and links to per-subproject OWNERS files for approver-level maintainers. All 6 graduation subprojects have active approvers listed in their respective OWNERS files. The maintainer count is appropriate for a project of Kubeflow's size and scope. + +- [X] **Document a complete maintainer lifecycle process (including roles, onboarding, offboarding, and emeritus status).** + +The [contributor ladder](https://github.com/kubeflow/community/blob/master/community-membership.md) defines the full lifecycle: onboarding criteria for each role (Member -> Reviewer -> Approver), offboarding (12-month inactivity -> org removal; `emeritus_approvers` for inactive approvers), and WG Lead/Chair removal via super-majority vote. + +- [X] **Demonstrate usage of the maintainer lifecycle with outcomes, either through the addition or replacement of maintainers as project events have required.** + +Addition examples (all verified merged 2025-2026): +- [Add @mprahl and @zazulam as KFP maintainers (pipelines#12059)](https://github.com/kubeflow/pipelines/pull/12059) - Added @mprahl + @zazulam as KFP maintainers with community announcement +- [Add @Electronic-Waste as approver, @astefanutti as reviewer (trainer#2659)](https://github.com/kubeflow/trainer/pull/2659) - Added with explicit criteria verification against the membership page +- [Add @Al-Pragliola as approver (hub#1153)](https://github.com/kubeflow/hub/pull/1153) - Added with full checklist verification and linked internal-acls PR + +Removal example: +- [@zw0610 moves to emeritus_approvers (trainer#2343)](https://github.com/kubeflow/trainer/pull/2343) - @zw0610 self-removed to `emeritus_approvers` + +- [X] **Project maintainers from at least 2 organizations that demonstrates survivability.** + +Maintainer org diversity is broad across all graduation subprojects, with approvers spanning a wide range of organizations ([LFX Insights](https://insights.lfx.dev/foundation/cncf/overview/github?project=kubeflow)). Org affiliation for governance-level maintainers is documented in the KSC and KOC member lists linked from [MAINTAINERS.md](https://github.com/kubeflow/community/blob/master/MAINTAINERS.md), and for WG leads/chairs in [wgs.yaml](https://github.com/kubeflow/community/blob/master/wgs.yaml). No subproject is controlled by a single organization. WG Notebooks leadership was updated in [Update WG Notebooks docs (community#983)](https://github.com/kubeflow/community/pull/983), to replace an inactive lead and ensure organizational diversity. + +- [X] **Code and Doc ownership in Github and elsewhere matches documented governance roles.** + +Kubeflow uses a GitOps ACL system ([kubeflow/internal-acls](https://github.com/kubeflow/internal-acls)) via Peribolos and Prow: org membership, team membership, and repo permissions are declared in `github-orgs/kubeflow/org.yaml` and reconciled against GitHub. The KSC team, WG lead teams, and security-team are declared in org.yaml and match governance documents. Per-repo code ownership is enforced via OWNERS files (Prow `lgtm` + `approve`). `default_repository_permission: read` enforces least privilege. + +- [X] **Document agreement that project will adopt CNCF Code of Conduct.** + +The CNCF Code of Conduct is adopted and documented at [kubeflow.org/docs/about/community/#code-of-conduct](https://www.kubeflow.org/docs/about/community/#code-of-conduct), and cross-linked from committee charters and WG governance docs. + +- [X] **CNCF Code of Conduct is cross-linked from other governance documents.** + +The CNCF CoC is referenced in WG governance docs and individual WG/committee charters. As of 2026-06-18, standalone `CODE_OF_CONDUCT.md` files are present in all 7 graduation subproject repos: community, spark-operator, trainer, katib, notebooks, hub, and pipelines. Resolved via [kubeflow/community#976](https://github.com/kubeflow/community/issues/976). + +- [X] **All subprojects, if any, are listed.** + +The 6 graduation subprojects are listed consistently across the graduation application, GTR, kubeflow.org architecture page, `wgs.yaml`, and `internal-acls/org.yaml`: +1. Kubeflow Trainer - [kubeflow/trainer](https://github.com/kubeflow/trainer) +2. Kubeflow Pipelines - [kubeflow/pipelines](https://github.com/kubeflow/pipelines) +3. Kubeflow Katib - [kubeflow/katib](https://github.com/kubeflow/katib) +4. Kubeflow Notebooks - [kubeflow/notebooks](https://github.com/kubeflow/notebooks) +5. Kubeflow Spark Operator - [kubeflow/spark-operator](https://github.com/kubeflow/spark-operator) +6. Kubeflow Hub - [kubeflow/hub](https://github.com/kubeflow/hub) + +The subproject add/remove process is documented in [community/subprojects/README.md](https://github.com/kubeflow/community/tree/master/subprojects) and is directly linked from the [kubeflow.org introduction page](https://www.kubeflow.org/docs/started/introduction/#kubeflow-subprojects). + +- [X] **If the project has subprojects: subproject leadership, contribution, maturity status documented, including add/remove process.** + +**Add process:** Documented in [community/subprojects/README.md](https://github.com/kubeflow/community/tree/master/subprojects): GitHub issue -> proposal doc -> community call demo -> PR -> KSC vote -> repo transfer. Prior applications (spark-operator, model-registry) are archived there. + +**Remove/retire process:** Documented in [WG lifecycle](https://github.com/kubeflow/community/blob/master/committee-steering/wg-governance.md#wg-lifecycle): WG retirement triggers (3 months without quorum -> should retire; 6 months -> must retire), archiving steps, and KSC final vote. + +**Contribution docs:** All 6 subprojects have `CONTRIBUTING.md` files (verified 2026-06-16). + +**Maturity/roadmap:** All 6 subprojects have `ROADMAP.md`. Subproject maturity levels are being introduced via [Add subproject maturity levels (community#965)](https://github.com/kubeflow/community/pull/965). + +## Contributors and Community + +### Suggested + +- [X] **Contributor ladder with multiple roles for contributors.** + +The [contributor ladder](https://github.com/kubeflow/community/blob/master/community-membership.md) defines 4 levels: Member -> Reviewer -> Approver -> WG Lead/Chair, each with documented requirements, responsibilities, and privileges. Onboarding and offboarding at each step are described. + +### Required + +- [X] **Clearly defined and discoverable process to submit issues or changes.** + +Issue submission guidance is at [kubeflow.org/docs/about/contributing/#starter-issues](https://www.kubeflow.org/docs/about/contributing/#starter-issues). Change proposals follow the [KEP process](https://github.com/kubeflow/community/tree/master/proposals) with a documented lifecycle (provisional -> implementable -> implemented) and PR template. + +- [X] **Project must have, and document, at least one public communications channel for users and/or contributors.** + +Kubeflow uses CNCF Slack with dedicated subproject channels. Channels and entry points are documented at [kubeflow.org/docs/about/community/#slack-channels](https://www.kubeflow.org/docs/about/community/#slack-channels). The [kubeflow-discuss mailing list](https://groups.google.com/g/kubeflow-discuss) is also documented. + +- [X] **List and document all project communication channels, including subprojects (mail list/slack/etc.). List any non-public communications channels and what their special purpose is.** + +The [community page](https://www.kubeflow.org/docs/about/community/) lists all public channels: CNCF Slack channels, kubeflow-discuss mailing list, YouTube channels, social media, and community meetings calendar. The non-public channel (KSC private mailing list: `ksc@kubeflow.org`) is documented with its purpose (private KSC deliberation and security escalation). + +- [X] **Up-to-date public meeting schedulers and/or integration with CNCF calendar.** + +Community-level meeting calendar is published and integrated with CNCF/LFX calendar at [kubeflow.org/docs/about/community/#list-of-available-meetings](https://www.kubeflow.org/docs/about/community/#list-of-available-meetings). WG-level meeting metadata was refreshed via [WG docs need to reflect current subproject ownership (community#967)](https://github.com/kubeflow/community/issues/967) and [WG README files have stale and broken links (community#969)](https://github.com/kubeflow/community/issues/969) - broken Zoom/calendar links, stale Slack workspace references, and stale organizer handles updated across all active WGs. + +- [X] **Documentation of how to contribute, with increasing detail as the project matures.** + +Central contributing guide at [kubeflow.org/docs/about/contributing/](https://www.kubeflow.org/docs/about/contributing/). All 6 graduation subprojects have `CONTRIBUTING.md` files in their repos with subproject-specific contribution guidance (verified 2026-06-17). + +- [X] **Demonstrate contributor activity and recruitment.** + +LF Insights shows strong contributor activity across a wide range of organizations, with positive YoY growth and healthy contributor retention. Recent maintainer additions across multiple subprojects demonstrate active recruitment (trainer#2659, pipelines#12059, hub#1153). + +## Engineering Principles + +### Suggested + +N/A + +### Required + +- [X] **Document project goals and objectives that illustrate the project's differentiation in the Cloud Native landscape as well as outlines how this project fulfills an outstanding need and/or solves a problem differently.** + +Addressed by the [GTR](https://github.com/kubeflow/community/blob/master/KUBEFLOW-GENERAL-TECHNICAL-REVIEW.md) (lines 13-152): Kubeflow is positioned as a "foundation of tools for AI Platforms on Kubernetes," differentiated by its composability, modularity, Kubernetes-native design, and support for the full ML lifecycle. Target personas (data scientists, ML engineers, platform engineers, vendors) and primary use cases (distributed training, fine-tuning, HPO, pipelines, serving) are explicitly documented, along with unsupported use cases (no GitOps implementation, no infrastructure beyond Kubernetes). + +- [X] **Document what the project does, and why it does it - including viable cloud native use cases.** + +Addressed by the [GTR](https://github.com/kubeflow/community/blob/master/KUBEFLOW-GENERAL-TECHNICAL-REVIEW.md) (lines 82-152): primary use cases include large-scale data processing, distributed pre-training of foundation models, LLM fine-tuning, hyperparameter optimization, end-to-end GenAI pipelines, interactive AI development, and multi-tenancy. The [kubeflow.org architecture page](https://www.kubeflow.org/docs/started/architecture/) with the AI lifecycle diagram provides the user-facing project overview. + +- [X] **Document and maintain a public roadmap or other forward looking planning document or tracking mechanism.** + +All 6 graduation subprojects now have `ROADMAP.md` files. `kubeflow/notebooks` ROADMAP.md was added via [Add ROADMAP.md to notebooks (notebooks#1188)](https://github.com/kubeflow/notebooks/pull/1188). Resolved via [kubeflow/community#988](https://github.com/kubeflow/community/issues/988). + +- [X] **Roadmap change process is documented.** + +Community-wide changes use the [KEP process](https://github.com/kubeflow/community/tree/master/proposals) (proposal -> community review -> implementation). Subproject-specific proposals are managed in subproject KEP directories (e.g., [trainer/docs/proposals](https://github.com/kubeflow/trainer/tree/master/docs/proposals)). The full KEP lifecycle (provisional -> implementable -> implemented) and template are documented. + +- [X] **Document overview of project architecture and software design that demonstrates viable cloud native use cases, as part of the project's documentation.** + +Architecture documentation is detailed: the [GTR](https://github.com/kubeflow/community/blob/master/KUBEFLOW-GENERAL-TECHNICAL-REVIEW.md) covers design principles, architecture requirements, service dependencies, IAM design, API topology, and release process. The [kubeflow.org architecture page](https://www.kubeflow.org/docs/started/architecture/) includes an AI lifecycle diagram and component descriptions. The [Kubeflow Community Distribution release pages](https://www.kubeflow.org/docs/kubeflow-distribution/releases/) provide per-release component version matrices. + +- [X] **Document the project's release process and guidelines publicly in a RELEASES.md or equivalent file that defines:** + - [X] Release expectations (scheduled or based on feature implementation) + - [X] Tagging as stable, unstable, and security related releases + - [X] Information on branch and tag strategies + - [X] Branch and platform support and length of support + - [X] Artifacts included in the release. + +All 6 graduation subprojects have `RELEASE.md` or equivalent. `kubeflow/notebooks` `RELEASE.md` was added via [Add RELEASE.md to notebooks (notebooks#1189)](https://github.com/kubeflow/notebooks/pull/1189); it points to detailed step-by-step release docs (minor/patch/RC/GA, signed tagging, artifact updates) in `releasing/README.md` on both `notebooks-v1` and `notebooks-v2` branches. Resolved via [kubeflow/community#990](https://github.com/kubeflow/community/issues/990). + +- [X] **History of regular, quality releases.** + +All 6 graduation subprojects have consistent release histories with regular minor and patch releases, verified across their GitHub releases pages: +- [Trainer releases](https://github.com/kubeflow/trainer/releases) +- [Pipelines releases](https://github.com/kubeflow/pipelines/releases) +- [Spark Operator releases](https://github.com/kubeflow/spark-operator/releases) +- [Katib releases](https://github.com/kubeflow/katib/releases) +- [Hub releases](https://github.com/kubeflow/hub/releases) +- [Notebooks releases](https://github.com/kubeflow/notebooks/releases) + +RC/pre-release tags are used before stable releases, indicating quality-focused practices. The [Kubeflow Community Distribution release history](https://www.kubeflow.org/docs/kubeflow-distribution/releases/) spans from v0.6 through the current release with detailed per-release notes. + +## Security + +### Suggested + +- [ ] **Achieving OpenSSF Best Practices silver or gold badge.** + +[Trainer](https://www.bestpractices.dev/projects/10435) and [Katib](https://www.bestpractices.dev/projects/9941) have achieved Gold. + +### Required + +- [X] **Clearly defined and discoverable process to report security issues.** + +All 6 graduation subprojects have `SECURITY.md` files documenting: supported versions, private vulnerability reporting via GitHub Security Advisories, escalation to `ksc@kubeflow.org`, a disclosure process (acknowledgment within 10 business days, assessment, resolution, public disclosure), and explicit guidance not to use public channels. Verified across kubeflow/trainer, pipelines, katib, notebooks, spark-operator, and hub (2026-06-17). + +- [X] **Enforcing Access Control Rules to secure the code base against attacks (Example: two factor authentication enforcement, and/or use of ACL tools.)** + +Kubeflow uses a GitOps ACL system via [kubeflow/internal-acls](https://github.com/kubeflow/internal-acls): org membership and repo permissions are declared in `github-orgs/kubeflow/org.yaml` and reconciled by Peribolos. 2FA is required for all GitHub org members. `default_repository_permission: read` enforces least privilege. + +- [X] **Document assignment of security response roles and how reports are handled.** + +Security response roles are documented in each subproject's security policy: reports handled by project owners, with KSC (`ksc@kubeflow.org`) as escalation. A dedicated `security-team` is declared in `internal-acls/org.yaml`. Disclosure process (acknowledgment, investigation, resolution, public disclosure) is documented consistently across all 6 subproject policies. + +- [X] **Document Security Self-Assessment.** + +The [Kubeflow Security Self-Assessment](https://github.com/kubeflow/community/blob/master/security/self-assessment.md) is published in kubeflow/community, actively maintained (updated 2026), and covers all 6 graduation subprojects. Content structure follows TAG Security guidance including metadata, security links, background, actors, goals, security functions, secure development practices, and security issue resolution. The self-assessment is also merged into cncf/toc via [Kubeflow Security Self-Assessment (toc#2201)](https://github.com/cncf/toc/pull/2201) and is currently located in this DD folder at [security-assessment/self-assessment.md](security-assessment/self-assessment.md). + +- [X] **Third Party Security Review.** + - [X] Moderate and low findings from the Third Party Security Review are planned/tracked for resolution. + +The Ada Logics security audit (Sept 2025) is publicly available: [security/Ada_Logics-Kubeflow-security-audit-2025.pdf](https://github.com/kubeflow/community/blob/master/security/Ada_Logics-Kubeflow-security-audit-2025.pdf), published via [Publish Ada Logics security audit (community#994)](https://github.com/kubeflow/community/pull/994). All in-scope findings are resolved or tracked with linked PRs/issues; KServe findings are out of scope as KServe is an independent CNCF project. Full tracking details are in [kubeflow/community#992](https://github.com/kubeflow/community/issues/992). + +- [X] **Achieve the Open Source Security Foundation (OpenSSF) Best Practices passing badge.** + +All 6 graduation subprojects hold at least Passing badge (verified via bestpractices.dev, 2026-06-17): +- [Katib](https://www.bestpractices.dev/projects/9941): **Gold** +- [Trainer](https://www.bestpractices.dev/projects/10435): **Gold** +- [Notebooks](https://www.bestpractices.dev/en/projects/9942): Passing +- [Pipelines](https://www.bestpractices.dev/en/projects/9938): Passing +- [Hub](https://www.bestpractices.dev/en/projects/9937): Passing +- [Spark Operator](https://www.bestpractices.dev/en/projects/10524): Passing + +## Ecosystem + +### Suggested + +N/A + +### Required + +- [X] **Publicly documented list of adopters, which may indicate their adoption level (dev/trialing, prod, etc.)** + +All 6 graduation subprojects maintain public `ADOPTERS.md` files listing production adopters across research, financial services, logistics, and AI/ML platform industries: +- [community/ADOPTERS.md](https://github.com/kubeflow/community/blob/master/ADOPTERS.md) +- [Spark Operator](https://github.com/kubeflow/spark-operator/blob/master/ADOPTERS.md) +- [Trainer](https://github.com/kubeflow/trainer/blob/master/ADOPTERS.md) +- [Katib](https://github.com/kubeflow/katib/blob/master/ADOPTERS.md) +- [Hub](https://github.com/kubeflow/hub/blob/main/ADOPTERS.md) +- [Pipelines](https://github.com/kubeflow/pipelines/blob/master/ADOPTERS.md) +- [Notebooks](https://github.com/kubeflow/notebooks/blob/main/ADOPTERS.md) + +The adopter interviews provided additional adoption confidence beyond what the public lists reflect. For example, CERN added themselves to the Notebooks ADOPTERS.md following the interview: [Add CERN to Notebooks ADOPTERS.md (notebooks#1190)](https://github.com/kubeflow/notebooks/pull/1190). + +- [X] **Used in appropriate capacity by at least 3 independent + indirect/direct adopters, (these are not required to be in the publicly documented list of adopters)** + +> **Status: Satisfied.** 4 adopter interviews completed across research (CERN), logistics (DHL Data & AI), AI/ML platform (NVIDIA), and financial services (anonymized), collectively covering all 6 graduation subprojects across diverse industries and deployment scales. + +- [X] **TOC verification of adopters.** + +> **Status: Satisfied.** TOC sponsors have conducted and verified 4 adopter interviews. All 4 adopters confirmed sustained production use across the graduation subprojects. + +#### Adoption + +> All 4 adopter interviews are complete and included below. CERN, DHL Data & AI, and NVIDIA are published with attribution; one additional interview is currently published in anonymized form pending explicit approval for named publication. + +##### Adopter 1 - CERN/Scientific research organization + +This adopter interview was conducted in June 2026 and recorded a strong production adopter running Kubeflow at scale across Notebooks, KServe, Trainer, Katib, and Pipelines, with plans to expand to thousands of users as part of a 3-5 year AI strategy. Refer to the [interview summary](adopter-interviews/kubeflow-adopter-interview-cern.md) for more details. + +##### Adopter 2 - Financial services organization (anonymized) + +This adopter interview was conducted in June 2026 with a large regulated financial services organization using Kubeflow extensively in production across multiple subprojects. Refer to the [interview summary](adopter-interviews/kubeflow-adopter-interview-adopter-1.md) for details. + +##### Adopter 3 - DHL Data & AI / Logistics organization + +This adopter interview was conducted in June 2026 with DHL Data & AI, operating Kubeflow in production at significant scale across Notebooks, Pipelines, Dashboard, and Spark Operator, with Katib, Trainer, and Hub on the near-term roadmap. Refer to the [interview summary](adopter-interviews/kubeflow-adopter-interview-dhl.md) for details. + +##### Adopter 4 - NVIDIA / AI/ML platform organization + +This adopter interview was conducted in June 2026 with NVIDIA Run:ai, using Kubeflow Trainer in production for large-scale distributed training. Refer to the [interview summary](adopter-interviews/kubeflow-adopter-interview-nvidia.md) for details. + +- [X] **Clearly documented integrations and/or compatibility with other CNCF projects as well as non-CNCF projects.** + +Kubeflow's integration documentation is organized per subproject (each has dedicated integration guides) rather than a single consolidated page. Maintainer confirmation (Andrey Velichkevich, 2026-06-17): "every Kubeflow subproject has its own list of integrations with other CNCF projects." An [Ecosystem page](https://www.kubeflow.org/docs/ecosystem/) provides a top-level view of ecosystem partners. + +Per-subproject integration evidence (verified 2026-06-17): + +| Subproject | Integration documented | Link | +|---|---|---| +| **Trainer** | Volcano, Kueue (job scheduling) | [Job Scheduling guide](https://www.kubeflow.org/docs/components/trainer/operator-guides/job-scheduling/) | +| **Trainer** | PyTorch, DeepSpeed, JAX, HuggingFace, XGBoost, MLX, Arrow, DataFusion | [Overview](https://www.kubeflow.org/docs/components/trainer/overview/) | +| **Spark Operator** | YuniKorn | [YuniKorn integration](https://www.kubeflow.org/docs/components/spark-operator/user-guide/yunikorn-integration/) | +| **Notebooks** | Jupyter, TensorFlow | [Jupyter + TF examples](https://www.kubeflow.org/docs/components/notebooks/jupyter-tensorflow-examples/) | +| **Hub** | KServe (CNCF Incubating) | [Hub getting-started](https://www.kubeflow.org/docs/components/hub/getting-started/#using-model-registry-metadata) | +| **Pipelines** | SeaweedFS / S3-compatible stores | [Object store config](https://www.kubeflow.org/docs/components/pipelines/operator-guides/configure-object-store/) | +| **Katib** | Argo Workflows (CNCF Graduated) | [Trial template guide](https://www.kubeflow.org/docs/components/katib/user-guides/trial-template/) | + +CNCF projects integrated across subprojects (non-exhaustive): Kubernetes, Argo Workflows, Istio, Cert-Manager, Knative, Kueue, JobSet, Volcano, OpenTelemetry, Envoy, Prometheus, KEDA. + +Non-CNCF projects integrated across subprojects (non-exhaustive): PyTorch, DeepSpeed, JAX, Jupyter, Apache Spark, Apache Arrow, Apache DataFusion, Horovod, MLX, XGBoost, HuggingFace, SeaweedFS, YuniKorn, TensorFlow, vLLM.