Skip to content

Add kernel pinning and post-driver reboot to accel_drivers role#1204

Draft
skovili wants to merge 3 commits into
openstack-k8s-operators:mainfrom
skovili:accel-drivers-kernel-pin-reboot
Draft

Add kernel pinning and post-driver reboot to accel_drivers role#1204
skovili wants to merge 3 commits into
openstack-k8s-operators:mainfrom
skovili:accel-drivers-kernel-pin-reboot

Conversation

@skovili

@skovili skovili commented Jun 25, 2026

Copy link
Copy Markdown

Summary

  • Add kernel version pinning support to ensure driver/kernel compatibility for NVIDIA vGPU deployments
  • Add post-driver-install reboot with nvidia-vgpu-mgr verification for MIG instance detection
  • Both features are disabled by default and opt-in via configuration

Problem

MIG vGPU deployments on NVIDIA A30 GPUs require:

  1. A specific kernel version matching the NVIDIA AIE driver — without pinning, kernel updates break DKMS driver builds
  2. A reboot after driver installation so nvidia-vgpu-mgr can detect MIG instances and expose vGPU types via SR-IOV VFs

These were previously handled by custom OpenStackDataPlaneService CRs outside the role, which is not sustainable.

New Configuration

edpm_accel_drivers:
  kernel:
    pin: false           # enable kernel pinning
    version: ""          # e.g. "5.14.0-570.26.1.el9_6"
    remove_old: true     # remove non-target kernels
  reboot:
    post_driver: false   # reboot after driver install
    timeout: 1800
    wait_for_connection_timeout: 900
    verify_vgpu_mgr: true

Updated Role Flow

  1. setup_config.yml
  2. kernel_pin.yml ← NEW (when kernel.pin is true)
  3. driver_install.yml
  4. Cleanup temp dir
  5. boot_config.yml
  6. reboot.yml ← NEW (when reboot.post_driver is true)
  7. nvidia_mig_partitioning.yml

🤖 Generated with Claude Code

MIG vGPU deployments require a specific kernel version matching the
NVIDIA driver and a reboot after driver installation for nvidia-vgpu-mgr
to detect MIG instances. These were previously handled by custom
OpenStackDataPlaneService CRs outside the role.

New features (disabled by default):
- kernel.pin: install a pinned kernel, remove old kernels, exclude
  kernel packages from future dnf updates
- reboot.post_driver: reboot after driver install and boot_config,
  wait for connection, verify nvidia-vgpu-mgr started

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-ci openshift-ci Bot requested review from abays and stuggi June 25, 2026 19:53
@openshift-ci

openshift-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: skovili
Once this PR has been reviewed and has the lgtm label, please assign slagle for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci

openshift-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Hi @skovili. Thanks for your PR.

I'm waiting for a openstack-k8s-operators member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

- Add kernel version format validation with assert before any
  destructive operations
- Detect architecture dynamically via uname -m instead of hardcoding
  x86_64
- Fix dnf.conf exclude syntax (exclude= not excludepkgs=)
- Add idempotency guard to reboot: skip when running kernel already
  matches target
- Document kernel and reboot config in argument_specs.yml
- Add kernel_pin_reboot molecule test scenario

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/25172752d3ee4d3cac97e7bf098e9634

openstack-k8s-operators-content-provider FAILURE in 7m 03s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
✔️ cifmw-pod-zuul-files SUCCESS in 5m 07s
⚠️ cifmw-crc-podified-edpm-baremetal-bootc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
✔️ noop SUCCESS in 0s
⚠️ edpm-ansible-tempest-multinode SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ edpm-ansible-baremetal-minor-update SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
edpm-ansible-molecule-edpm_accel_drivers FAILURE in 21m 58s

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/1910f93ddcb74d1a8bafd3f97f2ed154

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 51m 56s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 34m 20s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 44m 05s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 46s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 46m 39s
✔️ noop SUCCESS in 0s
✔️ edpm-ansible-tempest-multinode SUCCESS in 1h 46m 47s
✔️ edpm-ansible-baremetal-minor-update SUCCESS in 2h 38m 06s
edpm-ansible-molecule-edpm_accel_drivers FAILURE in 19m 44s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant