Skip to content

Make eessi configure gpu node automatically #841

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Make eessi configure gpu node automatically #841

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
2c45c6c
Adding tasks to essi/configure.yml to make eessi configure
claudia-lola Oct 28, 2025
fa78672
Adding tasks to essi/configure.yml to make eessi configure
claudia-lola Oct 28, 2025
d5ac3f7
applying edits to task names
claudia-lola Nov 4, 2025
2c22de1
Merge branch 'configure-gpus' of github.com:stackhpc/ansible-slurm-ap…
claudia-lola Nov 4, 2025
5f1ddde
replacing EESSI block with running the configure.yml task directly
claudia-lola Nov 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions ansible/roles/compute_init/files/compute-init.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,22 @@
ansible.builtin.command:
cmd: "cvmfs_config setup"

# configure gpus
- name: Check for NVIDIA driver
ansible.builtin.stat:
path: /dev/nvidia0
register: nvidia_driver

- name: Set fact if NVIDIA driver is present
ansible.builtin.set_fact:
has_nvidia_driver: "{{ nvidia_driver.stat.exists | default(false) }}"

- name: Expose GPU drivers
ansible.builtin.shell: |
source /cvmfs/software.eessi.io/versions/2023.06/init/bash
/cvmfs/software.eessi.io/versions/2023.06/scripts/gpu_support/nvidia/link_nvidia_host_libraries.sh
when: has_nvidia_driver

- name: Configure VGPUs
ansible.builtin.include_role:
name: stackhpc.linux.vgpu
Expand Down
17 changes: 17 additions & 0 deletions ansible/roles/eessi/tasks/configure.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,20 @@
- name: Ensure CVMFS config is setup # noqa: no-changed-when
ansible.builtin.command:
cmd: "cvmfs_config setup"

# configure gpus
- name: Check for NVIDIA driver
Copy link
Collaborator

@sjpb sjpb Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure whether there is always a /dev/nvidia0? Could you check with @jovial please for e.g MIG and vGPU configs? Else we'd have to do something like https://github.com/stackhpc/ansible-role-openhpc/blob/be6196540ca8007a0e45f2c3b2596ed0ff77fc13/library/gpu_info.py#L42 but TBH this approach here is much simpler!

Copy link
Collaborator

@jovial jovial Nov 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not in a slurm appliance, but on an openstack compute host with mig (it's hard for me to login into the slurm deployment with mig, but @priteau could check there):

[stack@gpu3 ~]$ ls -lia /dev/nvidia*
4226 crw-rw-rw-. 1 root root 503,   3 Feb 19  2025 /dev/nvidia-vgpu3
3610 crw-rw-rw-. 1 root root 503,   4 Feb 11  2025 /dev/nvidia-vgpu4
3665 crw-rw-rw-. 1 root root 503,   5 Feb 11  2025 /dev/nvidia-vgpu5
2205 crw-rw-rw-. 1 root root 503,   8 Jan 23  2025 /dev/nvidia-vgpu8
2091 crw-rw-rw-. 1 root root 503,   0 Jan 23  2025 /dev/nvidia-vgpuctl
2082 crw-rw-rw-. 1 root root 195,   0 Jan 23  2025 /dev/nvidia0
2086 crw-rw-rw-. 1 root root 195,   1 Jan 23  2025 /dev/nvidia1
2080 crw-rw-rw-. 1 root root 195, 255 Jan 23  2025 /dev/nvidiactl

So checking for /dev/nvidia0 looks good to me. You will likely not have the nvidia-vgpu* device nodes in a slurm deployment as they only appear if you create a vgpu instance.

Copy link
Collaborator

@sjpb sjpb Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strictly, this is checking for the device, which is only present if the driver is loaded. So I suggest:

Suggested change
- name: Check for NVIDIA driver
- name: Check for NVIDIA GPU

ansible.builtin.stat:
path: /dev/nvidia0
register: nvidia_driver

- name: Set fact if NVIDIA driver is present
Copy link
Collaborator

@sjpb sjpb Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- name: Set fact if NVIDIA driver is present
- name: Set fact if NVIDIA GPU is present

ansible.builtin.set_fact:
has_nvidia_driver: "{{ nvidia_driver.stat.exists | default(false) }}"

- name: Expose GPU drivers
ansible.builtin.shell: |
source /cvmfs/software.eessi.io/versions/2023.06/init/bash
/cvmfs/software.eessi.io/versions/2023.06/scripts/gpu_support/nvidia/link_nvidia_host_libraries.sh
when: has_nvidia_driver
changed_when: true
Loading