Add wildcard tolerations to ComputeDomain pods (issue #305)

[jgehrcke]

For #305. This is not yet tested. Will report back.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[jgehrcke]

OK, I tested this on Luna. Confirming that the expected tolerations are set:

$ kubectl get pod -n nvidia-dra-driver-gpu   imex-channel-injection-c9s9r-5pzxk -o yaml | grep -C15 tolerations
<snip>
  nodeSelector:
    resource.nvidia.com/computeDomain: 25c467f6-b235-477d-a306-09c8e88311a7
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  resourceClaims:
  - name: compute-domain-daemon
    resourceClaimTemplateName: imex-channel-injection-daemon-claim-template-w6qnx
<snip>
  tolerations:
  - effect: NoSchedule
    operator: Exists
  - effect: NoExecute
    operator: Exists
  - effect: PreferNoSchedule
    operator: Exists
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/disk-pressure
    operator: Exists

The top three tolerations are the new catch-alls. The other three are set by k8s by default.

[Copilot]

Pull Request Overview

This PR adds wildcard tolerations for ComputeDomain pods to address issue #305.

• Adds three wildcard tolerations with effects "NoSchedule", "NoExecute", and "PreferNoSchedule".
• Updates the ComputeDomain daemon template to include these tolerations.