Investigate soft enforcement for GPU reservation locking

[acalhounRH]

Context

From the May 28 demo meeting, Ashish suggested exploring soft enforcement mechanisms to lock specific GPU resources to namespaces upon reservation, preventing usage by other parties. This was flagged as "needs further discussion."

Current State

• Reservations are honor-based only — the system tracks bookings but does not enforce them at the cluster level
• No Kubernetes namespace locking, NetworkPolicy, RBAC, or quota enforcement is tied to reservations
• Hearth lock status is displayed read-only but not controlled by the reservation system
• Conflict detection prevents double-booking in the UI but does not prevent actual workload scheduling on the cluster

Proposed Investigation

• Evaluate feasibility of tying reservations to Kubernetes-level enforcement:
  • ResourceQuotas per namespace to limit GPU access
  • Kueue ClusterQueues/LocalQueues to gate GPU scheduling
  • NetworkPolicies or namespace isolation
  • Hearth lock API integration to lock/unlock clusters on reservation start/end
• Determine the appropriate level of enforcement (advisory vs. blocking)
• Assess impact on existing workflows and cross-team usage patterns
• Consider opt-in vs. mandatory enforcement

Decision Needed

The team needs to align on whether enforcement should be:

1. Advisory only (current state + better visibility)
2. Soft enforcement (warnings/notifications when violating reservations)
3. Hard enforcement (Kubernetes-level blocking)