Add startup taint removal functionality to azurelustre CSI driver

dabradley · dabradley · commit 3abbd5678c73 · 2025-09-11T15:24:51.000-04:00
The taint key follows the pattern: {driverName}/agent-not-ready
e.g., azurelustre.csi.azure.com/agent-not-ready

This allows users to apply startup taints to prevent scheduling before the CSI driver is ready,
addressing potential race conditions during node startup.
diff --git a/README.md b/README.md
@@ -3,7 +3,7 @@
 [![Coverage Status](https://coveralls.io/repos/github/kubernetes-sigs/azurelustre-csi-driver/badge.svg?branch=main)](https://coveralls.io/github/kubernetes-sigs/azurelustre-csi-driver?branch=main)
 [![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fkubernetes-sigs%2Fazurelustre-csi-driver.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2Fkubernetes-sigs%2Fazurelustre-csi-driver?ref=badge_shield)
 
-### About
+## About
 
 This driver allows Kubernetes to access Azure Lustre file system.
 
@@ -12,7 +12,7 @@ This driver allows Kubernetes to access Azure Lustre file system.
 
 &nbsp;
 
-### Container Images & Kubernetes Compatibility:
+### Container Images & Kubernetes Compatibility
 
 | Driver version  | Image                                                           | Supported k8s version | Lustre client version |
 |-----------------|-----------------------------------------------------------------|-----------------------|-----------------------|
diff --git a/deploy/rbac-csi-azurelustre-node.yaml b/deploy/rbac-csi-azurelustre-node.yaml
@@ -14,6 +14,9 @@ rules:
   - apiGroups: [""]
     resources: ["secrets"]
     verbs: ["get", "list"]
+  - apiGroups: [""]
+    resources: ["nodes"]
+    verbs: ["get", "patch"]
 
 ---
 kind: ClusterRoleBinding
diff --git a/docs/csi-debug.md b/docs/csi-debug.md
@@ -448,6 +448,75 @@ Check for solutions in [Resolving Common Errors](errors.md)
 
 ---
 
+## Pod Scheduling and Node Readiness Issues
+
+### Pods Stuck in Pending Status with Taint-Related Errors
+
+**Symptoms:**
+
+- Pods requiring Azure Lustre storage remain in `Pending` status
+- Pod events show taint-related scheduling failures
+- Error messages mentioning `azurelustre.csi.azure.com/agent-not-ready` taint
+
+**Check pod scheduling status:**
+
+```sh
+kubectl describe pod <pod-name>
+```
+
+Look for events such as:
+
+- `Warning  FailedScheduling  ... node(s) had taint {azurelustre.csi.azure.com/agent-not-ready: }, that the pod didn't tolerate`
+- `0/X nodes are available: X node(s) had taint {azurelustre.csi.azure.com/agent-not-ready}`
+
+**Check node taints:**
+
+```sh
+kubectl describe nodes | grep -A5 -B5 "azurelustre.csi.azure.com/agent-not-ready"
+```
+
+**Check CSI driver readiness on nodes:**
+
+```sh
+# Check if CSI driver pods are running on all nodes
+kubectl get pods -n kube-system -l app=csi-azurelustre-node -o wide
+
+# Check CSI driver logs for startup issues
+kubectl logs -n kube-system -l app=csi-azurelustre-node -c azurelustre --tail=100 | grep -i "taint\|ready\|error"
+```
+
+**Common causes and solutions:**
+
+1. **CSI Driver Still Starting**: Wait for CSI driver pods to reach `Running` status
+
+   ```sh
+   kubectl wait --for=condition=ready pod -l app=csi-azurelustre-node -n kube-system --timeout=300s
+   ```
+
+2. **Lustre Module Loading Issues**: Check if Lustre kernel modules are properly loaded
+
+   ```sh
+   kubectl exec -n kube-system <csi-azurelustre-node-pod> -c azurelustre -- lsmod | grep lustre
+   ```
+
+3. **Manual Taint Removal** (Emergency only - not recommended for production):
+
+   ```sh
+   kubectl taint nodes <node-name> azurelustre.csi.azure.com/agent-not-ready:NoSchedule-
+   ```
+
+**Verify taint removal functionality:**
+
+Check that startup taint removal is enabled in the CSI driver:
+
+```sh
+kubectl logs -n kube-system -l app=csi-azurelustre-node -c azurelustre | grep -i "remove.*taint"
+```
+
+Expected log output should show taint removal activity when the driver becomes ready.
+
+---
+
 ## Get Azure Lustre Driver Version
 
 ```sh
diff --git a/docs/driver-parameters.md b/docs/driver-parameters.md
@@ -4,6 +4,29 @@ These are the parameters to be passed into the custom StorageClass that users mu
 
 For more information, see the [Azure Managed Lustre Filesystem (AMLFS) service documentation](https://learn.microsoft.com/en-us/azure/azure-managed-lustre/) and the [AMLFS CSI documentation](https://learn.microsoft.com/en-us/azure/azure-managed-lustre/use-csi-driver-kubernetes).
 
+## CSI Driver Configuration Parameters
+
+These parameters control the behavior of the Azure Lustre CSI driver itself and are typically configured during driver installation rather than in StorageClass definitions.
+
+### Node Startup Taint Management
+
+Name | Meaning | Available Value | Default Value | Configuration Method
+--- | --- | --- | --- | ---
+remove-not-ready-taint | Controls whether the CSI driver automatically removes startup taints from nodes when the driver becomes ready. This ensures pods are only scheduled to nodes where the CSI driver is fully operational and Lustre filesystem capacity is available. Nodes should have a taint of the form: `azurelustre.csi.azure.com/agent-not-ready:NoSchedule` | `true`, `false` | `true` | Command-line flag `--remove-not-ready-taint` in driver deployment
+
+#### Startup Taint Details
+
+When enabled (default), the Azure Lustre CSI driver will:
+
+1. **Monitor Node Readiness**: Check if the CSI driver is fully initialized on the node
+2. **Remove Blocking Taint**: Automatically remove the `azurelustre.csi.azure.com/agent-not-ready:NoSchedule` taint when ready
+
+This mechanism prevents pods requiring Azure Lustre storage from being scheduled to nodes where:
+
+- Lustre kernel modules are not yet loaded
+- CSI driver components are not fully initialized  
+- Network connectivity to Lustre filesystems is not established
+
 ## Dynamic Provisioning (Create an AMLFS Cluster through AKS) - Public Preview
 
 > **Public Preview Notice**: Dynamic provisioning functionality is currently in public preview. Some features may not be supported or may have constrained capabilities.
diff --git a/docs/errors.md b/docs/errors.md
@@ -11,6 +11,9 @@ This document describes common errors that can occur during volume creation and
     - [Error: Resource not found](#error-resource-not-found)
     - [Error: Cannot create AMLFS cluster, not enough IP addresses available](#error-cannot-create-amlfs-cluster-not-enough-ip-addresses-available)
     - [Error: Reached Azure Subscription Quota Limit for AMLFS Clusters](#error-reached-azure-subscription-quota-limit-for-amlfs-clusters)
+- [Pod Scheduling Errors](#pod-scheduling-errors)
+  - [Node Readiness and Taint Errors](#node-readiness-and-taint-errors)
+    - [Error: Node had taint azurelustre.csi.azure.com/agent-not-ready](#error-node-had-taint-azurelustrecsiazurecomagent-not-ready)
 - [Volume Mounting Errors](#volume-mounting-errors)
   - [Node Mount Errors](#node-mount-errors)
     - [Error: Could not mount target](#error-could-not-mount-target)
@@ -31,7 +34,7 @@ This document describes common errors that can occur during volume creation and
     - [Controller Logs](#controller-logs)
     - [Node Logs](#node-logs)
     - [Comprehensive Log Collection](#comprehensive-log-collection)
-  
+
 ---
 
 ## Volume Creation Errors
@@ -211,6 +214,87 @@ There is not enough room in the /subscriptions/<sub-id>/resourceGroups/<rg>/prov
 
 ---
 
+## Pod Scheduling Errors
+
+### Node Readiness and Taint Errors
+
+#### Error: Node had taint azurelustre.csi.azure.com/agent-not-ready
+
+**Symptoms:**
+
+- Pods requiring Azure Lustre storage remain stuck in `Pending` status
+- Pod events show taint-related scheduling failures:
+  - `Warning  FailedScheduling  ... node(s) had taint {azurelustre.csi.azure.com/agent-not-ready: }, that the pod didn't tolerate`
+  - `0/X nodes are available: X node(s) had taint {azurelustre.csi.azure.com/agent-not-ready}`
+- Kubectl describe pod shows scheduling failures due to taints
+
+**Possible Causes:**
+
+- CSI driver is still initializing on nodes
+- Lustre kernel modules are not yet loaded
+- CSI driver failed to start properly on affected nodes
+- Node is not ready to handle Azure Lustre volume allocations
+- CSI driver startup taint removal is disabled
+
+**Debugging Steps:**
+
+```bash
+# Check pod scheduling status
+kubectl describe pod <pod-name> | grep -A10 Events
+
+# Check which nodes have the taint
+kubectl describe nodes | grep -A5 -B5 "azurelustre.csi.azure.com/agent-not-ready"
+
+# Verify CSI driver pod status on nodes
+kubectl get pods -n kube-system -l app=csi-azurelustre-node -o wide
+
+# Check CSI driver startup logs
+kubectl logs -n kube-system -l app=csi-azurelustre-node -c azurelustre --tail=100 | grep -i "taint\|ready\|error"
+
+# Verify taint removal is enabled (should be true by default)
+kubectl logs -n kube-system -l app=csi-azurelustre-node -c azurelustre | grep -i "remove.*taint"
+```
+
+**Resolution:**
+
+1. **Wait for CSI Driver Readiness** (most common case):
+
+   ```bash
+   # Wait for CSI driver pods to reach Running status
+   kubectl wait --for=condition=ready pod -l app=csi-azurelustre-node -n kube-system --timeout=300s
+   ```
+
+   The taint should be automatically removed once the CSI driver is fully operational.
+
+2. **Check Lustre Module Loading**:
+
+   ```bash
+   # Verify Lustre modules are loaded on nodes
+   kubectl exec -n kube-system <csi-azurelustre-node-pod> -c azurelustre -- lsmod | grep lustre
+   ```
+
+3. **Verify CSI Driver Configuration**:
+
+   ```bash
+   # Check if taint removal is enabled (default: true)
+   kubectl get deployment csi-azurelustre-node -n kube-system -o yaml | grep "remove-not-ready-taint"
+   ```
+
+4. **Emergency Manual Taint Removal** (not recommended for production):
+
+   ```bash
+   # Only use if CSI driver is confirmed working but taint persists
+   kubectl taint nodes <node-name> azurelustre.csi.azure.com/agent-not-ready:NoSchedule-
+   ```
+
+**Prevention:**
+
+- Ensure CSI driver has sufficient time to initialize during cluster updates
+- Monitor CSI driver health during node scaling operations
+- Use pod disruption budgets to prevent scheduling issues during maintenance
+
+---
+
 ## Volume Mounting Errors
 
 ### Node Mount Errors
diff --git a/docs/install-csi-driver.md b/docs/install-csi-driver.md
@@ -121,3 +121,23 @@ The CSI driver deployment includes automated **exec-based readiness probes** for
 
 **Important**: The enhanced validation ensures the driver reports ready only when LNet is fully functional for Lustre operations. Wait for all CSI driver node pods to pass enhanced readiness checks before creating PersistentVolumes or mounting Lustre filesystems.
 
+## Startup Taints
+
+When the CSI driver starts on each node, it automatically removes the following taint if present:
+
+- **Taint Key**: `azurelustre.csi.azure.com/agent-not-ready`
+- **Taint Effect**: `NoSchedule`
+
+This ensures that:
+
+1. **Node Readiness**: Pods requiring Azure Lustre storage are only scheduled to nodes where the CSI driver is fully initialized
+2. **Lustre Client Ready**: The node has successfully loaded Lustre kernel modules and networking components
+
+### Configuring Startup Taint Behavior
+
+The startup taint functionality is enabled by default but can be configured during installation:
+
+- **Default Behavior**: Startup taint removal is **enabled** by default
+- **Disable Taint Removal**: To disable, set `--remove-not-ready-taint=false` in the driver deployment
+
+For most AKS users, the default behavior provides optimal pod scheduling and should not be changed
diff --git a/go.mod b/go.mod
@@ -19,7 +19,9 @@ require (
 	golang.org/x/net v0.44.0
 	google.golang.org/grpc v1.75.1
 	google.golang.org/protobuf v1.36.9
+	k8s.io/api v0.31.13
 	k8s.io/apimachinery v0.31.13
+	k8s.io/client-go v1.5.2
 	k8s.io/klog/v2 v2.130.1
 	k8s.io/kubernetes v1.31.13
 	k8s.io/mount-utils v0.31.6
@@ -122,10 +124,8 @@ require (
 	gopkg.in/inf.v0 v0.9.1 // indirect
 	gopkg.in/yaml.v2 v2.4.0 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
-	k8s.io/api v0.31.13 // indirect
 	k8s.io/apiextensions-apiserver v0.31.1 // indirect
 	k8s.io/apiserver v0.31.13 // indirect
-	k8s.io/client-go v1.5.2 // indirect
 	k8s.io/cloud-provider v0.31.1 // indirect
 	k8s.io/component-base v0.31.13 // indirect
 	k8s.io/component-helpers v0.31.13 // indirect
diff --git a/pkg/azurelustre/azurelustre.go b/pkg/azurelustre/azurelustre.go
diff --git a/pkg/azurelustre/azurelustre_test.go b/pkg/azurelustre/azurelustre_test.go
diff --git a/pkg/azurelustreplugin/main.go b/pkg/azurelustreplugin/main.go
