Fix NVMe hot-swap LED stuck in FAILURE under VMD controllers#275
Open
tasleson wants to merge 4 commits into
Open
Fix NVMe hot-swap LED stuck in FAILURE under VMD controllers#275tasleson wants to merge 4 commits into
tasleson wants to merge 4 commits into
Conversation
NVMe udev events may arrive with a virtual nvme-subsystem sysfs path (e.g. /sys/devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1) that cannot be resolved to a PCI controller. This causes block_device_init() to fail in _compare(), silently dropping add and remove events. Add a devnode name fallback to _compare() so that virtual nvme-subsystem paths are matched to their corresponding block device. This restores udev event processing for NVMe hot-swap under VMD controllers. Signed-off-by: Tony Asleson <tasleson@redhat.com> Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Claude Opus 4.6 <noreply@anthropic.com>
FAILED_DRIVE is intentionally sticky across sysfs scan cycles to prevent RAID member fault LEDs from flickering. However, this also prevents standalone NVMe drives from recovering after a hot-swap cycle, leaving the failure LED on permanently. When a non-RAID device in FAILED_DRIVE state reappears in the sysfs scan, transition it to ADDED so the normal state machine can drive it back to healthy operation. RAID members remain sticky and require mdadm or other raid tool intervention to clear the fault. Signed-off-by: Tony Asleson <tasleson@redhat.com> Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Claude Opus 4.6 <noreply@anthropic.com>
When ledctl walks sysfs for --list-slots, VMD PCI slot entries may persist after a drive is physically removed because the VMD controller maintains the PCI topology. This causes ledctl to report a device in the slot that is no longer present. Add a stat() check on the device node to verify the block device actually exists before associating it with the slot. Signed-off-by: Tony Asleson <tasleson@redhat.com> Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Claude Opus 4.6 <noreply@anthropic.com>
mtkaczyk
reviewed
Apr 10, 2026
mtkaczyk
reviewed
Apr 10, 2026
Collaborator
The previous commit unconditionally allows non-RAID devices to recover from FAILED_DRIVE state on re-add, which changes legacy behavior that some deployments may rely on (e.g. using persistent failure LEDs as an indicator that a drive misbehaved). Add BLINK_PERSISTENT_FAIL_ON_READD config option (default TRUE) so that legacy behavior is preserved. Users who want automatic recovery for standalone NVMe hot-swap cycles can set it to FALSE. Signed-off-by: Tony Asleson <tasleson@redhat.com> Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Claude Opus 4.6 <noreply@anthropic.com>
Collaborator
Author
Collaborator
|
I'm working on running tests on this PR. |
Collaborator
|
@tasleson |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
NVMe udev events arrive with a virtual nvme-subsystem sysfs path (e.g.
/sys/devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1), but ledmon stores block devices using their physicalPCI sysfs path. The
_compare()function in udev event handling callsblock_device_init()on the virtual path,which fails because
block_get_controller()cannot match a virtual path to any PCI controller. This silently dropsall add and remove udev events for NVMe devices.
Without udev event processing, ledmon detects removal only through a timestamp mismatch in
_send_msg(), whichsets
FAILED_DRIVE. On re-insertion the device reappears in the sysfs scan, butFAILED_DRIVEis intentionallysticky in
_add_block()(to protect RAID members), so the state is never cleared. The udevaddevent that wouldnormally break out of this via the
ADDED→ONESHOT_NORMAL→UNKNOWNstate machine is never matched.Changes
Fix udev event matching — Add a devnode name fallback to
_compare()so that virtual nvme-subsystem pathsare matched to their corresponding block device.
Allow non-RAID recovery from FAILED_DRIVE — When a non-RAID device in
FAILED_DRIVEstate reappears in thesysfs scan, transition it to
ADDEDso the state machine can drive it back to normal. RAID members remain stickyand require explicit intervention regardless of whether the removal was intentional or caused by
hardware. Note: This change may need to be placed behind a configuration setting
Validate device node in ledctl slot reporting — Add a
stat()check on the device node before associating ablock device with a PCI slot, so
ledctl --list-slotsdoes not report a device that is no longer present.config option — The previous commit unconditionally allows non-RAID devices to recover from FAILED_DRIVE state on re-add, which changes legacy behavior that some deployments may rely on (e.g. using persistent failure LEDs as an indicator that a drive misbehaved). Add a config option to enable this new behavior.
This needs careful review and ideally testing from user supplied issue.
Resolves: #274