Snapshot of DRBD,LUKS,STORAGE with two diskful replicas deadlocks: SuspendLayerUtils also suspends the LUKS layer (sub-root filter is dead code) #540

Versions: LINSTOR 1.35.2 (also 1.34.2), DRBD 9.3.4 (also 9.3.3), drbd-utils 9.34.3,
piraeus-operator 2.12.0, kernel 6.18.

Topology: layer stack DRBD,LUKS,STORAGE (ZFS zvol), protocol C, two diskful replicas plus a
diskless tiebreaker. The Primary has a mounted XFS filesystem; the diskful Secondary has nothing
mounted.

Symptom

Every CSI snapshot of these resources fails on the Primary:

Failed to suspend IO for resource <rsc> on layer DrbdLayer
Command 'drbdsetup suspend-io <minor>' did not terminate within 5 seconds   (drbdadm exit 20)

Application writes on the Primary freeze for 9–46 s per attempt. Each freeze ends only when the
Secondary's dm-crypt device is resumed externally, or when DRBD's ko-count (7 × 6 s) ejects the
Secondary.

Resources that have LUKS with a single diskful replica, or no LUKS at all, are not affected. The
same topology was clean on LINSTOR 1.33.3 with DRBD 9.3.2, and fails from 1.34.2 onward.

Root cause

1. LUKS gets flagged for suspension below DRBD. In
   SuspendLayerUtils.setShouldSuspendStateRec (controller, SuspendLayerUtils.java:70-86 @v1.35.2):

   • line 77 calls absRscLayerObjRef.setShouldSuspendIo(suspendStateRef) unconditionally;
   • line 84 then recurses into every child;
   • so the LAYERS_TO_SUSPEND_SUB_ROOT filter {WRITECACHE, CACHE, DRBD} at lines 80-83 never
     takes effect.

   Every layer is flagged, including LUKS and STORAGE. You can see this in the layer-resource
   records: all layers of a resource are updated in lockstep. The behaviour dates from 7047c32
   (1.33.0).

   It was harmless until d5b4701 (1.34.0), which made LuksVlmData report exists(). Since
   then the satellite's SuspendManager runs dmsetup suspend Linstor-Crypt-… right after
   drbdadm suspend-io, in the same DeviceManager run, on every diskful node
   (SuspendManager.java:69-121).

2. DRBD ≥ 9.3.0 freezes the filesystem first. Admin suspend-io calls bdev_freeze() before it
   sets susp_user and waits for io_drained (drbd_nl.c:6614-6651 @drbd-9.3.4; commits 27ca01bf,
   28219cfb). On the Primary, the XFS freeze always writes log and superblock blocks. Under
   protocol C, those writes need the Secondary's ack.

3. No cross-node ordering, and nothing bounds the wait. The satellites suspend independently.

   1. The Secondary, with nothing mounted, finishes its DRBD and LUKS suspends in about 0.1 s.
   2. The Primary's freeze write then queues inside the Secondary's suspended dm-crypt.
   3. The Primary's drbdsetup blocks in bdev_freeze (D state).
   4. drbdadm's cmd-timeout-short alarm exits 20, but the orphaned drbdsetup keeps the
      inherited stdout/stderr pipes open.
   5. ExtCmd.syncProcess → OutputReceiver.finish() waits for EOF with no timeout, so the
      Primary's DeviceManager stays blocked.
   6. The controller aborts the snapshot only after every satellite has answered, so it cannot
      resume the Secondary in the meantime.

   Only ko-count or an external dm-crypt resume breaks the cycle.

Evidence (production cluster, 2026-10-01..10-08)

• 84 of 84 failed attempts show the Secondary reporting "Suspended IO" 0.10–0.17 s after
  registration, while the Primary never reports.
• The Primary's suspend completes 5–23 ms after an external dmsetup resume of the Secondary's
  Linstor-Crypt-*, or exactly at a ko-count ejection (42–43 s). A DRBD-only resume releases
  nothing.
• Successful attempts are those where the Secondary happened to be slower than the Primary's freeze
  write. That makes the outcome depend on placement: with the Primary on the other node, the same
  resource succeeded.

Proposed fix (PR to follow)

Make the recursion honour the documented set: the topmost layer is always flagged, lower layers only
if they are listed.

-        absRscLayerObjRef.setShouldSuspendIo(suspendStateRef);
+        if (absRscLayerObjRef.getParent() == null || layersToApply.contains(absRscLayerObjRef.getLayerKind()))
+        {
+            absRscLayerObjRef.setShouldSuspendIo(suspendStateRef);
+        }

The per-child conditional at lines 80-83 is removed and the recursion is kept. Resume still clears
all layers.

This restores the ≤1.33 behaviour for DRBD,LUKS,STORAGE. DRBD is suspended and drained on all nodes
before the controller takes snapshots, and the LUKS layer is left alone. That is consistent:

• dm-crypt completes a write bio only after the lower device has completed it;
• DRBD acks a peer write only after local completion;
• the controller takes the snapshot only after every node reports suspended.

LUKS-as-root stacks are unchanged. WRITECACHE and CACHE below DRBD keep their flush-only suspend.

Possible separate hardening:

• a bounded wait for pipe EOF after the child process exits;
• a suspend-phase timeout that re-checks resources whose suspend lands late.

We will open a PR with the fix and unit tests (stacks DRBD,LUKS,STORAGE; LUKS,STORAGE;
DRBD,WRITECACHE,STORAGE; DRBD,CACHE,STORAGE; resume) after validating it on a test cluster.

Note that this issue was created with the help of Claude-Opus-5.5 and might contain some hallucinations/inconsistancies as I am not that familiar with the source code. Take everything it says on "root cause" onwards with a grain of salt. Thanks