feat(recovery): readiness-based reset policy, configurable bounds (no fixed device constants)

[andrescera]

What

Replaces the wedged-device recovery's fixed timing with a readiness-driven, bounded policy.

• Removes RESET_SETTLE_MS 4000 (a settle measured on one DJI Osmo Pocket 3) and the start-path usleep(1000000) // Wait 1 second for USB to settle.
• After the one port reset: poll uvc_find_devices() on a 25→200 ms micro-backoff until the device re-enumerates, reopen, and require an actual delivered frame before declaring recovery. The proving frame is returned via g_async_queue_push_front(), so proving recovery costs no video.
• Two new properties, following the existing max-payload / transfer-buffers pattern (enum + g_object_class_install_property + locked setter/getter):
  • reset-settle-max-ms (uint, 0..120000, default 8000)
  • reset-rearm-frames (uint, 1..100000, default 30)
• Adds tests/board/wedge-recovery.sh — real-hardware only, deliberately not registered in tests/CMakeLists.txt, skips with exit 77 unless CERALIVE_BOARD_TEST=1.

Unchanged on purpose: one port reset per silence episode (retries inside the budget are reopens only), and the reconnect=true ladder. The ladder's loop body was factored into a shared reopen_once() with identical attempt accounting — test_reconnect_exhaustion's exact open_attempts == 6 still passes.

Why

The 4 s constant was device-specific: it over-waited on faster devices and burned the single reopen on slower ones. More seriously, the old code declared success on a clean uvc_start_streaming() — which a still-wedged device returns happily while delivering nothing, i.e. the recovery could report success on the exact condition it exists to fix. libuvc exposes no readiness API, so a delivered frame is the only trustworthy signal.

Three further defects were found on real hardware and each pinned red-first in the mock before its fix:

1. A failed reopen ended the whole recovery — reopen_once() returning false breaks the loop, discarding the remaining budget (observed giving up after 6.9 s of a 30 s budget). Now retried with micro-backoff.
2. The libuvc context goes stale across a re-enumeration — after a real port reset a freshly started process was streaming again at +14.4 s while this element, holding the context it had open across the reset, could not reopen at all within 30 s. uvc_exit() + uvc_init() on the recovery path took the measured recovery from never to 288 ms.
3. The budget could be overrun — deadline checks between steps cap it at one in-flight libuvc call.

How to verify

cmake -B build -DENABLE_SANITIZERS=ON && cmake --build build
ctest --test-dir build --output-on-failure     # 158/158

7 new ctest entries, all red-first (the first five failed with object class 'GstLibuvcH264Src' has no property named 'reset-settle-max-ms'):

reset_policy_property_defaults, reset_readiness_polls_reenumeration, reset_readiness_bounded_by_property, reset_requires_frame_delivery, reset_survives_transient_reopen_failure, reset_refreshes_libuvc_context, reset_rearm_frames_property

The mock gained a MOCK_UVC_FRAME_SILENT mode (start succeeds, endpoint delivers nothing — the measured wedge, which DISCONNECT cannot model) and a mock_uvc_init_count() counter.

No device-measured constants remain:

grep -rn "RESET_SETTLE_MS\b|\b4000\b" libuvch264src/src/   # no match
grep -rn usleep libuvch264src/src/                          # only a historical comment in stop()

Real hardware (RK3588, Osmo Pocket 3 2ca3:0023, gated fault injection):

GATE OPEN pid=520846 drivers=usbfs/usbfs buffers 17->32 advancing=true
T0 external USBDEVFS_RESET on 5-1
PASS cycle 1: reset-to-advancing-frames=277ms (bound 8000ms), buffers 37->97

Declared recovery bound: ~280 ms fast path (277/288 ms), 25 s worst-case tail (21880/21893 ms). RØDE 10-1 negative control read uvcvideo/uvcvideo before and after every run without exception.

Risks / out of scope

reset-settle-max-ms is not a hard bound and is documented as such. It budgets the element's own readiness loop; uvc_stop_streaming() / uvc_close() are synchronous with no interruption seam in this libuvc, and on a re-enumerating device push the total past it (measured ~22 s against 8 s). That is why the declared bound has two tiers — the tail is teardown-bound, not policy-bound.

Named follow-ups, none attempted here:

#	Finding	Owner
A	libuvc's reattach helper double-forks and setsid()s but stays in the creator's cgroup, so KillMode=mixed + FinalKillSignal=9 kills it during unit teardown. Consider KillMode=process. Confirmed live: helper ppid=1 sid=508217 sharing the holder's cgroup	cerastream — packaging/systemd/cerastream.service
B	uvc_claim_if() arms the guard only after a successful claim and never rolls back a successful libusb_detach_kernel_driver(), stranding an interface driverless in a live process	libuvc — src/device.c:1049-1057
C	No session re-establishment after an engine crash-restart: the engine returns to idle and waits for a human, so no in-element recovery can ever run	cerastream / CeraUI
D	The board runs a pre-PR-#20 plugin (USB port reset string count = 0); the device image needs a refreshed gstreamer1.0-libuvch264src	image-building-pipeline / release
E	reset-settle-max-ms was documented as a bound it cannot honour — resolved in this PR as a semantics correction	this repo — done
F	Osmo Pocket 3 intermittently fails to re-enumerate after USBDEVFS_RESET (2 occurrences, error -71, physical replug required). Pre-existing in PR #20's mechanism — this PR changed the timing around the reset, not the reset itself	hardware investigation / camera-compat.md
G	Non-blocking teardown (detached thread + fresh libuvc context) would make the budget real. Not attempted — a real architectural change to UVC teardown needing its own design, red-first tests and board proof	this repo — future

The --repeat 3 unbroken board run was not achieved and further attempts were deliberately not made: repeated port resets in a tight loop knocked the Osmo off the USB bus twice (error -71, physical replug required). Four independent real-hardware observations across separate runs stand in its place. This deviation is recorded explicitly in the frozen board proof.