fix(capture): AU-alignment split — aggregate NALs into one buffer per access unit - #23
Merged
Merged
Conversation
… access unit The element advertises `alignment=au` on both pad templates but pushed one GstBuffer per NAL unit, so any multi-slice picture (common at 2160p) fragmented across multiple buffers that each claimed to be a whole access unit. Downstream (`h264parse`, `v4l2slh264dec`/`mppvideodec`, the muxers) trusts that claim to find frame boundaries. Fix: `frame_callback()` now partitions each libuvc delivery into access units via `split_access_units()` and emits exactly one buffer per AU. Boundary detection, in priority order: an Access Unit Delimiter when present (exact, no heuristic), else first-slice-of-new-picture detection via a one-bit test on the slice payload (H.264 `first_mb_in_slice`/H.265 `first_slice_segment_in_pic_flag`), else never split (deliberately biased toward under-splitting). PTS convention: an aggregated AU carries the delivery's arrival running-time, identical to what its first slice would have received under the old per-NAL path. No caps change, no new timing constant. Board-proven at both resolutions on the Osmo Pocket 3 (192.168.78.131, 2026-07-30 ~08:26-08:43Z): 1080p30 parses cleanly through `h264parse`, byte-identical to the pre-aggregation path; 2160p30 parses cleanly with full SPS-derived caps (3840x2160/high/5.2/4:2:0); exact 90-in/90-out buffer count verified via identity probes on both sides of `h264parse` for a 90-buffer 2160p30 request, confirming the contract holds with zero drift at the resolution most likely to expose a splitting bug. Local gate: 103/103 baseline ctest, both new AU-alignment cases red before the fix (3 buffers / 2 buffers where 1 was expected), 106/106 after (103 + 3 new), 161/161 under `-DENABLE_SANITIZERS=ON` with zero ASan/TSan findings. `compat_caps_contract` unmodified; `alignment=au` unchanged. Board-Evidence-SHA256: 4585c999f72a419edd079cf5b10e1af86105d0eee6cfbfbf76b7ab43e8683800
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes the
alignment=auoutput-buffer contract: the element advertisesalignment=(string)auon both pad templates but pushed oneGstBufferper NALunit, so a multi-slice picture (common at 2160p) fragmented across multiple
buffers that each claimed to be a whole access unit.
frame_callback()now partitions each libuvc delivery into access units via anew
split_access_units()and emits exactly one buffer per AU. Boundarydetection, in priority order: an Access Unit Delimiter when present (exact, no
heuristic), else first-slice-of-new-picture detection (a one-bit test on the
slice payload, covering both H.264 and H.265), else never split.
Why
Downstream (
h264parse,v4l2slh264dec/mppvideodec, the muxers) reads thealignment=aucaps claim to find frame boundaries. A device that emitsmulti-slice pictures — the realistic case at 2160p — was violating that
contract silently; nothing failed loudly, but the output was mis-framed. Caps
are untouched (still
alignment=au— the contract is correct, the emitter wasnot);
compat_caps_contractpasses unmodified.How to verify
Local: characterization-first —
au_single_slice_characterizationwas greenon the unmodified tree before any behavior change; both new multi-slice cases
(
au_multi_slice_aud,au_aud_less_fallback) were red first (3 buffers / 2buffers where 1 was expected), green after the fix.
103/103baseline ctest,106/106after (103 + 3 new),161/161under-DENABLE_SANITIZERS=ONwithzero ASan/TSan findings.
Board hardware evidence is frozen and cited in the commit trailer
(
Board-Evidence-SHA256):.omo/evidence/device-quality-wave3/task-15-board-proof.mdin the rootworkspace repo — real Osmo Pocket 3 capture at both 1080p30 (byte-identical to
the pre-aggregation path through
h264parse) and 2160p30 (full SPS-derivedcaps:
3840x2160/high/5.2/4:2:0), with an exact 90-in/90-out buffer countverified via
identityprobes on both sides ofh264parseat 2160p30 — theAU-alignment contract holds with zero drift at the resolution most likely to
expose a splitting bug.
Risks
comparison against a recorded baseline (
issues.md:2178-2183). Thatmeasurement was not independently re-taken on this board session — the
freeze file says so explicitly rather than fabricating a number. The
pre-existing design reasoning (an AU is emitted exactly when the last NAL of
its delivery used to be pushed, so aggregation adds no additional wait) is
recorded but is design reasoning, not a board measurement.
GST_BUFFER_OFFSETis now an access-unit counter rather than a per-NALcounter. It was already documented as a frame counter, so this makes the
documentation and behavior agree; unchanged for the single-slice case.