Runtime audio-volume probe seeks live GSAP timeline and applies future tl.set state to earlier chunks

[JonathanRosado]

Describe the bug

bindMediaMetadataListeners() calls probeAndCacheElementVolume() for every <audio>/<video>, including media with only static data-volume. The probe synchronously seeks the live captured GSAP timeline at 60 Hz through the media/composition window, then restores only the numeric playhead.

In a distributed worker cold-started at a non-zero frame, a later zero-duration tl.set(..., { visibility: "hidden" }) in a nested/sub-composition can remain materialized before its authored time. The render succeeds with no error but silently omits visible content. Sequential preview/check can pass.

Reproduced with v0.7.72 and current main e2e61b0767b4b4dd282773eb50aa7ebaea98f0e5; the relevant source is unchanged on main:

• Runtime binds and probes media: init.ts
• Probe walks the live timeline at 60 Hz and restores only totalTime: mediaVolumeEnvelope.ts
• Existing regression checks a timeline mock's number and audio.volume, but not non-audio DOM/plugin/nested state: mediaVolumeEnvelope.test.ts

Link to reproduction

https://github.com/heygen-com/hyperframes/tree/v0.7.72/packages/core/src/runtime

Minimal reproduction

Root composition, 12 s / 24 fps:

<div id="root" data-composition-id="main" data-start="0" data-duration="12"
     data-width="640" data-height="360">
  <div id="child-host" class="clip" data-composition-id="child"
       data-composition-src="compositions/child.html"
       data-start="2" data-duration="10" data-track-index="0"></div>
  <audio id="bed" src="assets/silence.wav" data-start="0" data-duration="12"
         data-track-index="1" data-volume="1"></audio>
</div>
<script>
window.__timelines = window.__timelines || {};
window.__timelines.main = gsap.timeline({paused:true});
</script>

Child composition:

<div id="child-root" data-composition-id="child" data-start="0" data-duration="10"
     data-width="640" data-height="360">
  <div class="future-state">MUST BE VISIBLE</div>
</div>
<script>
window.__timelines = window.__timelines || {};
const tl = gsap.timeline({paused:true});
tl.to(".future-state", {opacity:.2, duration:.25}, 9.4);
tl.set(".future-state", {visibility:"hidden"}, 9.65);
window.__timelines.child = tl;
</script>

Generate silence:

ffmpeg -f lavfi -i anullsrc=r=48000:cl=mono -t 12 \
  -c:a pcm_s16le assets/silence.wav

1. npx hyperframes@0.7.72 check passes.
2. Render through the distributed path with 120-frame chunks so a worker cold-starts at 5 s.
3. Inspect master 7 s (child local 5 s), well before the hide at master 11.65 s.
4. Control: remove only the terminal tl.set, or suppress runtime volume probing before runtime initialization.

The public fixture needs a true distributed/non-zero cold-start execution. Local multi-worker capture has a warmup path and may not reproduce the Cloud chunk behavior.

Expected behavior

The child panel is visible at 7 s. Metadata/audio discovery is observational: it must not alter visual timeline state at any time.

Actual behavior

The worker encodes the panel hidden before its authored hide time.

In the production reproduction, a nested subtree had computed visibility:hidden at child-local 20.792 s although its only hide was at 43.833 s. Parent host/scene visibility and the child playhead were correct. Removing only the terminal set restored the subtree. Setting immediateRender:false on the preceding tween did not.

All eight audio clips were static (data-volume="1"), with no volume tween or .volume= assignment.

Root cause / regression history

#1118 introduced runtime full-timeline volume sampling. #2141/#2143 fixed one symptom by restoring the playhead number, but timeline.totalTime(originalTime) is not a sufficient transactional rollback for all GSAP CSS/plugin/nested-timeline state during a non-zero cold start. The probe also runs unnecessarily for static-volume media.

Closest prior work, not duplicates:

• fix(core): apply renderer volume-automation solution to preview #1118
• Timeline volume tween causes --variables to be silently ignored at render #2141
• fix(core): preserve playhead during volume probing #2143
• Cold-seek lint work fix(lint): catch cold-seek opacity reveals #2503/feat(lint): seek-order safety and SVG draw-on rules for GSAP timelines #2611 covers adjacent author-side hazards; this issue is framework initialization mutating valid authored state.

Proposed invariant and fix

Invariant: runtime initialization, metadata binding, and audio-envelope discovery must perform zero seeks on the live visual capture timeline. A worker's first authored seek must produce the same DOM/computed state as a fresh page seek to that time.

Minimum safe fix:

1. Reuse/extend hasScriptedAudioVolumeAutomation() as a compiled hint and skip probeAndCacheElementVolume() entirely for static-volume compositions/elements.
2. For real scripted volume automation, discover the envelope in an isolated probe page/timeline and serialize/hydrate the keyframes; do not scan the live chunk-capture timeline.
3. Do not treat restoring the numeric playhead as a DOM-state rollback.

Regression gates:

• Unit: static-volume media causes zero timeline seek calls and no cache write.
• Distributed golden: nested child with a future terminal tl.set, static audio, and a non-zero chunk start; first/mid chunk frames must match the sequential baseline.
• Assert computed visibility remains visible before the terminal-set timestamp.

Verified workaround evidence

A renderer-only guard returned before probing for a composition proven to have static audio, injected before runtime initialization in probe/chunk pages. It changed no composition or pixel path.

A fresh 28-chunk production render then passed all 28 cold-start boundaries and restored the previously missing subtree; 6,611/6,611 frames completed. A regression test asserted seek count 0 and no volume-envelope cache entry.

Environment

HyperFrames: 0.7.72 and main e2e61b0767b4b4dd282773eb50aa7ebaea98f0e5
Runtime: GCP Cloud Run distributed render / Plan V2
Chrome: 148, screenshot fallback
FPS: 24
Trigger: non-zero-start chunk with nested GSAP sub-composition

The invariant should be capture-backend independent.

Duplicate search

No matching issue was found. #2141 is about variables being ignored when audio volume is animated; it does not cover static audio causing initialization-time seeks or future DOM state leaking into earlier cold-start chunks.

[jrusso1020]

Closed by #2816 (merge commit 6c98e540649e).

Root cause

This was a real framework bug, not an authored-timeline or user configuration issue. Runtime media initialization called probeAndCacheElementVolume() for every audio/video element, including static data-volume media. The helper sampled the live captured GSAP timeline across the media window and restored only the numeric playhead. On a non-zero distributed worker cold start, future nested zero-duration state such as tl.set(..., { visibility: "hidden" }) could remain materialized before its authored time, silently removing valid content from earlier chunks.

Producer renders did not need this live visual-timeline probe: scripted volume automation was already discovered in an isolated producer probe page and baked into audio before frame capture.

Fix

• Mark producer-served render/probe pages before the HyperFrames runtime loads.
• Disable live timeline volume-envelope discovery on those pages.
• Preserve Studio/preview behavior and the producer-side isolated scripted-volume discovery and baking path.
• Add the reported 12-second, 24-fps, 120-frame nested future-tl.set distributed regression fixture.

Validation

Core runtime and producer typechecks passed, along with 811 core runtime tests, the producer file-server suite, in-process and distributed-simulated goldens, Windows, and full regression CI. Live owner-isolated GCP v1/v2 runs were byte-identical with exact decoded video/audio/metadata parity, and the non-zero-start chunk correctly retained the MUST BE VISIBLE panel at master 7 seconds.