Audio-reactive parameters - #109
Open
git-chad wants to merge 21 commits into
Open
Conversation
Adds the dependency-free half of audio-reactive parameters. Nothing imports it yet, so there is no runtime impact. Two-stage analysis, which is the load-bearing design choice: Stage A (expensive, once per source) decodes to mono PCM, runs an STFT and reduces each frame to 64 log-spaced band magnitudes plus a windowed RMS scalar, then discards the PCM. Stage B (~5ms for a 5 minute track) turns that into one normalized, smoothed envelope per band. Because it is cheap it can re-run on every band-config edit, which is what makes editing frequency ranges feel instant instead of triggering a multi-second re-analysis. Normalization is self-calibrating: each band maps its own 5th..99th percentile onto [0,1]. A fixed dB window does not survive real material — measured against a loudness-compressed master, a 48dB window left every band clustered between 0.70 and 0.95, which reads as visually static. A 9dB span floor stops a near-constant band being stretched into full-swing noise, and a 60dB cross-band gate stops FFT spectral leakage from faking a band with no content. Smoothing runs after normalization, which gives a provable invariant: every envelope sample lies in [0,1]. The modulation math depends on it. Everything here is free of browser globals so it runs under `bun test`; the 83 tests cover band isolation on synthetic PCM, temporal discrimination, quiet-track usability, silence, attack/release curves, and the [0,1] invariant. Modulation deliberately avoids a new AnimatedPropertyBinding kind: the binding says what is driven, the band says what drives it. That keeps the existing non-exhaustive `binding.kind` branches correct and means the published runtime package needs no mirrored type changes.
Adds "audio" to ASSET_KINDS with mime + extension sniffing (browsers report audio types inconsistently, and often not at all for .m4a), and an `<audio preload="metadata">` duration probe mirroring the existing video one. Order matters in inferFileAssetKind: audio/* is checked after video/*, so an mp4 carrying an audio track still resolves as a video asset. Also removes asset-store's private `inferAssetKind`, which was a byte-identical copy of `inferFileAssetKind` in lib/editor/media-file. Adding the audio branch twice would have let the upload and drag-and-drop paths drift apart. No layer type takes an audio asset — the project audio source is chosen from the timeline, not attached to a layer — so getExpectedAssetKind still returns null for every layer type.
Decoding stays on the main thread because Web Audio is unavailable in a worker; only the FFT pass is offloaded, receiving already-decoded PCM by transfer rather than copy (~57MB for a 5 minute track). Falls back to a chunked main-thread generator where Workers are unavailable. updateBand re-runs stage B only. It must never re-decode or re-run the FFT, or dragging a frequency boundary would stall for seconds. restoreSnapshot is separate from replaceState on purpose: undo must restore link and band config while keeping the decoded spectrogram, whereas loading a project has to discard it because blob URLs do not persist. Measured on a 130s track: stage A 294ms, stage B 8ms, 2MB retained spectrogram.
…nshots buildRendererFrame gains an explicit optional `audio` input and applies applyAudioModulation after evaluateTimelineForLayers. Audio runs last so it wins on conflict, and so a per-component vector link merges into a keyframed value rather than discarding it. evaluateTimelineForLayers is untouched. Passed explicitly rather than read from a store so buildRendererFrame stays a pure function of its input. The offline exporter has to be able to reproduce a frame exactly, and buildRenderProjectState exists precisely so an export is a self-describing snapshot; a module-level registry would have made an export's output depend on hidden ambient state. `audio` sits as a sibling of `timeline` in RenderProjectState, never inside it: renderFrameToCanvas structuredClones the timeline several times per frame, and cloning the envelopes there would cost gigabytes of allocation churn over a full export. The agent screenshot path gets the same treatment — without it a screenshot at time t would show static values while the canvas showed audio-driven ones. Verified against a real 130s track through buildRendererFrame: 119/120 distinct values over a 4s window, always within the parameter's declared bounds, byte identical at the same timestamp, and layer.params left untouched.
Serializes bands, links, offset and source. Envelopes and the spectrogram are derived and deliberately excluded — they are rebuilt by re-analysing on import. Bumps the format to version 3 and replaces the closed `1 | 2` literal union with a max-version check, so older files keep importing and bumping the format is now a one-constant change. A newer-than-known version is still rejected with the existing message rather than silently loading a project this build cannot represent. normalizeProjectAudio coerces absent, partial or hand-edited audio blocks rather than failing the import: files written before audio existed simply get defaults, and malformed links are dropped.
Adds the serializable audio slice to EditorHistorySnapshot and a third store subscription alongside layers and timeline. Audio is part of the history signature: flushPendingHistory compares signatures and discards a snapshot it believes unchanged, so without this an audio-only edit would be silently un-undoable. Undo routes through restoreSnapshot rather than replaceState so multi-second analysis work survives undoing a link edit.
Pre-existing bug, unrelated to audio; found while debugging why an audio link vanished on undo. getHistorySnapshotSignature included the whole timeline snapshot, which contains currentTime. While the timeline plays, currentTime changes every frame, so the signature changed every frame, so every frame re-armed the 220ms commit debounce and it never fired. Any edit made while playing was never committed to history, and the next undo restored a base snapshot from before it — silently discarding the work. Traced in a real session: dozens of `schedule` calls per second and a single `push`, whose base predated the edit being undone. The signature now covers document state only — layers, tracks, duration, loop and audio. Consequences worth noting: moving the playhead and changing selection no longer create undo entries. Both are view state, so I believe that is the correct behaviour, but it is an app-wide change to undo semantics rather than something scoped to this feature.
There were three divergent copies of isEditableTarget, all deciding purely from event.target. That is not safe for a destructive binding: pressing Delete while typing in a popover field deleted the selected layer, because the popup keeps focus on its trigger so event.target was a button, not the input. Replaces all three with one shared guard that also consults document.activeElement and bails inside any [role=dialog]/menu/listbox. The event target and the active element can legitimately disagree when an overlay manages focus, and the destructive shortcuts are exactly the ones that must not guess.
NumberInput applies `cn(className)` with no chrome of its own, so a call site that forgets className renders a borderless, transparent field that reads as static text. Every existing call site happened to style itself, so the trap was invisible until a new one hit it. Exports the canonical style rather than baking in a default, which would change the appearance of existing call sites.
Transport gets a music-note button. With no track loaded it opens the file picker directly; once loaded it opens a popover with live band meters, offset and a collapsed advanced editor for per-band frequency range, gain, attack and release. A reactive dot beside the icon pulses with the overall level. The filename is deliberately not shown — it made the button wide enough to collide with the duration readout. Every animatable parameter row gains a meter button beside the existing keyframe rhombus, in renderFieldLabel so one insertion covers every control type plus the General opacity/hue/saturation sliders. It only appears once a track is loaded, keeping rows uncluttered for anyone not using audio. Live values are published as CSS custom properties by a single reference-counted rAF driver, never React state. Meters and fills read var(--audio-band-bass) directly, so nothing re-renders at 60fps and audio-driven values cannot fight the pointer. MeasuringLayoutProvider stops the sidebar's invisible height-measuring mirrors from mounting real popovers. The sidebar renders its content several times to measure height, which meant ~100 popover roots and store subscriptions once every row gained a button; mirrors now render a same-sized placeholder, with the visibility guards shared so measured heights stay identical. Number fields inside the popovers focus explicitly on pointer-down: base-ui suppresses the popup's default pointer behaviour, which also suppresses native focus-on-click, so without this a keystroke went to the editor's global shortcuts instead of the field. Audio monitoring is on by default and always slaved to timeline.currentTime, re-seeking only past 150ms of drift. The timeline is the clock; letting audio lead would make playback position unreproducible for the exporter.
The editor only renders under WebGPU, so UI regressions here cannot be caught by unit tests and the chrome-devtools tooling could not launch a WebGPU-capable browser. This drives Chrome Canary directly. It writes screenshots as well as assertions, deliberately: Playwright's fill() will type into a field a human can neither see nor focus, so an assertion-only run reported success against a UI whose inputs were invisible. Both real bugs it found — the unstyled range fields and the 4x popover duplication in the measuring mirrors — were visible in screenshots and DOM counts, not in assertions. Drop this and the playwright devDependency if you would rather not carry it.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
EQ-style display in the audio popover: log frequency across X, decibels up Y, with the Bass/Mid/High ranges shaded behind the curve so the frequency boundaries are visible instead of existing only as Hz number inputs. Uses the real spectrum rather than a decorative fake, because it costs the same. The FFT already ran at load time and its 64 log-spaced magnitudes per frame are sitting in the cached spectrogram, so per-frame work either way is ~64 multiply-adds plus one path build. Keeping it real also means the drawn boundaries stay meaningful if they become draggable later. Three pieces of cosmetic shaping get it to look like a hardware analyser: Absolute dB scaling with a fixed floor, not the self-calibrating per-band normalization used for modulation — reusing that would make the curve rescale itself as the track's dynamics change. A tilt that lifts the high end, since real music rolls off steeply enough that an untilted spectrum is nearly empty above ~5kHz. Catmull-Rom interpolation, because 64 bins drawn as a polyline looks visibly stepped at panel width. Per-bin temporal smoothing (fast rise, slow fall) lives in the driver. The spectrogram holds raw magnitudes — only the four band envelopes are smoothed — so drawn straight it flickers hard frame to frame. The path is written straight into the element's `d` attribute from the existing rAF loop. The four band scalars fit in CSS custom properties, but ~64 numbers and a fresh path do not, and routing this through React state would re-render the whole properties sidebar 60 times a second. The band meters are kept below for now so the two can be compared before removing them.
Linking a parameter to audio previously left no trace in the timeline, so it was not obvious which parameters were being driven. Linked parameters now get a lane automatically, with the band's envelope drawn faintly behind it. The lane is not backed by a real TimelineTrack. Keyframe-less tracks cannot survive: evaluate.ts returns null for them, and removeKeyframe and removeSelectedKeyframes filter them out, so an empty track created here would be silently deleted by ordinary keyframe editing. The lane filter simply widens to include rows that have a link, and the keyframe rendering is guarded on a track actually being present. The backdrop is static — the visible window of the envelope is known up front, so the path is built once per (band, offset, duration) and only the playhead moves over it. It reads the window the timeline actually shows rather than the whole song, which for a long track would compress into an unreadable smear. Also trims two costs in the per-frame driver, unrelated to the lane: band values are quantised to two decimals and skipped when unchanged, since setting a custom property on the document root invalidates style document-wide and a held or silent passage should not be writing at all; and the driver is now held only while a track is loaded rather than for the whole lifetime of the transport control.
useAudioMonitor ran an unconditional requestAnimationFrame loop for the whole lifetime of the timeline overlay — including for users who never loaded any audio. Each frame it did three store reads plus a linear asset lookup, purely to notice transport changes and correct drift. Neither needs 60Hz. Transport changes are events, and drift accumulates slowly enough that ~4Hz is ample. Now: a timeline subscription that early-outs on two boolean comparisons in the common case, an audio subscription for source changes, and a 250ms interval for drift. This removes an entire always-on animation-frame loop competing with the renderer's own.
…SS variables The band driver published live values as custom properties on document.documentElement. Setting a custom property there invalidates style for the whole document, so this was forcing a document-wide style recalculation on every animation frame for the entire time a track was loaded. Now the ~10 elements that actually display a band register with the driver and get written directly, so the per-frame cost is proportional to what is on screen rather than to the size of the document. useBandValueElement returns a callback ref rather than an object ref, which matters here: several of these elements live inside popover content that mounts long after the component holding the hook, and an effect keyed on mount alone ran while the ref was still null and silently never registered. Changing the band changes the callback's identity, so React reattaches and re-registers for free. Also splits the band picker's level bars into their own component for the same lifecycle reason, and keeps the driver's skip-unchanged guard so a held or silent passage stops writing entirely.
…acks An audio-linked parameter was hard to find in the timeline panel. The left rail's only cue for "this is animated" is a brighter label plus the enable toggle, and both were gated on a real keyframe track existing — so a parameter driven by audio looked identical to one that was not animated at all. Audio links now get the same treatment: brighter label, dimming when disabled, and an eye toggle that flips the link's enabled flag (both, when a parameter has keyframes and a link). They also get a small purple meter glyph beside the label. The point is to make them first-class *and* still distinguishable — a lane driven by the music behaves differently from a keyframed one, so collapsing them into the same appearance would trade one confusion for another.
Records the export-throughput findings, the audio-driven duration design, and what the referenced Premiere clone actually does (server-side ffmpeg plus a real-time MediaRecorder fallback, no WebCodecs despite the filename).
Export was not GPU-bound. `waitForRenderedFrame` was awaited once per exported frame and was two nested requestAnimationFrame calls, so throughput was pinned to the display refresh with the GPU idle for the wait. It also did not do what it claimed: `renderer.render()` is a synchronous submit with no fence, so waiting on rAF observed canvas presentation, not GPU completion. Replaces it with an explicit GPU sync. `EditorRenderer` gains `waitForGpuIdle()`, implemented over `backend.device.queue.onSubmittedWorkDone()`, returning false when the device is unreachable so the old rAF wait remains as a fallback rather than a hard dependency. Four supporting fixes, all per frame: The live preview loop kept rendering throughout an export — a second WebGPURenderer running the whole pass chain, competing for the GPU and queueing ahead of the very rAF callbacks the export waited on. A ref-counted lock now parks it, released in the export's finally so a cancelled export cannot leave the editor frozen. The parked path still advances lastFrameTime, otherwise the first frame after an export would see a delta of the entire export duration and jump the playhead. `renderer.resize()` ran every frame with identical dimensions. three has no early-out, so `updateSize()` was destroying and recreating the GPUCanvasContext configuration on every frame. Now guarded. The throwaway first render per frame existed only to populate PipelineManager.passes, which is filled by syncLayers inside render(), so that prepareForExportFrame had something to iterate. That is only true for the first frame, so it is now opt-in via bootstrapPasses and frames 2..N render once. This also fixes a silent bug: PipelineManager.render early-returns when not dirty, so for effects-only comps the encoded pixels came from the first pass, which was built with a hardcoded delta of 0, discarding the real per-frame delta. onProgress fired per frame straight into a React setState, so a long export meant one synchronous dialog re-render per frame inside the loop. Throttled to 10Hz, with the final frame always reported. Measured over 240 frames at draft quality, same machine, via a scripted export: old vsync wait, visible 16.84 ms/frame gpu sync, visible 11.06 ms/frame gpu sync, tab hidden 11.23 ms/frame Output is byte-identical between the old and new wait paths in matched conditions, so determinism is preserved. Two pre-existing quirks were confirmed to be unrelated to this change, since the old path reproduces them exactly: a hidden tab yields different bytes from a visible one, and the first export after a cold load can differ from later ones. Both point at the prewarm resource-poll race rather than the frame loop.
Picking an audio file now sets the timeline duration to the track length once, then leaves it alone, so the duration stays freely editable in both the timeline panel and the export panel. Raises the cap from 120s to 1800s and moves the bounds into timeline-duration.ts so the store, the ruler and the duration input all read the same numbers instead of the input hardcoding max=120. Fixes three duration bugs found on the way: - setDuration pinned every keyframe past the new end onto the new end, so shrinking the timeline collapsed them onto identical times with no way back. Keyframes now stay where they are, matching replaceState. - getLongestVideoLayerDuration was unclamped while setDuration was, so a video longer than the cap put the ruler and the store on different values and the sync effect re-dispatched forever. - The ruler generated a tick every 20s at any length, so a 4 minute composition rendered 360 tick nodes. Tick steps now scale, labels read m:ss past a minute, and the endpoint tick is dropped when it would collide with the last regular one.
The history debounce is 250ms and analysing a 4 minute track takes longer, so setting the duration after analysis produced two undo entries: one for the audio source, one for the duration. A single Cmd+Z then reverted the duration while leaving the audio loaded. The asset already carries a duration from its media element by the time loadAsset resolves, so use that and fall back to the decoded sample count only when the element could not report one. Both changes now land inside one debounce window, and the ruler extends immediately instead of after the FFT finishes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Drive shader parameters from music. Load a track, click the meter icon on any numeric parameter, pick a band, set an output range.
Prompted by a musician using Shader Lab for music videos who wants to use it for projected stage backgrounds.
Design
Audio is a modulation source, not a keyframe track. Keyframes are deterministic curves over time; audio is a signal. Modelling it as a per-parameter track would fight the actual goal — one source feeding many effects across many layers.
So: one project audio source → four bands (Bass/Mid/High/Level) → each band linkable to any number of parameters on any layers, with a per-link output range.
value = lerp(outMin, outMax, band), clamped to the parameter's declared bounds.Envelopes are precomputed, not sampled from an AnalyserNode. This is forced, not preferred:
export.tsis an offline frame-stepped loop that renders the same timestamp up to three times, so band values must be a pure function of time. The payoff is that preview and export agree exactly, scrubbing works, and the whole envelope is known up front.Two-stage analysis so band edits stay instant — stage A (decode + STFT, 294 ms for a 130 s track) runs once per source in a Worker; stage B (normalize + smooth, ~8 ms) re-runs on every band-config change.
Normalization is self-calibrating per band. A fixed dB window did not survive real material: measured against a loudness-compressed master, a 48 dB window left every band clustered between 0.70 and 0.95, which reads as visually static. Mapping each band's own 5th–99th percentile onto
[0,1]took the usable span from 0.32 to ~0.70.No new
AnimatedPropertyBindingkind — the binding says what is driven, the band says what drives it. That keeps the existing non-exhaustivebinding.kindbranches correct, and is whypackages/shader-lab-reactneeds no mirrored type changes.Two fixes that are not about audio
Both are pre-existing bugs found while debugging this, and are separate commits — worth reviewing on their own:
fix(history)—getHistorySnapshotSignatureincludedtimeline.currentTime, which changes every frame during playback, so every frame re-armed the 220 ms commit debounce and it never fired. Any edit made while the timeline was playing was never committed to history and was discarded by the next undo. This would have eaten keyframe and parameter work too, not just audio. It changes undo semantics app-wide: moving the playhead and changing selection no longer create undo entries. I think that is correct — both are view state — but it is broader than this feature.fix(shortcuts)— three divergent copies ofisEditableTarget, all deciding purely fromevent.target. Pressing Delete while typing in a popover field deleted the selected layer, because the popup keeps focus on its trigger. Now one shared guard that also consultsdocument.activeElementand bails inside any dialog/menu/listbox.Verification
268 tests pass, typecheck clean, lint unchanged at 2 pre-existing warnings, production build succeeds.
83 tests cover the analysis core with no browser: band isolation on synthetic PCM, temporal discrimination, quiet-track usability (a 60 Hz sine at amplitude 0.008 still drives bass past 0.9), silence, attack/release curves, and the
[0,1]invariant across six signal types. Plus an aliasing guard asserting modulation never returns or mutates a tuple owned by the layer store — a missed clone there corrupts theparamsCloneCacheWeakMap in a way that persists across frames.Driven end to end against a real 130 s track through
buildRendererFrame: 119/120 distinct values over a 4 s window, always within declared bounds, byte-identical at the same timestamp,layer.paramsuntouched.Known gaps
MAX_DURATION), so a full song cannot be covered yet. Deferred deliberately;offsetSecondsselects a section meanwhile.decode.tsalready handles mp4/webm via the same path..labwith audio links renders static values in the published runtime with no warning.