Skip to content

Commit 7585f79

Browse files
authored
feat(producer): add services/distributed/plan.ts (#808)
## What Phase 3 of the distributed rendering plan: the first half of the public distributed primitives. Adds `plan(projectDir, config, planDir)` and its supporting types as a new module at `packages/producer/src/services/distributed/plan.ts`. See `DISTRIBUTED-RENDERING-PLAN.md` §11 Phase 3. ## Why Phase 1 extracted the in-process renderer's six pipeline phases into individually-callable stage functions; Phase 2 added the determinism-hardening utilities and flags those stages needed. This PR is the first caller that flips those flags `true` — composing the stages into Activity A of the three-activity distributed pipeline (`plan` → `renderChunk` × N → `assemble`). Output is a self-contained `<planDir>/` with the documented §4.1 layout plus a content-addressed `planHash` (§4.2). Adapter authors (Temporal, AWS Lambda + Step Functions, etc.) consume the directory + hash; the OSS library never touches transport. ## How `plan()` composes (in order): 1. `validateNoGpuEncode` — typed `PlanValidationError` if GPU encode / hardware GL slipped through caller-supplied config. 2. `runCompileStage` — threaded through `failClosedFontFetch: true` so font-fetch failures throw `FontFetchError` instead of silently falling back to system fonts. Required a new optional `failClosedFontFetch` field on `CompileStageInput` and a new `options` argument on `compileForRender(projectDir, htmlPath, downloadDir, options)`. Both default to behavior-preserving values for the in-process renderer. 3. `validateNoSystemFonts(compiled.html)` — runs against the post-compile HTML so we catch system primary fonts on the same surface chunk workers will render. 4. `runProbeStage` — near-zero when `staticDuration > 0`; spins Chrome only when the composition genuinely needs runtime probing. 5. `runExtractVideosStage` with `materializeSymlinks: true` so per-video frame sequences live as real files inside the planDir (symlinks don't survive S3 / GCS round-trips). 6. `runAudioStage` — produces `<planDir>/audio.aac` if the composition has audio. 7. Materialize the `<planDir>/{compiled,video-frames,audio.aac,meta}/...` layout from the staged work tree. 8. `freezePlan` — writes `meta/{composition,encoder,chunks}.json` + `plan.json`, then computes `planHash` from the actual on-disk bytes (so consumers can re-validate a plan by hashing). `freezePlan` was previously a typed skeleton with `throw new Error("not implemented")`; this PR implements its body, including a `stripUndefined` helper because `LockedRenderConfig` has optional fields (`crf`, `bitrate`) and `canonicalJsonStringify` deliberately throws on `undefined`. Chunking (§6) lives in `resolveChunkPlan(totalFrames, chunkSize, maxParallelChunks)` + `buildChunkSlices(...)` — exported from `plan.ts` so PR 3.2 (renderChunk) and adapter code can import them directly. ### What did NOT change `executeRenderJob`, the `hyperframes render` CLI, the producer HTTP `/render` routes, and every existing stage signature are untouched. The Phase 2 flags continue to default to `false`/`undefined` for in-process callers; only `plan()` flips them. PSNR baselines for the regression harness should be unchanged. ## Test plan - [x] Unit tests added — `packages/producer/src/services/distributed/plan.test.ts`. 10 cases covering: chunking math (`resolveChunkPlan` defaults / cap-clamp / invalid input), slice construction (`buildChunkSlices`), golden planDir layout against a tiny fixture, and `planHash` determinism across two `plan()` invocations on the same inputs. - [x] `bun test packages/producer/src/services/distributed/` — 10 pass. - [x] `bun test packages/producer/src/` — 312 pass, 1 fail. The one failure is `writeCompiledArtifacts — external assets on Windows drive-letter paths (GH #321) > rejects a maliciously crafted key that tries to escape compileDir`, which also fails on a clean checkout of `origin/main` with no working-tree changes (pre-existing flake, not introduced by this PR). - [x] `bun run --filter @hyperframes/producer typecheck` — clean. - [x] `bun run --filter @hyperframes/producer build` — clean. - [x] `bunx oxlint` + `bunx oxfmt --check` — clean on changed files. - [ ] Producer Docker regression harness — pending CI run. `executeRenderJob` is unchanged here, so PSNR baselines should hold; the new code path is reachable only through the not-yet-exported `plan()`. This is PR 1 of a 6-PR Phase 3 stack: - **3.1 (this PR)** — `services/distributed/plan.ts` - 3.2 — `services/distributed/renderChunk.ts` - 3.3 — `services/distributed/assemble.ts` - 3.4 — `planDir` size cap (`PLAN_TOO_LARGE`) - 3.5 — distributed format banlist (webm + HDR mp4) - 3.6 — public exports + `@hyperframes/producer/distributed` subpath 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2 parents a99db41 + eff4cf6 commit 7585f79

5 files changed

Lines changed: 1066 additions & 25 deletions

File tree

Lines changed: 211 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,211 @@
1+
/**
2+
* Unit tests for `services/distributed/plan.ts`.
3+
*
4+
* Covers:
5+
* - Golden planDir layout produced from a tiny fixture (no browser probe
6+
* required — the fixture declares `data-duration` so the probe stage
7+
* short-circuits).
8+
* - planHash determinism across two `plan()` calls on the same inputs.
9+
* - Chunking math (`resolveChunkPlan`, `buildChunkSlices`).
10+
*
11+
* The "no browser probe" path is deliberate: spinning Chrome inside `bun test`
12+
* is expensive and flaky. The chunking helpers + planDir layout are tested
13+
* with synchronous compile-only fixtures; the BeginFrame / probe path lives
14+
* inside the regression harness (`bun run --cwd packages/producer docker:test`).
15+
*/
16+
17+
import { afterAll, beforeAll, describe, expect, it } from "bun:test";
18+
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
19+
import { tmpdir } from "node:os";
20+
import { join } from "node:path";
21+
import {
22+
buildChunkSlices,
23+
DEFAULT_CHUNK_SIZE,
24+
DEFAULT_MAX_PARALLEL_CHUNKS,
25+
plan,
26+
resolveChunkPlan,
27+
} from "./plan.js";
28+
29+
// Composition the tests render. `data-duration="1"` keeps the probe stage's
30+
// `needsBrowser` gate `false` so plan() completes without launching Chrome.
31+
const FIXTURE_HTML = `<!doctype html>
32+
<html>
33+
<head><meta charset="utf-8"><title>plan-test fixture</title></head>
34+
<body>
35+
<div data-composition-id="root" data-width="320" data-height="240" data-duration="1">
36+
<p>plan-test fixture</p>
37+
</div>
38+
</body>
39+
</html>`;
40+
41+
let projectDir: string;
42+
let runRoot: string;
43+
44+
beforeAll(() => {
45+
runRoot = mkdtempSync(join(tmpdir(), "hf-plan-test-"));
46+
projectDir = join(runRoot, "project");
47+
mkdirSync(projectDir, { recursive: true });
48+
writeFileSync(join(projectDir, "index.html"), FIXTURE_HTML, "utf-8");
49+
});
50+
51+
afterAll(() => {
52+
rmSync(runRoot, { recursive: true, force: true });
53+
});
54+
55+
describe("resolveChunkPlan", () => {
56+
it("returns 1 chunk when totalFrames fits in configChunkSize", () => {
57+
const result = resolveChunkPlan(60, 240, 16);
58+
expect(result.chunkCount).toBe(1);
59+
expect(result.effectiveChunkSize).toBeGreaterThanOrEqual(60);
60+
});
61+
62+
it("caps chunkCount at maxParallelChunks for very long renders", () => {
63+
// 54000 frames / 240 = 225 naive chunks → must cap at 16.
64+
const result = resolveChunkPlan(54000, 240, 16);
65+
expect(result.chunkCount).toBe(16);
66+
// 54000 / 16 = 3375 → chunkSize at least that big so the union covers
67+
// all frames in 16 slices.
68+
expect(result.effectiveChunkSize).toBeGreaterThanOrEqual(Math.ceil(54000 / 16));
69+
});
70+
71+
it("naive count drives chunkCount when below cap", () => {
72+
// 600 frames / 240 = 3 naive chunks; well below the 16 cap.
73+
const result = resolveChunkPlan(600, 240, 16);
74+
expect(result.chunkCount).toBe(3);
75+
expect(result.effectiveChunkSize).toBe(240);
76+
});
77+
78+
it("rejects non-positive totalFrames", () => {
79+
expect(() => resolveChunkPlan(0, 240, 16)).toThrow();
80+
expect(() => resolveChunkPlan(-1, 240, 16)).toThrow();
81+
expect(() => resolveChunkPlan(Number.NaN, 240, 16)).toThrow();
82+
});
83+
84+
it("rejects non-positive configChunkSize / maxParallelChunks", () => {
85+
expect(() => resolveChunkPlan(60, 0, 16)).toThrow();
86+
expect(() => resolveChunkPlan(60, 240, 0)).toThrow();
87+
});
88+
89+
it("rejects non-integer inputs (would produce fractional endFrames)", () => {
90+
expect(() => resolveChunkPlan(10.5, 240, 16)).toThrow(/positive integer/);
91+
expect(() => resolveChunkPlan(60, 240.5, 16)).toThrow(/positive integer/);
92+
expect(() => resolveChunkPlan(60, 240, 16.5)).toThrow(/positive integer/);
93+
expect(() => resolveChunkPlan(60, 240, Number.POSITIVE_INFINITY)).toThrow(/positive integer/);
94+
});
95+
});
96+
97+
describe("buildChunkSlices", () => {
98+
it("produces consecutive non-overlapping ranges covering all frames", () => {
99+
const slices = buildChunkSlices(700, 3, 240);
100+
expect(slices).toHaveLength(3);
101+
expect(slices[0]).toEqual({ index: 0, startFrame: 0, endFrame: 240 });
102+
expect(slices[1]).toEqual({ index: 1, startFrame: 240, endFrame: 480 });
103+
// Last chunk absorbs the remainder so endFrame === totalFrames exactly.
104+
expect(slices[2]).toEqual({ index: 2, startFrame: 480, endFrame: 700 });
105+
});
106+
107+
it("handles a single-chunk render", () => {
108+
const slices = buildChunkSlices(50, 1, 240);
109+
expect(slices).toHaveLength(1);
110+
expect(slices[0]).toEqual({ index: 0, startFrame: 0, endFrame: 50 });
111+
});
112+
});
113+
114+
describe("plan() defaults", () => {
115+
it("exports the documented chunking defaults", () => {
116+
expect(DEFAULT_CHUNK_SIZE).toBe(240);
117+
expect(DEFAULT_MAX_PARALLEL_CHUNKS).toBe(16);
118+
});
119+
});
120+
121+
describe("plan() — golden planDir + planHash determinism", () => {
122+
// Each `plan()` call is reasonably expensive (compile pass parses + inlines
123+
// the HTML), so we run it once for the layout assertions and once more for
124+
// the determinism assertion. The 30s timeout absorbs cold-start font /
125+
// runtime resolution variance on the CI host.
126+
const TIMEOUT_MS = 30_000;
127+
128+
it(
129+
"produces the documented planDir layout",
130+
async () => {
131+
const planDir = join(runRoot, "plan-layout");
132+
mkdirSync(planDir, { recursive: true });
133+
const result = await plan(
134+
projectDir,
135+
{ fps: 30, width: 320, height: 240, format: "mp4" },
136+
planDir,
137+
);
138+
139+
// planDir directory layout
140+
expect(existsSync(join(planDir, "plan.json"))).toBe(true);
141+
expect(existsSync(join(planDir, "compiled", "index.html"))).toBe(true);
142+
expect(existsSync(join(planDir, "video-frames"))).toBe(true);
143+
// No audio in the fixture — audio.aac must NOT exist.
144+
expect(existsSync(join(planDir, "audio.aac"))).toBe(false);
145+
expect(existsSync(join(planDir, "meta", "composition.json"))).toBe(true);
146+
expect(existsSync(join(planDir, "meta", "encoder.json"))).toBe(true);
147+
expect(existsSync(join(planDir, "meta", "chunks.json"))).toBe(true);
148+
// The temporary work tree must be cleaned up.
149+
expect(existsSync(join(planDir, ".plan-work"))).toBe(false);
150+
151+
// ── PlanResult contract ─────────────────────────────────────────────
152+
expect(result.planDir).toBe(planDir);
153+
expect(result.planHash).toMatch(/^[0-9a-f]{64}$/);
154+
expect(result.chunkCount).toBeGreaterThanOrEqual(1);
155+
expect(result.totalFrames).toBe(30); // 1s @ 30fps
156+
expect(result.width).toBe(320);
157+
expect(result.height).toBe(240);
158+
expect(result.format).toBe("mp4");
159+
expect(result.ffmpegVersion).toMatch(/ffmpeg/i);
160+
expect(result.producerVersion).toMatch(/^\d+\.\d+\.\d+/);
161+
162+
// ── chunks.json shape ───────────────────────────────────────────────
163+
const chunks = JSON.parse(
164+
readFileSync(join(planDir, "meta", "chunks.json"), "utf-8"),
165+
) as Array<{ index: number; startFrame: number; endFrame: number }>;
166+
expect(chunks).toHaveLength(result.chunkCount);
167+
// Slices must cover [0, totalFrames) with no gaps.
168+
let cursor = 0;
169+
for (const chunk of chunks) {
170+
expect(chunk.startFrame).toBe(cursor);
171+
cursor = chunk.endFrame;
172+
}
173+
expect(cursor).toBe(result.totalFrames);
174+
175+
// ── plan.json shape ─────────────────────────────────────────────────
176+
const planJson = JSON.parse(readFileSync(join(planDir, "plan.json"), "utf-8")) as Record<
177+
string,
178+
unknown
179+
>;
180+
expect(planJson.planHash).toBe(result.planHash);
181+
expect(planJson.hasAudio).toBe(false);
182+
expect(planJson.totalFrames).toBe(result.totalFrames);
183+
},
184+
TIMEOUT_MS,
185+
);
186+
187+
it(
188+
"produces a byte-identical planHash on a second invocation",
189+
async () => {
190+
const planDirA = join(runRoot, "plan-determinism-a");
191+
const planDirB = join(runRoot, "plan-determinism-b");
192+
mkdirSync(planDirA, { recursive: true });
193+
mkdirSync(planDirB, { recursive: true });
194+
195+
const config = { fps: 30 as const, width: 320, height: 240, format: "mp4" as const };
196+
const a = await plan(projectDir, config, planDirA);
197+
const b = await plan(projectDir, config, planDirB);
198+
199+
expect(a.planHash).toBe(b.planHash);
200+
expect(a.chunkCount).toBe(b.chunkCount);
201+
expect(a.totalFrames).toBe(b.totalFrames);
202+
203+
// Encoder JSON must be byte-identical — its bytes feed planHash, so any
204+
// drift here would silently change the hash framing.
205+
const encoderA = readFileSync(join(planDirA, "meta", "encoder.json"));
206+
const encoderB = readFileSync(join(planDirB, "meta", "encoder.json"));
207+
expect(encoderA.equals(encoderB)).toBe(true);
208+
},
209+
TIMEOUT_MS,
210+
);
211+
});

0 commit comments

Comments
 (0)