Skip to content

Commit 050b7bd

Browse files
committed
feat(aws-lambda): support plan protocol v2
1 parent 25d95a8 commit 050b7bd

44 files changed

Lines changed: 4432 additions & 101 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.fallowrc.jsonc

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -397,6 +397,12 @@
397397
// require intrusive middleware changes beyond this PR's scope.
398398
"minLines": 6,
399399
"ignore": [
400+
// AWS Lambda and GCP Cloud Run deliberately mirror the same distributed
401+
// rendering lifecycle while retaining provider-specific SDK, storage, and
402+
// retry semantics. The Plan v2 AWS adapter extends that existing symmetry;
403+
// extracting a shared cloud abstraction would couple independent packages.
404+
"packages/aws-lambda/src/handler.ts",
405+
"packages/aws-lambda/src/s3Transport.ts",
400406
// sourcePatcher.ts: pre-existing internal clones between the inline-style
401407
// and attribute tag-patchers; only the PatchOperation type gained two
402408
// optional fields here, but the line shift makes fallow re-flag them.

examples/aws-lambda/README.md

Lines changed: 26 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,7 @@ aws stepfunctions start-execution \
7575
"ProjectS3Uri": "s3://${RENDER_BUCKET}/projects/my-project.tar.gz",
7676
"PlanOutputS3Prefix": "s3://${RENDER_BUCKET}/renders/$(date +%s)/",
7777
"OutputS3Uri": "s3://${RENDER_BUCKET}/output.mp4",
78+
"PlanProtocol": "v1",
7879
"Config": {
7980
"fps": 30,
8081
"width": 1920,
@@ -91,6 +92,9 @@ EOF
9192

9293
The Step Functions execution kicks off Plan, fans out RenderChunk via
9394
the Map state, and finally Assemble. Final mp4 lands at `OutputS3Uri`.
95+
`PlanProtocol` may be `"v1"` or `"v2"`; absent defaults to v1. V2 uses
96+
separate manifest and content-addressed artifact locators throughout the
97+
workflow and never places a v2 object in `PlanS3Uri`.
9498

9599
## Local invocation
96100

@@ -119,13 +123,14 @@ the architecture works on a deployed Lambda — use the local smoke
119123
script:
120124

121125
```bash
122-
# All defaults (mp4-h264-sdr fixture, chunk counts 2/4/8, PSNR >= 40 dB).
126+
# Defaults use the fixture's meta.json minPsnr (30 dB for mp4-h264-sdr).
123127
./scripts/smoke.sh
124128

125129
# Customised:
126130
./scripts/smoke.sh \
127131
--fixture mp4-h264-sdr \
128132
--chunk-counts 2,4,8,16 \
133+
--plan-protocol both \
129134
--psnr-threshold 40 \
130135
--reserved-concurrency 8
131136

@@ -141,7 +146,21 @@ per-run stack name, renders the fixture at each chunk count via the
141146
Step Functions state machine, PSNR-compares against the in-process
142147
baseline (which is git-LFS tracked under
143148
`packages/producer/tests/distributed/<fixture>/output/`), captures
144-
per-execution Step Functions history, and tears the stack down.
149+
per-execution Step Functions history, and tears the stack down. Use
150+
`--plan-protocol both` to run v1 and v2 through the same deployed Lambda
151+
package and baseline. Each v1/v2 pair is also gated directly on per-chunk
152+
hashes from Step Functions history, normalized decoded RGBA frame hashes,
153+
decoded 48 kHz stereo s16le PCM hashes and byte counts, normalized stream
154+
metadata, and duration. Encoded MP4 SHA equality is reported but is
155+
informational unless `--require-encoded-sha-equal` is set. The script
156+
assigns unique function/state-machine names, uses a
157+
dedicated temporary SAM artifact bucket, and removes render objects,
158+
retained buckets, the implicit Lambda log group, and deployment artifacts
159+
on teardown. Suspended-version buckets are purged in 1,000-entry batches,
160+
including concrete versions, null versions, and delete markers. It then
161+
verifies that the stack, both buckets, Lambda, state-machine, and both log
162+
groups are absent; an otherwise-successful run fails if cleanup cannot be
163+
proven.
145164

146165
**Wall-clock methodology caveat (`eval.sh` only).** `eval.sh` reports a
147166
local-vs-Lambda "speedup" column. The local timing includes `bun` +
@@ -162,9 +181,11 @@ spend is roughly $0.10-$0.20 per pass before S3 transfer. Lower
162181

163182
Outputs land under `<repo-root>/lambda-smoke-artifacts/`:
164183

165-
- `results.json``chunkCount × wallClockMs × psnrAvgDb`
166-
- `renders/N<N>-output.mp4` — each rendered chunk count
167-
- `renders/N<N>-history.json` — full Step Functions execution history
184+
- `results.json``planProtocol × chunkCount × wallClockMs × psnrAvgDb`
185+
- `semantic-comparisons.json` — direct v1/v2 semantic gate results
186+
- `renders/<protocol>-N<N>-output.mp4` — each rendered variant
187+
- `renders/<protocol>-N<N>-history.json` — full Step Functions execution history
188+
- `renders/v1-v2-N<N>.*` — normalized frame hashes, ffprobe metadata, and comparison JSON
168189

169190
Prerequisites: `aws` (v2), `sam` (≥ 1.100), `bun` (≥ 1.3), `ffmpeg`,
170191
`jq`, `zip`. AWS credentials come from the standard resolution chain
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
{
2+
"Action": "assemble",
3+
"PlanProtocol": "v2",
4+
"PlanV2ManifestS3Uri": "s3://example-bucket/renders/sample/v2/manifest.json",
5+
"PlanV2ArtifactS3Prefix": "s3://example-bucket/renders/sample/v2/artifacts/sha256",
6+
"PlanHash": "0000000000000000000000000000000000000000000000000000000000000000",
7+
"ChunkS3Uris": ["s3://example-bucket/renders/sample/chunks/0000.mp4"],
8+
"AudioS3Uri": null,
9+
"OutputS3Uri": "s3://example-bucket/renders/sample/output.mp4",
10+
"Format": "mp4"
11+
}
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
{
2+
"Action": "plan",
3+
"PlanProtocol": "v2",
4+
"ProjectS3Uri": "s3://example-bucket/projects/sample.tar.gz",
5+
"PlanOutputS3Prefix": "s3://example-bucket/renders/sample/",
6+
"Config": {
7+
"fps": 30,
8+
"width": 1920,
9+
"height": 1080,
10+
"format": "mp4",
11+
"chunkSize": 240,
12+
"maxParallelChunks": 8,
13+
"runtimeCap": "lambda"
14+
}
15+
}
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{
2+
"Action": "renderChunk",
3+
"PlanProtocol": "v2",
4+
"PlanV2ManifestS3Uri": "s3://example-bucket/renders/sample/v2/manifest.json",
5+
"PlanV2ArtifactS3Prefix": "s3://example-bucket/renders/sample/v2/artifacts/sha256",
6+
"PlanHash": "0000000000000000000000000000000000000000000000000000000000000000",
7+
"ChunkIndex": 0,
8+
"ChunkOutputS3Prefix": "s3://example-bucket/renders/sample/",
9+
"Format": "mp4"
10+
}
Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
#!/usr/bin/env bash
2+
# AWS resource-name isolation and failed-deploy discovery helpers.
3+
4+
hf_derive_project_name() {
5+
local stack_name="$1" prefix digest
6+
prefix=$(printf '%s' "$stack_name" |
7+
tr -c '[:alnum:]-' '-' |
8+
sed -E 's/^-+//; s/-+$//' |
9+
cut -c1-36)
10+
[ -n "$prefix" ] || prefix="hf-smoke"
11+
digest=$(printf '%s' "$stack_name" | sha256sum | awk '{print substr($1,1,12)}')
12+
printf '%s-%s\n' "$prefix" "$digest"
13+
}
14+
15+
hf_known_absent() {
16+
local pattern="$1" output_file="$2"
17+
grep -Eiq "$pattern" "$output_file"
18+
}
19+
20+
hf_assert_command_absent() {
21+
local label="$1" absent_pattern="$2"
22+
shift 2
23+
local output_file status detail
24+
output_file=$(mktemp)
25+
if "$@" >"$output_file" 2>&1; then
26+
echo "ERROR: destructive-isolation collision: $label already exists" >&2
27+
rm -f "$output_file"
28+
return 1
29+
else
30+
status=$?
31+
fi
32+
if ! hf_known_absent "$absent_pattern" "$output_file"; then
33+
detail=$(tr '\n' ' ' < "$output_file" | cut -c1-240)
34+
echo "ERROR: could not prove $label absent (exit=$status): $detail" >&2
35+
rm -f "$output_file"
36+
return 2
37+
fi
38+
rm -f "$output_file"
39+
}
40+
41+
hf_assert_named_list_absent() {
42+
local label="$1"
43+
shift
44+
local output_file output status detail
45+
output_file=$(mktemp)
46+
if output=$("$@" 2>"$output_file"); then
47+
if [ -n "$output" ]; then
48+
echo "ERROR: destructive-isolation collision: $label already exists ($output)" >&2
49+
rm -f "$output_file"
50+
return 1
51+
fi
52+
else
53+
status=$?
54+
detail=$(tr '\n' ' ' < "$output_file" | cut -c1-240)
55+
echo "ERROR: could not verify $label absence (exit=$status): $detail" >&2
56+
rm -f "$output_file"
57+
return 2
58+
fi
59+
rm -f "$output_file"
60+
}
61+
62+
# Fail closed unless every exact name this smoke run can destructively clean
63+
# is absent. Call before arming cleanup or creating any AWS resource.
64+
hf_assert_deploy_isolation() {
65+
local stack_name="$1" project_name="$2"
66+
local function_name="${project_name}-render"
67+
local lambda_log="/aws/lambda/${function_name}"
68+
local states_log="/aws/states/${function_name}"
69+
70+
hf_assert_command_absent "CloudFormation stack $stack_name" "does not exist" \
71+
aws cloudformation describe-stacks --stack-name "$stack_name" &&
72+
hf_assert_command_absent "Lambda function $function_name" \
73+
"ResourceNotFoundException|Function not found" \
74+
aws lambda get-function --function-name "$function_name" &&
75+
hf_assert_named_list_absent "Step Functions state machine $function_name" \
76+
aws stepfunctions list-state-machines \
77+
--query "stateMachines[?name=='$function_name'].stateMachineArn" --output text &&
78+
hf_assert_named_list_absent "log group $lambda_log" \
79+
aws logs describe-log-groups --log-group-name-prefix "$lambda_log" \
80+
--query "logGroups[?logGroupName=='$lambda_log'].logGroupName" --output text &&
81+
hf_assert_named_list_absent "log group $states_log" \
82+
aws logs describe-log-groups --log-group-name-prefix "$states_log" \
83+
--query "logGroups[?logGroupName=='$states_log'].logGroupName" --output text
84+
}
85+
86+
# Return a JSON object with any physical resources CloudFormation managed to
87+
# create, even when stack outputs were never populated. A genuinely absent
88+
# stack is an empty result; auth/network/query failures are errors.
89+
hf_discover_stack_resources() {
90+
local stack_name="$1" output_file error_file status detail
91+
output_file=$(mktemp)
92+
error_file=$(mktemp)
93+
if aws cloudformation list-stack-resources \
94+
--stack-name "$stack_name" --output json >"$output_file" 2>"$error_file"; then
95+
jq '{
96+
renderBucket: (
97+
[.StackResourceSummaries[]?
98+
| select(.LogicalResourceId == "RenderBucket")
99+
| .PhysicalResourceId][0] // ""
100+
),
101+
stateMachineArn: (
102+
[.StackResourceSummaries[]?
103+
| select(.LogicalResourceId == "RenderStateMachine")
104+
| .PhysicalResourceId][0] // ""
105+
)
106+
}' "$output_file"
107+
rm -f "$output_file" "$error_file"
108+
return
109+
else
110+
status=$?
111+
fi
112+
if hf_known_absent "does not exist" "$error_file"; then
113+
printf '{"renderBucket":"","stateMachineArn":""}\n'
114+
rm -f "$output_file" "$error_file"
115+
return
116+
fi
117+
detail=$(tr '\n' ' ' < "$error_file" | cut -c1-240)
118+
echo "ERROR: failed to discover physical stack resources (exit=$status): $detail" >&2
119+
rm -f "$output_file" "$error_file"
120+
return 2
121+
}
Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
#!/usr/bin/env bash
2+
# S3 bucket cleanup helpers. Sourcing this file has no side effects.
3+
4+
# Delete every concrete object version and delete marker from a bucket.
5+
#
6+
# We intentionally re-list the first 1,000 entries after every delete batch
7+
# instead of advancing markers through a mutating result set. This handles
8+
# arbitrary pagination depth while avoiding skipped keys when the page being
9+
# used as a cursor has just been removed. It is also required for buckets with
10+
# versioning Suspended: `aws s3 rm` only creates null-version delete markers
11+
# and leaves the historical/null versions behind.
12+
hf_purge_s3_bucket_versions() {
13+
local bucket="$1" work page delete_request delete_response count errors rounds=0
14+
work=$(mktemp -d)
15+
page="$work/page.json"
16+
delete_request="$work/delete.json"
17+
delete_response="$work/delete-response.json"
18+
19+
while true; do
20+
rounds=$((rounds + 1))
21+
if [ "$rounds" -gt 100000 ]; then
22+
echo "ERROR: S3 purge exceeded 100000 batches for s3://$bucket" >&2
23+
rm -rf "$work"
24+
return 1
25+
fi
26+
27+
if ! aws s3api list-object-versions \
28+
--bucket "$bucket" \
29+
--max-keys 1000 \
30+
--no-paginate \
31+
--output json > "$page"; then
32+
echo "ERROR: failed to list object versions for s3://$bucket" >&2
33+
rm -rf "$work"
34+
return 1
35+
fi
36+
37+
jq '{
38+
Objects: [
39+
(.Versions // [])[],
40+
(.DeleteMarkers // [])[]
41+
] | map({Key, VersionId}),
42+
Quiet: true
43+
}' "$page" > "$delete_request"
44+
count=$(jq '.Objects | length' "$delete_request")
45+
if [ "$count" -eq 0 ]; then
46+
break
47+
fi
48+
49+
if ! aws s3api delete-objects \
50+
--bucket "$bucket" \
51+
--delete "file://$delete_request" \
52+
--output json > "$delete_response"; then
53+
echo "ERROR: failed to delete a version batch from s3://$bucket" >&2
54+
rm -rf "$work"
55+
return 1
56+
fi
57+
# Successful Quiet=true deletes may produce a zero-byte response body.
58+
# Slurp mode treats that as an empty input set and therefore zero errors,
59+
# while still counting per-object Errors when AWS returns a JSON object.
60+
errors=$(jq -s '[.[] | (.Errors // [])[]] | length' "$delete_response")
61+
if [ "$errors" -ne 0 ]; then
62+
echo "ERROR: S3 returned per-object deletion errors for s3://$bucket:" >&2
63+
jq -c '.Errors[]' "$delete_response" >&2
64+
rm -rf "$work"
65+
return 1
66+
fi
67+
echo " purged $count object versions/delete markers from s3://$bucket"
68+
done
69+
70+
rm -rf "$work"
71+
}
72+
73+
hf_delete_s3_bucket_completely() {
74+
local bucket="$1"
75+
hf_purge_s3_bucket_versions "$bucket" &&
76+
aws s3api delete-bucket --bucket "$bucket"
77+
}

0 commit comments

Comments
 (0)