feat(media): add sharded media object-key paths - #4533
Open
bradseiler wants to merge 16 commits into
Open
Conversation
This was referenced Aug 3, 2026
bradseiler
changed the title
bradseiler
marked this pull request as ready for review
August 7, 2026 04:55
bradseiler
force-pushed
the
seiler/media-layout-migration
branch
from
August 7, 2026 05:13
ada0dd3 to
828b51e
Compare
Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Classify sharded payload keys in storage sweeps, preserve physical totals, and deduplicate logical legacy/sharded copies. Export read resolution, fallback, and duplicate-layout metrics. Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Keep self-hosted upgrades on legacy writes by default, expose the write-layout gate through Helm and Compose, and document the explicit legacy-to-dual-to-sharded migration sequence. Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Consolidate read and write policy behind one upgrade-safe phase and ship guarded maintenance binaries for backfill and legacy cleanup. Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz> Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Signed-off-by: Brad Seiler <seiler@squareup.com>
Signed-off-by: Brad Seiler <seiler@squareup.com>
Replace the Option + expect("checked above") pattern in both upload
short-circuit paths with a filtered if-let binding, so the presence of
the blob key is guaranteed structurally instead of by a comment on a
non-local invariant. Also collapse the longhand match on
existing_write_key into `?`. No behavior change.
Signed-off-by: Brad Seiler <seiler@squareup.com>
Document that returning to legacy-only after accepting uploads in sharded-only makes sharded-layout objects unreadable until the phase is raised again. Ship sharded-only in the Compose example environment so fresh stacks never need a backfill, and correct the chart README to reflect that only the chart default remains legacy-only for upgrade safety. Signed-off-by: Brad Seiler <seiler@squareup.com>
Co-authored-by: Brad Seiler <seiler@squareup.com> Signed-off-by: Brad Seiler <seiler@squareup.com>
Co-authored-by: Brad Seiler <seiler@squareup.com> Signed-off-by: Brad Seiler <seiler@squareup.com>
Co-authored-by: Brad Seiler <seiler@squareup.com> Signed-off-by: Brad Seiler <seiler@squareup.com>
Co-authored-by: Brad Seiler <seiler@squareup.com> Signed-off-by: Brad Seiler <seiler@squareup.com>
bradseiler
force-pushed
the
seiler/media-layout-migration
branch
from
August 10, 2026 20:17
5fc442c to
1cfde28
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Media payload keys are currently flat, so high-volume traffic lacks hash-leading S3 prefix entropy. Existing deployments also need a safe, explicit rollout that preserves old reads and rollback behavior without changing storage behavior merely by upgrading.
What
media/<sha[0:2]>/<sha[2:4]>/<community>/<filename>while leaving Git/CAS namespaces unchangedBUZZ_MEDIA_MIGRATION_PHASEwith three complete media object-key path policies:legacy-only(default): read and write only legacy paths, preserving existing deployment behaviordual-read-and-write: prefer sharded reads with legacy fallback and write both pathssharded-only: read and write only sharded pathslegacy-only→dual-read-and-write→ backfill →sharded-onlysharded-onlybefore their first upload, avoiding any future backfill or legacy cleanupbuzz-media-layout-backfillandbuzz-media-layout-delete-legacyin both relay image targets, with bounded paging, checkpoints, request-rate limiting, idempotency, destination verification, dry-run, and destructive confirmationValidation
cargo test -p buzz-media --all-targets: 119 passed; live-MinIO test ignoredcargo test -p buzz-relay --lib config::tests::: 32 passedcargo test -p buzz-relay --lib storage_sweep: 15 passedcargo clippy -p buzz-media -p buzz-relay --all-targets -- -D warnings: cleancargo fmt --all -- --check, desktop Tauri fmt, andgit diff --check: cleanorigin/main: Rust, desktop, Tauri, mobile, and typecheck suites passedGenerated with Goose