i18n(privacy): propagate self-hosted image URLs into all 18 translations (+ fix a latent 54-file deletion) - #1910
Conversation
…names
Three source pages were renamed for capitalization on main:
Accept_Payments_AS_A_Merchant.md -> Accept_Payments_As_A_Merchant.md
Journalist_privacy_setup.md -> Journalist_Privacy_Setup.md
Private_Community_treasury.md -> Private_Community_Treasury.md
The source moved, but `translation/curated-pages.txt`, the 54 translated
files (3 pages x 18 locales) and their 54 manifest entries all kept the old
names. On a case-sensitive filesystem that leaves `curated-pages.txt`
pointing at three paths that do not exist, so the `curated subset of site`
invariant is broken, and the staleness detector reports the three pages as
ORPHANS in every locale.
That misclassification is destructive, not cosmetic: the sync agent's orphan
path (`processOrphan`) rmSync's the translation file and drops the manifest
entry, and the de-curation stage then removes the page from the curated list
once no locale references it. An unscoped sync run would therefore have
deleted 54 good translations and silently cut coverage from 181 to 178 pages,
while the new capitalized names never entered curation at all.
This renames the 54 translated files to match, re-keys the 54 manifest
entries (values untouched, so provenance is preserved), and fixes the three
curated-pages.txt lines.
Verified after the change:
- translation/check-invariants.mjs: "Manifest invariants hold: 181 curated
pages, 18 locales"
- detect-staleness.mjs: orphan=0 in all 18 locales (was 3 each)
- the 3 pages now register as ordinary stale entries; their English diff
since the manifest was seeded is image-URL-only (10 URLs each, from the
self-hosted-images migration), so no re-translation is required
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The self-hosting migration (3ec0585, PR ZecHub#1888) rewrote image references in `site/**` from third-party hosts to `/content-images/...`, but never touched `translations/**`. Because the frontend prefers a curated translation over the English source and performs no runtime image rewriting, every non-English reader was still hotlinking the original hosts — i.e. the privacy leak the migration set out to close remained live across the majority of the site's content surface. Before this change, translated files still referenced: i.ibb.co 540 files raw.githubusercontent.com 288 files pbs.twimg.com 144 files hackmd.io 72 files img.youtube.com 72 files github.com/ZecHub/zechub/assets 54 files Mechanical URL substitution only — no re-translation, no model involved. The old->new pairs are taken from the migration commit itself (1276 pairs across 250 pages), so the mapping is authoritative rather than reconstructed. Applied: 9458 URL replacements across 1908 files. The diff is +10205/-10205 — byte-for-byte symmetric, no lines added or removed in any file. Alt text, captions and link text are untouched (verified across ar/zh/ru/sw and the NLLB/GT locales). Each file had to pass four post-conditions or it was left alone: 1. no mapped old URL remains 2. image-reference count unchanged 3. line count unchanged 4. every differing line is reproducible by applying only the mapped substitutions to the original line Zero gate failures occurred. Deliberately NOT propagated: the 74 pairs whose migrated value is `/content-images/_unavailable.svg`, plus any target with no corresponding file in the frontend's `public/content-images/`. For those the migration lost the image, so English is privacy-clean but broken while the translation still holds a working URL. Copying the placeholder would degrade 18 locales to match a regression. Those 19 pages keep their working URLs and stay flagged stale, so they remain visible until the English side is fixed upstream. Residual after this change: 3 pages x 18 locales (Creators_and_Tips, Visualizing_the_Zcash_Network, ZK_Shielded_Asset_Platforms) still reference pbs.twimg.com / raw.githubusercontent.com. Two of the three are at parity with English, which still hotlinks the same asset; all three trace back to `_unavailable.svg` losses and are fixed by re-migrating those images upstream. Verified: translation/check-invariants.mjs -> "Manifest invariants hold: 181 curated pages, 18 locales". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
With the self-hosted image URLs now propagated into the translated trees, the 94 curated pages whose English changed ONLY by image URL are genuinely current again. This re-stamps their manifest provenance (`src` = normalized hash of the current English, `src_commit` = fd4274d) for all 18 locales: 1692 entries. Without this the detector keeps reporting those pages stale and the sync agent would spend LLM calls re-translating prose that never changed — ~1700 work items of pure waste. Hashes are computed with the repo's own translation/lib/normalize-hash.mjs, so they match what the detector and the seeder produce. Four conditions had to hold before any entry was stamped: A. no old (pre-migration) URL from that page's mapping survives in the translation B. every /content-images/... URL in the translation is one the English page also references (no stray or invented targets) C. the entry is not flagged `edited` (never overwrite a hand-edited page) D. the page's English change since the seed was image-URL-only Zero gate failures. Condition D specifically excluded four pages that were both image-churned AND prose-edited (Wallets, Pepper_Sync, Zcash_Shielded_Assets, Raspberry_Pi_4_Full_Node) — stamping those would have marked genuinely stale prose as fresh and hidden it from the detector permanently. Detector, per locale, before -> after this branch: orphan 3 -> 0 (the case-rename fix) stale 108 -> 17 (111 after renames reclassified, then -94 stamped) The remaining 17 are exactly the work that still needs doing: 6 pages with genuine prose changes -> normal LLM sync 11 pages held because the migration lost their image (`_unavailable.svg`); their translations keep working URLs and stay visible until the English side is repaired upstream Verified: check-invariants.mjs -> "Manifest invariants hold: 181 curated pages, 18 locales". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
9e07db3 to
e2cdc1e
Compare
|
Force-pushed to correct a defect I introduced in the first push: the rewrite script read and wrote in Python text mode, which silently converted 10 translated files from CRLF to LF. Those files' entire contents showed as changed even though only their image URLs differed. Fixed by redoing those 10 substitutions in binary mode from the pre-change bytes. CRLF file count is back to 28, matching the original tree, and The privacy commit's diffstat drops from +10205/-10205 to +9369/-9369 across the same 1908 files. That is now consistent with the 9458 URL replacements (a few lines carry more than one image URL), where the earlier figure was inflated by ~750 line-ending-only changes. Re-verified after the rebase: Worth noting for anyone writing similar tooling against this repo: 28 files under |
The self-hosted-image migration (#1888, commit
3ec05852) rewrote image references insite/**from third-party hosts to/content-images/…, but never touchedtranslations/**. Since the frontend prefers a curated translation over the English source and does no runtime image rewriting, every non-English reader was still hotlinking the original hosts — so the privacy leak the migration set out to close stayed live across the majority of the site's content surface.This also fixes a latent data-loss bug in the curated/translation bookkeeping (commit 1), which would have deleted 54 good translations on the next sync run.
1 —
f2215e8crestore curated/translation parity after the case-only renamesThree source pages were renamed for capitalization on
main:Accept_Payments_AS_A_Merchant.mdAccept_Payments_As_A_Merchant.mdJournalist_privacy_setup.mdJournalist_Privacy_Setup.mdPrivate_Community_treasury.mdPrivate_Community_Treasury.mdThe source moved, but
translation/curated-pages.txt, the 54 translated files (3 pages × 18 locales) and their 54 manifest entries all kept the old names. On a case-sensitive filesystem that leavescurated-pages.txtpointing at three paths that do not exist, breaking thecurated ⊆ siteinvariant, and the staleness detector reports all three as orphans in every locale.That misclassification is destructive rather than cosmetic. The sync agent's orphan path
rmSyncs the translation file and drops the manifest entry, and the de-curation stage then removes the page from the curated list once no locale references it. An unscoped sync run would have deleted 54 good translations and silently cut coverage from 181 to 178 pages, while the new capitalized names never entered curation at all.Renames the 54 translated files to match, re-keys the 54 manifest entries (values untouched, provenance preserved), fixes the three
curated-pages.txtlines.2 —
9ca89c97propagate self-hosted image URLs into all 18 translationsBefore this change, translated files still referenced:
i.ibb.coraw.githubusercontent.compbs.twimg.comhackmd.ioimg.youtube.comgithub.com/ZecHub/zechub/assetsMechanical URL substitution only — no re-translation, no model involved. The old→new pairs are taken from the migration commit itself (1276 pairs across 250 pages), so the mapping is authoritative rather than reconstructed.
Applied: 9458 URL replacements across 1908 files. The diff is +10205/−10205 — byte-for-byte symmetric, no line added or removed in any file. Alt text, captions and link text are untouched (spot-verified across
ar,zh,ru,swand the NLLB/GT locales).Each file had to pass four post-conditions or it was left alone:
Zero gate failures occurred.
Deliberately not propagated: the 74 pairs whose migrated value is
/content-images/_unavailable.svg, plus any target with no corresponding file in the frontend'spublic/content-images/. In those cases the migration could not fetch the image, so the English side now shows a placeholder. Rather than copy a placeholder into 18 locales, those pairs were skipped and the affected pages stay flagged stale, so the decision stays visible instead of being silently stamped fresh.Note that most of these placeholders reflect a pre-existing loss rather than anything the migration broke: 30 of the 74 replaced relative paths from a Notion export whose asset directory was never committed to this repo, so those images were already 404ing before any of this work. The remaining 44 were hotlinks that failed to fetch. Repairing them is separate work — see the companion PR.
3 —
9e07db3ere-stamp provenance for the image-only pagesThe 94 curated pages whose English changed only by image URL are genuinely current again, so their manifest provenance is re-stamped for all 18 locales (1692 entries). Without this the detector keeps reporting them stale and the sync agent would spend ~1700 work items of LLM calls re-translating prose that never changed.
Hashes come from the repo's own
translation/lib/normalize-hash.mjs, so they match what the detector and seeder produce.Four conditions had to hold before any entry was stamped:
/content-images/…URL in the translation is one the English page also references (no stray targets)edited(never overwrite a hand-edited page)Zero gate failures. Condition D specifically excluded four pages that were both image-churned and prose-edited (
Wallets,Pepper_Sync,Zcash_Shielded_Assets,Raspberry_Pi_4_Full_Node) — stamping those would have marked genuinely stale prose as fresh and hidden it from the detector permanently.Net effect
Detector, per locale:
orphanstaleThe remaining 17 are exactly the work that still needs doing — 6 pages with genuine prose changes (normal sync), and 11 pages held pending the upstream image repair.
translation/check-invariants.mjs→Manifest invariants hold: 181 curated pages, 18 locales.Reviewer note
translation/sync-state.jsonshows ~3384 insertions/deletions because the file was re-serialised with sorted keys after the rename re-keying. The entry count is unchanged (3258) and no data was lost — only key order and the re-stampedsrc/src_commitvalues differ.Provenance
The analysis behind this was adversarially reviewed by two independent models before any file was touched; both confirmed the 102-image-only/6-prose split under stricter methods than the original, and both rejected an earlier plan of mine to strip images from the canonical hash (it would have made all future image-only drift invisible). The 54-translation deletion trap in commit 1 was surfaced by that review, not by the original analysis.
🤖 Generated with Claude Code