Add content_set_hash: order-independent lookup so content can be found from its bytes without a ctid

[t-bhendarkar]

Content can only be looked up by exact hash if you already know the media order, so there is no way to answer "is this registered?" from the bytes alone.

The problem

content_hash folds in media_canonical_hash, which concatenates media_ids in registered order:

media_canonical_hash = shake256(media.map(m => m.media_id).join(""))
content_hash         = shake256(media_canonical_hash + textHash)

That is correct for verification — a verifier reads the order from the chain and re-derives the hash — and it is deliberate: the author's intended media order ends up inside what they signed, so it is tamper-evident.

It is useless for search. Given only the files and the text, you would have to try all n! orderings to find the matching content_hash. And every content route is ctid-keyed, so there is no lookup endpoint at all today:

GET  /content/:ctid
GET  /content/:ctid/similar
POST /content/:ctid/verify
...

There is a second, sharper consequence. 001_content_hash_index's own comment says the exact-hash index exists to catch "identical content by another author". For multi-media that only fires if the second registrant happens to submit the media in the same order. Reorder the images and content_hash differs, so step 0a of _findNearDuplicates silently misses the duplicate.

Proposed fix: a second, order-independent key

Add a node-local lookup column. Do not change content_hash.

content_set_hash = shake256( "tip:content-set:v1" ‖ sorted(media_ids) ‖ textHash )
                   -- newline delimited; NULL when the content has no media

Two keys answering two different questions:

question	key
is this byte-for-byte the artifact the author signed, in their order?	content_hash (signed, frozen)
is this content registered, in any media order?	content_set_hash (unsigned, derived, rebuildable)

Why it is safe to add

• No signed bytes change. Purely derived and stored; every existing signature still verifies. No new tx_type needed.
• Invisible to consensus. Leave it out of _canonContent and state_merkle_root is unchanged, so no fork and no activation gate. _canonContent is an explicit field whitelist (verified — it names every field, no spread), so a new column is not picked up automatically.
• Rebuildable. It is a derived index, same category as verify_count / dispute_count, which already sit outside the root.

Design decisions and the evidence behind them

Text is folded in, and we accept that it cannot be backfilled.

The raw text is not stored anywhere — only content_hash — and you cannot invert content_hash to recover textHash. So a text-inclusive key can only ever be computed at registration time, going forward. A media-only variant would be backfillable, but it is a coarser key (it also matches same-images-different-caption) and needs a two-step lookup: find candidates, then re-derive content_hash per candidate using that row's own stored media order.

Mainnet data made the choice clear:

233 content rows
 49 with media
  3 with 2 or more media   ->  2 items, 2 items, 6 items

The un-backfilled blind spot is 3 rows, it never grows (every future registration is covered), and even those three are brute-forceable at 2, 2 and 720 permutations. Trading that for a precise single-query lookup is clearly the better side.

This is a one-way door: choosing text-inclusive forecloses backfill permanently.

Separator and domain prefix, unlike media_canonical_hash.

mch joins with no separator, which is injective only because every media_id is exactly 64 hex — an invariant now enforced on both the API path (content-register.js:192) and the consensus path (verifyTx, #259). It cannot gain a separator now without breaking every existing signature.

This column is new and not frozen, so it should not borrow that invariant. The \n delimiter keeps it injective regardless of id format, and the tip:content-set:v1 prefix stops a single-item set hashing to the same digest as that item's mch.

Scope

• migration: nullable content_set_hash column on content + index
• compute in content-service.register
• wire through MemoryStore / SQLiteStore / KnexAdapter (the adapter must mirror the store primitives)
• POST /v1/content/lookup taking { media_ids[], content } — the ctid-free check
• third probe in _findNearDuplicates, closing the reorder-dedup miss described above
• tests, including a freeze test for the new recipe alongside content-hash-recipe.test.js

Explicitly not doing

• Sorting media_canonical_hash. docs/SIGNING_VERSIONING.md line 65 lists "reorder" as breaking, requiring a new tx_type. It would also give up the property that the author's intended order is signed.
• Adding a separator to mch. Same breaking change, and unnecessary given the enforced 64-hex invariant.

Related

• fix(content): validate media_id shape in verifyTx and freeze the content_hash recipe #259 — enforced the 64-hex media_id invariant in verifyTx, and froze the content_hash recipe
• docs/SIGNING_VERSIONING.md — why the existing recipe cannot change in place

Separate follow-up worth its own issue: fingerprint observability — count media components arriving with no fingerprint, record which client sent the envelope, and persist rejected components instead of dropping them silently. Same additive, node-local shape.