Stop rank-all from indexing NSFW and VF-dropped posts into SFW retrieval

[Pitchfork-and-Torch]

Problem

Phoenix Rank All is the out-of-network retrieval index for For You (1fav, video, post_creation, topic options, evergreen_video with a 5-year window, and tail). Several independent holes on that write path let adult / NSFW-author / VF-unsafe posts into the SFW indexes.

1. Lightweight tweet rebuild stripped author NSFW bits

getPostMetadataLightweight is what every index event uses. It reconstructed TweetCoreData with user/text/reply/share only — no nsfwUser / nsfwAdmin.

The older helper, getTweetypieResponse, already fetches those federated fields and uses them for hasEligibleMedia. The lightweight path dropped the fetches, so isAuthorEligible always saw the Thrift defaults (false/false).

isAuthorEligible itself then fail-opened when coreData was missing (getOrElse(true)).

2. Processor called the wrong adult check

The processor named the local isAdultPost but called isNsfwPost:

val isAdultPost = eventProcessing.isNsfwPost(unifiedPostAnnotations)

isAdultPost is isNsfw || isAdult. isNsfwPost is isNsfw only. Adult-labeled, non-NSFW posts went into SFW 1fav / video / post_creation / topic indexes. Combined with (1), NSFW accounts posting unlabeled media also went in — especially on PostCreation, before unified annotations exist.

3. Evergreen skipped VF entirely

EvergreenVideo / EvergreenNsfwVideo indexed any immersive video with no VF and no adult/author-NSFW check. evergreen_video is retained for 5 years.

4. VF lookup fail-opened

shouldDropPostByVF treated a thrown VF read or a missing verdict as “do not drop” (getOrElse(false)). A VF outage indexed unsafe posts into every SFW window.

5. Dropped NSFW video was dumped as generic metadata → tail

SidTail only accepts indexName == "metadata" (default when the field is absent). The shouldDropPost branch still called buildMetadataDump with no indexName, so NSFW videos that were correctly kept out of 1fav were written into tail.

6. Missing author became -1

getPostPhoenixRankAllMetadata used authorId.getOrElse(-1). Consumers only rejected author_id == 0, so every authorless row collided on -1 in tail / metadata / topic / main.

Fix

• Fetch nsfwUser / nsfwAdmin in getPostMetadataLightweight and write them onto TweetCoreData (same federated columns as getTweetypieResponse).
• isAuthorEligible fail-closes when coreData is missing.
• Processor drop = VF || isAdultPost || NSFW author. Apply that to SFW evergreen. NSFW evergreen still respects VF (spam / violence / withheld) but allows adult.
• VF errors and missing verdicts fail closed (getOrElse(true)).
• NSFW metadata dumps use indexName = "nsfw_metadata" so SidTail skips them.
• Metadata column returns None instead of authorId = -1.
• Rank-all processors reject post_id <= 0 or author_id <= 0.

Proof

Input	Before	After
NSFW-author post, annotations missing (typical PostCreation)	Indexed into post_creation / video / 1fav	Dropped (isAuthorEligible == false)
isAdult=true, isNsfw=false	Indexed as SFW (isNsfwPost)	Dropped (isAdultPost)
VF service error	Indexed (getOrElse(false))	Dropped
EvergreenVideo + VF-drop / NSFW author	Indexed, 5-year window	Dropped
NSFW video dropped from SFW, metadata dump	SidTail indexes as tail	nsfw_metadata, SidTail skips
Missing author	authorId = -1, stored	Column returns None; processors reject <= 0

Rust processors cover sentinel -1 and nsfw_metadata skip. phoenix-rankall is not in the OSS Cargo workspace, so those tests are for the internal crate.

Scope

One path: rank-all index writes. No RankingScorer, no ads blender, no home-mixer VF leftovers.

[arquieta88-art]

Thanks for working on this. I have one concern regarding the author-level NSFW check.

As I understand the proposed behavior, a post from an account marked nsfwUser / nsfwAdmin can be excluded from SFW retrieval based on the author’s account-level status, even when the individual post itself has no adult/NSFW annotation.

Would it make sense to distinguish between:

• an NSFW-author post that is itself adult/NSFW, and
• a genuinely SFW post published by an NSFW-labeled account?

The second case matters because adult creators can also publish completely non-sensitive posts (text, commentary, everyday photos, SFW video, etc.). Using the account-level flag as a blanket retrieval exclusion could prevent otherwise eligible SFW posts from reaching out-of-network audiences solely because of the author’s account classification.

This is related to the account-level vs. post-level visibility question raised in #79.

Could the author-level flag be used as a safety signal while still allowing a post that is positively classified as SFW to enter appropriate SFW retrieval surfaces?