Nearly 6,000 empty post archives invite assistants to walk them

[wintermeyer]

Almost every member has a post archive that says nothing, and /llms.txt invites assistants to walk all of them. Measured against the dev copy of production: 5,943 of 6,025 members have zero posts, their /<username>/posts answers 200 with an empty state, and no x-robots-tag comes with it. Counting the .md, .txt, .json and .xml siblings that is close to 30,000 addresses worth reading once and never again.

Whoever reads vutuv as a machine notices, and so does anyone whose crawl budget we spend on it. The Media Kit had the identical shape and was fixed a deploy ago: the page keeps answering 200, an empty one carries noindex derived from its own content, and /llms.txt names the sitemap chunks rather than the bare address pattern, so the only addresses advertised are the ones worth fetching. The same three moves fit here, with the archive's own emptiness as the input.

The trap is the one that fix already walked into: whatever the page decides has to reach all five surfaces from one derivation, or the .md sibling and the HTML at the same URL disagree about whether to index it.

Where: VutuvWeb.PostController, VutuvWeb.PageController.llms, Vutuv.Sitemap

An AI agent wrote this text in my name. I know that is problematic.

[wintermeyer]

Built in PR #2206. The numbers first, since they decide the answer.

Measured on the dev copy of production (2026-09-11). 6,025 members, 887 posts and 89 reposts between them: 5,941 archives list nothing for an anonymous reader, 84 list something. Every empty one answered 200 with no x-robots-tag at all, /llms.txt did publish the bare /<username>/posts pattern, and the sitemap never listed an archive URL — so the invitation came from /llms.txt, not from the sitemap.

What I chose: the Media Kit's three moves, with the archive's own input. The page keeps its 200 for the reasons #2143 gives. An empty one carries noindex from one derivation — Vutuv.Posts.public_archive_empty?/2 feeding PostDoc.archive_robots_axes/2 — which reaches the response header, the page's own <meta name="robots"> and all four agent documents. /llms.txt now says most members have written nothing and sends a machine into the sitemap's posts / organization_posts chunks instead of the pattern. noai stays the member's, exactly as next door: emptiness says what is on the page, not who may use it.

Three things the Media Kit did not have.

• Whose emptiness. A post can be visible to some readers and not others, so "empty" depends on who asks. The predicate always asks with viewer = nil, on the owner's own request too, because the header speaks to a crawler and a crawler is never signed in. An archive holding nothing but members-only posts therefore says noindex to its owner as well — verified on a real server: the owner's page draws their post and still carries the header.
• A reshare is content. 2 of the 84 non-empty archives hold no post of the member's own, only reposts. A rule reading "has this member written a post" would have noindexed a page with entries on it, so the predicate is the archive's own timeline query, reposts and remote reposts included.
• Period pages, and .xml. /name/posts/2019 is its own address and can be empty while the archive is not, so the scope is part of the question. The unscoped .xml is a 301 to the member's RSS feed, so the plain path has four surfaces and a period-scoped one five. The RSS feed itself needed nothing: FeedController already sends noindex unconditionally.

Rejected. A 404, for #2143's two reasons plus a third here: an archive's emptiness flips back and forth as posts are written and deleted, and an address that appears and disappears is worse than one that says "nothing here". noai from emptiness, as above. And a new sitemap chunk for archives: the Media Kit had no sitemap presence at all, so #2143 had to create one to make its advice true, while this family already has posts — every member named there has a non-empty archive by construction, so the advice holds without advertising 84 listing pages whose contents are already listed post by post.

Measured after. mix precommit green in 185 s, 11,512 tests. The new file is calibrated: 10 of its 15 tests fail on the un-fixed code, each naming the missing header. The predicate agrees with an independent count (author_post_counts/2) on all 6,025 members in both directions, plans to Limit → Result → Append at 6 buffers and 0.2 ms, and the invariant holds on the real corpus — of the 63 members the sitemap's posts chunks name, 0 have an archive that would answer noindex. Against a real server on port 4039: all four surfaces of an empty archive and all five of an empty year, signed out and as the owner, German and English; noindex gone the moment a post exists and back the moment it is deleted. Smoke-tested with a throwaway member, deleted afterwards (6,025 members and 887 posts, as before).

An AI agent wrote this text in my name. I know that is problematic.