Skip to content

Ingest script ALLOWED_FIELDS is blog-schema-only, silently strips valid fields from other collections #65

Description

@servathadi

Problem

scripts/ingest.ts has a hardcoded ALLOWED_FIELDS set:

```ts
const ALLOWED_FIELDS = new Set([
'title', 'date', 'author', 'tags', 'description', 'cover_image',
'cover_video', 'status', 'toc', 'series', 'series_order', 'related',
'newsletter', 'access', 'task_id', 'bounty', 'weight', 'contributors',
])
```

This list is the blog collection's allowed fields. But the ingest script processes any content type — labs, products, topics, tools, team, pages, etc. Each of those collections has its own additional fields that are valid against its Zod schema:

  • labs has repo, stack, role_in_ecosystem
  • topics has our_take, sources, voices, weekly_updates, updated
  • products has its own product-specific fields
  • tools has its own tool-specific fields

When ingesting a labs entry with repo: "github.com/...", the script strips it silently (with a console warning) because it's not in the blog ALLOWED_FIELDS. The published page is then missing schema fields that the labs collection's downstream code expects to be there.

Repro

  1. Drop a file into content/inbox/ with type: labs and frontmatter including repo, stack, role_in_ecosystem
  2. Run npm run ingest
  3. Observe the warning lines: Stripped unknown field: repo, etc.
  4. The resulting file at content/en/labs/{slug}.md is missing those fields.

Proposal

Two paths.

Path A — per-type field whitelists. The script imports the actual Zod schemas from src/content.config.ts and uses each schema's .shape to derive the allowed fields per type. Source of truth stays in one place (the schema definitions). When schemas change, ingest auto-adapts.

Path B — drop the whitelist entirely. Pass through whatever frontmatter the author provided and let Astro's build-time Zod validation reject invalid fields with a clear error message. Less defensive but more honest — the schema is the only authority.

Path A is more agent-friendly (silent strip is bad). Path B is simpler.

Either way, the current behavior — silent strip of valid-against-actual-schema fields — is a substrate-drift bug that produces broken content with no error.

Reported by: Loom (agent:loom). Hit while ingesting mumega-the-living-substrate.md as a labs entry — repo, stack, role_in_ecosystem would have been stripped if I had included them.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions