Skip to content

Commit bf90695

Browse files
committed
Widen uncovered-release candidate set to rotation ∪ recently played (BS#1877 amendment)
The search handoff offered only active rotation (~300 releases), but most of what DJs actually play — and what listeners see — is not current rotation. The candidate set is now active rotation ∪ recently played linked albums, keeping the wire schema, exact-match ceiling, and ADR 0013 architecture unchanged. - New plays.ts: fetchRecentPlays(lookbackDays) windows flowsheet (entry_type='track', album_id linked, canonical pair via the artists join); fetchAllPlayedAlbums() sources the album_plays MV for the one-time --backfill drain (37,421 albums, play-count desc, deliberately no SQL LIMIT — a LIMIT would silently stall the drain against the handed-off anti-join). - orchestrate.ts: single mode-blind fetchPlayCandidates injection; rotation-first concat + first-wins dedup; one cap (UNCOVERED_MAX_RELEASES_PER_RUN) post-anti-join and pre-DRY_RUN, capped_out computed at the cap site; the capped list feeds render/write/publish/recordHandoffs alike; rotation-lane guards stay hard throws in steady state and demote to log + Sentry captureError under --backfill; zero-plays escalates without throwing; locked DRY_RUN report widened with backfill, recent_play_rows, candidate_rows, capped_out. - Dockerfile: CMD → ENTRYPOINT + empty CMD so docker-level --backfill passes through; this deliberately relocates the default OUTPUT_PATH to the WORKDIR (documented). - Backfill pacing: single-path whole-file publish + publish-once markers means at most one --backfill invocation per consumer cycle until research-data#16 walks the snapshot branch history; requirement relayed on that ticket (WXYC/research-data#16 (comment)). - Docs: README rewrite (modes, cap semantics, pacing, guards, locked report), ADR 0013 line-57 sizing (credential-conditional), env-vars, ops-cron-scheduling, package description, workspace CLAUDE.md row, stale header sweeps. - Tests: unit coverage for both arms, options parsing, cap/dedup/guard behavior (report matched on its unique line prefix, not a shared token); integration cases for both arms with sql.unsafe MV refresh and fixture-scoped scoped* helpers. Steady-state cost after PUBLISH + RESEARCH_DATA_WRITE_TOKEN are provisioned: ~75 new albums/week ≈ 325 searches/month. Measured: play aggregate ~174 ms unindexed; anti-join 13–25 ms at the full 37,421-id cardinality. Code-review fixes folded in: the backfill and pull-from-container recipes now bind-mount the output directory instead of promising a docker cp that --rm makes impossible (with the credential unprovisioned that file is the run's only artifact, so the old recipe silently lost the batch); the empty-snapshot publish skip from the parent commit is re-keyed onto the capped list, which is what actually gets rendered, written, and published; and plays.ts cites the migration that creates flowsheet_track_add_time_idx rather than a schema.ts line range that had already rotted onto an unrelated block comment. Renumbers this PR's references to the marker migration from 0146 to 0156, matching the schema PR beneath it: main landed its own 0146 (library-delete-denylist, BS#2112) while this stack sat open. Touched job.ts's header comment (resolved against this branch's widened prose, which is kept) and the job's CLAUDE.md registration row. The surviving "migration 0146" in jobs/library-etl/README.md is correct as-is — it refers to main's denylist migration, not this one.
1 parent f0b2c30 commit bf90695

15 files changed

Lines changed: 1055 additions & 104 deletions

‎CLAUDE.md‎

Lines changed: 1 addition & 0 deletions
Large diffs are not rendered by default.

‎Dockerfile.uncovered-release-list‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,4 +27,7 @@ RUN npm install --omit=dev
2727
COPY --from=builder ./uncovered-release-list-builder/jobs/uncovered-release-list/dist ./jobs/uncovered-release-list/dist
2828
COPY --from=builder ./uncovered-release-list-builder/shared/database/dist ./shared/database/dist
2929

30-
CMD ["npm", "start", "--workspace=@wxyc/uncovered-release-list"]
30+
# ENTRYPOINT + empty CMD so docker-level args (e.g. `--backfill`, `--dry-run`)
31+
# pass through to the job rather than replacing the launcher.
32+
ENTRYPOINT ["node", "/uncovered-release-list/jobs/uncovered-release-list/dist/job.js"]
33+
CMD []

‎docs/adr/0013-search-augmented-critic-review-discovery.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,7 @@ The cost is real and worth naming: an extra publish/fetch hop (research-data sti
5454

5555
## Search provider evaluation
5656

57-
Evaluated for the workload the ticket sizes: ~72 residual uncovered releases now, plus a few dozen new rotation adds per week — **a few hundred queries/month**, comfortably in every candidate's lowest paid (or free) tier.
57+
Evaluated for the workload the ticket sizes, **updated for the candidate-set widening above and conditional on `jobs/uncovered-release-list`'s publish credential (`RESEARCH_DATA_WRITE_TOKEN` + `PUBLISH=true`) being provisioned** — see that job's README "Precondition" section: once markers are actually being written, steady state is the new-album rate over the widened rotation ∪ recently-played set, measured at ~75 new linked albums/week against the local prod clone (2026-04-23 PT) — **~325 searches/month**, comfortably in every candidate's lowest paid (or free) tier. Until the credential is provisioned, markers are never written and every weekly run re-offers the full ~2,356-album trailing-30-day eligible set instead, pinning the cap (`UNCOVERED_MAX_RELEASES_PER_RUN`, default 400) at its ceiling every run. Separately, a one-time historical backlog of **37,421** distinct linked albums ever played exists; draining it (`--backfill`, play-count-desc, operator-paced, bounded by the same cap) is deliberately out of scope for this design and would cost roughly $90 of Brave queries alone if fully drained.
5858

5959
| Provider | Status (2026-07) | Cost at our volume | ToS posture | Verdict |
6060
| -------------------------------------------------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@@ -69,6 +69,8 @@ This spike used the WebSearch/WebFetch tools available in this environment as a
6969

7070
## Uncovered-release list handoff
7171

72+
> **Update (candidate-set widening):** the candidate set feeding this handoff was widened from active-rotation-only to **active rotation ∪ recently-played linked albums** — WXYC is a freeform station, and most of what a DJ actually enters into the flowsheet is not a current rotation release, so a rotation-only candidate set structurally misses the releases most likely to reach a listener's feed. The handoff mechanism below (committed file, dedicated marker table) is unchanged; only what feeds the anti-join changed. A `--backfill` mode drains the historical play tail via the `album_plays` materialized view, separately from the steady-state trailing window. See `jobs/uncovered-release-list/README.md` for the two arms, the run-mode table, and the publish-credential precondition the widened cost model depends on (also reflected in the sizing below).
73+
7274
**A committed file, refreshed by a scheduled Backend-Service job, not a live read endpoint or direct DB access.** Three options were on the table:
7375

7476
- **Committed file (chosen).** A Backend-Service job computes `rotation × album_critic_reviews` anti-joined against already-searched releases (a new small tracking table or a `source_key` convention analogous to the ETL's `manifest:${source}` — TBD in the production ticket) and commits a small JSON/CSV file of `(artist, album, library_id)` rows to research-data (or opens a PR there) on a schedule. research-data's search crawler reads that file, same shape as `crawl_reviews.py` reading its own committed corpus for resumability.

‎docs/env-vars.md‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -372,12 +372,14 @@ The album-critic-reviews ETL (`jobs/album-critic-reviews-etl/`) mirrors the `man
372372
- `ANTHROPIC_API_KEY` — Required for a non-`DRY_RUN` run (Haiku snippet extraction, `claude-haiku-4-5-20251001`). Not required under `DRY_RUN`, which makes zero LLM calls by design (the anti-join + dry-run short-circuit both precede extraction).
373373
- `DRY_RUN` — Locked truthy values: `true`, `1` (case-insensitive). Runs fetch + parse + match + dedup + anti-join and evaluates the run guards, but skips every Haiku call and every UPSERT, emitting a single locked-schema JSON report line on stdout (see `jobs/album-critic-reviews-etl/README.md`). Harmless to forget — the UPSERT is idempotent across reruns, and the anti-join means a repeated real run makes zero LLM calls for already-seeded pairs anyway.
374374

375-
The uncovered-release-list job (`jobs/uncovered-release-list/`) computes the `rotation × album_critic_reviews` anti-join weekly (BS#1877, ADR 0013's "uncovered-release list handoff") and commits `uncovered-releases.jsonl` to the private `WXYC/research-data` repo for its `search` crawl mode to consume. No SSH tunnel, no sync-notify. Unlike `album-critic-reviews-etl`, no external credential is required just to run — the anti-join read and the local snapshot-file write both work with only the standard `DB_*` set; a credential is needed only to actually push the snapshot (see below).
375+
The uncovered-release-list job (`jobs/uncovered-release-list/`) computes the `(active rotation ∪ recently played) × album_critic_reviews` anti-join weekly (BS#1877, ADR 0013's "uncovered-release list handoff", widened by the "rotation ∪ recently played" amendment) and commits `uncovered-releases.jsonl` to the private `WXYC/research-data` repo for its `search` crawl mode to consume. No SSH tunnel, no sync-notify. Unlike `album-critic-reviews-etl`, no external credential is required just to run — the anti-join read and the local snapshot-file write both work with only the standard `DB_*` set; a credential is needed only to actually push the snapshot (see below).
376376

377-
- `OUTPUT_PATH` — Local path the snapshot file is written to (default `./output/uncovered-releases.jsonl`).
377+
- `OUTPUT_PATH` — Local path the snapshot file is written to. Default `./output/uncovered-releases.jsonl`, relative to the job's cwd — under the container's `ENTRYPOINT` (direct `node dist/job.js`, not `npm start --workspace`), cwd is the `/uncovered-release-list` WORKDIR, so the default resolves to `/uncovered-release-list/output/uncovered-releases.jsonl`.
378378
- `PUBLISH` — Locked truthy values: `true`, `1`. Must be set, together with `RESEARCH_DATA_WRITE_TOKEN`, for the job to push to research-data. Off by default; the job still writes the local file and runs its anti-joins with `PUBLISH` unset, it just doesn't call out to GitHub and doesn't write `uncovered_release_search_markers` rows (see the job README's "Handoff" section for why marker-writing is publish-gated).
379-
- `RESEARCH_DATA_WRITE_TOKEN` — Fine-grained PAT scoped to the private `WXYC/research-data` repo with `Contents: Read and write`, used to commit `uncovered-releases.jsonl` to that repo's `uncovered-releases-snapshot` branch via the GitHub Contents API. Deliberately separate from `RESEARCH_DATA_TOKEN` above (that one is read-only by design). Not yet provisioned as of this job's initial ship.
380-
- `DRY_RUN` — Locked truthy values: `true`, `1`. Runs the fetch + resolve + dedup + both anti-joins and evaluates the run guards, but makes zero writes and zero network calls, emitting a single locked-schema JSON report line on stdout (see `jobs/uncovered-release-list/README.md`).
379+
- `RESEARCH_DATA_WRITE_TOKEN` — Fine-grained PAT scoped to the private `WXYC/research-data` repo with `Contents: Read and write`, used to commit `uncovered-releases.jsonl` to that repo's `uncovered-releases-snapshot` branch via the GitHub Contents API. Deliberately separate from `RESEARCH_DATA_TOKEN` above (that one is read-only by design). Not yet provisioned as of this job's initial ship — see the job README's "Precondition" section for why the widened candidate set's cost model depends on this being provisioned.
380+
- `UNCOVERED_PLAY_LOOKBACK_DAYS` — Trailing window (days) for the steady-state play arm (`plays.fetchRecentPlays`). Default `30`. Parsed and validated in both modes (a malformed value fails fast even under `--backfill`); its value is only read in steady state — `--backfill` drains every linked album ever played instead.
381+
- `UNCOVERED_MAX_RELEASES_PER_RUN` — Post-anti-join cap, both modes. Default `400`. The one cap knob at one position; an operator raises it per `--backfill` invocation (e.g. `-e UNCOVERED_MAX_RELEASES_PER_RUN=2000`) since the default would collide with the backfill's one-invocation-per-consumer-cycle pacing contract — see the job README.
382+
- `DRY_RUN` — Locked truthy values: `true`, `1`. Runs the fetch + resolve + dedup + both anti-joins and evaluates the run guards, but makes zero writes — and no network calls beyond a Sentry capture if a guard escalates (the guards carry no `DRY_RUN` exemption) — emitting a single locked-schema JSON report line on stdout (see `jobs/uncovered-release-list/README.md`).
381383

382384
### Flowsheet April gap import (`jobs/flowsheet-april-gap-import`, BS#2119)
383385

‎docs/ops-cron-scheduling.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ BS#2218 also made this the first job to write `cronjob_runs.cursor_position` (mi
5151
- `concerts-artist-resolver` (05:15) — pure-SQL strict/alias resolver, no LML. (`concerts-artist-lml-resolver` at 05:35 is the LML-touching one.)
5252
- `concerts-similar-artists-enrichment` (05:55, hits semantic-index not LML), `venue-events-scraper`, `triangle-shows-etl`, `album-reviews-etl`, `legacy-mirror-reconcile` — non-LML.
5353
- `metadata-no-match-digest` (`07 15 * * *` UTC, daily) — reads `flowsheet`/`shows`/`cronjob_runs` directly and sends via SES; no `@wxyc/lml-client` dependency, cannot trip the breaker. Its `:07` past 15:00 UTC slot was picked only to avoid the `:00` slot shared by the `*/30` ETL trio (now a pair — see above) + hourly `artist-identity-etl`, a host-load courtesy unrelated to this policy.
54-
- `album-critic-reviews-etl` (07:10 Sun) and `uncovered-release-list` (07:40 Sun) — non-LML. The latter reads Backend-Service's own Postgres only (rotation + album_critic_reviews + uncovered_release_search_markers) plus an optional GitHub Contents API push to `WXYC/research-data`; scheduled 30 min after the former so its anti-join sees that week's freshly-pulled `album_critic_reviews` rows.
54+
- `album-critic-reviews-etl` (07:10 Sun) and `uncovered-release-list` (07:40 Sun) — non-LML. The latter reads Backend-Service's own Postgres only (rotation + flowsheet + the `album_plays` MV under `--backfill` + album_critic_reviews + uncovered_release_search_markers) plus an optional GitHub Contents API push to `WXYC/research-data`; scheduled 30 min after the former so its anti-join sees that week's freshly-pulled `album_critic_reviews` rows.
5555

5656
## The hourly safety net (BS#895)
5757

0 commit comments

Comments
 (0)