Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CLAUDE.md

Large diffs are not rendered by default.

5 changes: 4 additions & 1 deletion Dockerfile.uncovered-release-list
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,7 @@ RUN npm install --omit=dev
COPY --from=builder ./uncovered-release-list-builder/jobs/uncovered-release-list/dist ./jobs/uncovered-release-list/dist
COPY --from=builder ./uncovered-release-list-builder/shared/database/dist ./shared/database/dist

CMD ["npm", "start", "--workspace=@wxyc/uncovered-release-list"]
# ENTRYPOINT + empty CMD so docker-level args (e.g. `--backfill`, `--dry-run`)
# pass through to the job rather than replacing the launcher.
ENTRYPOINT ["node", "/uncovered-release-list/jobs/uncovered-release-list/dist/job.js"]
CMD []
4 changes: 3 additions & 1 deletion docs/adr/0013-search-augmented-critic-review-discovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ The cost is real and worth naming: an extra publish/fetch hop (research-data sti

## Search provider evaluation

Evaluated for the workload the ticket sizes: ~72 residual uncovered releases now, plus a few dozen new rotation adds per week — **a few hundred queries/month**, comfortably in every candidate's lowest paid (or free) tier.
Evaluated for the workload the ticket sizes, **updated for the candidate-set widening above and conditional on `jobs/uncovered-release-list`'s publish credential (`RESEARCH_DATA_WRITE_TOKEN` + `PUBLISH=true`) being provisioned** — see that job's README "Precondition" section: once markers are actually being written, steady state is the new-album rate over the widened rotation ∪ recently-played set, measured at ~75 new linked albums/week against the local prod clone (2026-04-23 PT) — **~325 searches/month**, comfortably in every candidate's lowest paid (or free) tier. Until the credential is provisioned, markers are never written and every weekly run re-offers the full ~2,356-album trailing-30-day eligible set instead, pinning the cap (`UNCOVERED_MAX_RELEASES_PER_RUN`, default 400) at its ceiling every run. Separately, a one-time historical backlog of **37,421** distinct linked albums ever played exists; draining it (`--backfill`, play-count-desc, operator-paced, bounded by the same cap) is deliberately out of scope for this design and would cost roughly $90 of Brave queries alone if fully drained.

| Provider | Status (2026-07) | Cost at our volume | ToS posture | Verdict |
| -------------------------------------------------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
Expand All @@ -69,6 +69,8 @@ This spike used the WebSearch/WebFetch tools available in this environment as a

## Uncovered-release list handoff

> **Update (candidate-set widening):** the candidate set feeding this handoff was widened from active-rotation-only to **active rotation ∪ recently-played linked albums** — WXYC is a freeform station, and most of what a DJ actually enters into the flowsheet is not a current rotation release, so a rotation-only candidate set structurally misses the releases most likely to reach a listener's feed. The handoff mechanism below (committed file, dedicated marker table) is unchanged; only what feeds the anti-join changed. A `--backfill` mode drains the historical play tail via the `album_plays` materialized view, separately from the steady-state trailing window. See `jobs/uncovered-release-list/README.md` for the two arms, the run-mode table, and the publish-credential precondition the widened cost model depends on (also reflected in the sizing below).

**A committed file, refreshed by a scheduled Backend-Service job, not a live read endpoint or direct DB access.** Three options were on the table:

- **Committed file (chosen).** A Backend-Service job computes `rotation × album_critic_reviews` anti-joined against already-searched releases (a new small tracking table or a `source_key` convention analogous to the ETL's `manifest:${source}` — TBD in the production ticket) and commits a small JSON/CSV file of `(artist, album, library_id)` rows to research-data (or opens a PR there) on a schedule. research-data's search crawler reads that file, same shape as `crawl_reviews.py` reading its own committed corpus for resumability.
Expand Down
10 changes: 6 additions & 4 deletions docs/env-vars.md
Original file line number Diff line number Diff line change
Expand Up @@ -372,12 +372,14 @@ The album-critic-reviews ETL (`jobs/album-critic-reviews-etl/`) mirrors the `man
- `ANTHROPIC_API_KEY` — Required for a non-`DRY_RUN` run (Haiku snippet extraction, `claude-haiku-4-5-20251001`). Not required under `DRY_RUN`, which makes zero LLM calls by design (the anti-join + dry-run short-circuit both precede extraction).
- `DRY_RUN` — Locked truthy values: `true`, `1` (case-insensitive). Runs fetch + parse + match + dedup + anti-join and evaluates the run guards, but skips every Haiku call and every UPSERT, emitting a single locked-schema JSON report line on stdout (see `jobs/album-critic-reviews-etl/README.md`). Harmless to forget — the UPSERT is idempotent across reruns, and the anti-join means a repeated real run makes zero LLM calls for already-seeded pairs anyway.

The uncovered-release-list job (`jobs/uncovered-release-list/`) computes the `rotation × album_critic_reviews` anti-join weekly (BS#1877, ADR 0013's "uncovered-release list handoff") and commits `uncovered-releases.jsonl` to the private `WXYC/research-data` repo for its `search` crawl mode to consume. No SSH tunnel, no sync-notify. Unlike `album-critic-reviews-etl`, no external credential is required just to run — the anti-join read and the local snapshot-file write both work with only the standard `DB_*` set; a credential is needed only to actually push the snapshot (see below).
The uncovered-release-list job (`jobs/uncovered-release-list/`) computes the `(active rotation ∪ recently played) × album_critic_reviews` anti-join weekly (BS#1877, ADR 0013's "uncovered-release list handoff", widened by the "rotation ∪ recently played" amendment) and commits `uncovered-releases.jsonl` to the private `WXYC/research-data` repo for its `search` crawl mode to consume. No SSH tunnel, no sync-notify. Unlike `album-critic-reviews-etl`, no external credential is required just to run — the anti-join read and the local snapshot-file write both work with only the standard `DB_*` set; a credential is needed only to actually push the snapshot (see below).

- `OUTPUT_PATH` — Local path the snapshot file is written to (default `./output/uncovered-releases.jsonl`).
- `OUTPUT_PATH` — Local path the snapshot file is written to. Default `./output/uncovered-releases.jsonl`, relative to the job's cwd — under the container's `ENTRYPOINT` (direct `node dist/job.js`, not `npm start --workspace`), cwd is the `/uncovered-release-list` WORKDIR, so the default resolves to `/uncovered-release-list/output/uncovered-releases.jsonl`.
- `PUBLISH` — Locked truthy values: `true`, `1`. Must be set, together with `RESEARCH_DATA_WRITE_TOKEN`, for the job to push to research-data. Off by default; the job still writes the local file and runs its anti-joins with `PUBLISH` unset, it just doesn't call out to GitHub and doesn't write `uncovered_release_search_markers` rows (see the job README's "Handoff" section for why marker-writing is publish-gated).
- `RESEARCH_DATA_WRITE_TOKEN` — Fine-grained PAT scoped to the private `WXYC/research-data` repo with `Contents: Read and write`, used to commit `uncovered-releases.jsonl` to that repo's `uncovered-releases-snapshot` branch via the GitHub Contents API. Deliberately separate from `RESEARCH_DATA_TOKEN` above (that one is read-only by design). Not yet provisioned as of this job's initial ship.
- `DRY_RUN` — Locked truthy values: `true`, `1`. Runs the fetch + resolve + dedup + both anti-joins and evaluates the run guards, but makes zero writes and zero network calls, emitting a single locked-schema JSON report line on stdout (see `jobs/uncovered-release-list/README.md`).
- `RESEARCH_DATA_WRITE_TOKEN` — Fine-grained PAT scoped to the private `WXYC/research-data` repo with `Contents: Read and write`, used to commit `uncovered-releases.jsonl` to that repo's `uncovered-releases-snapshot` branch via the GitHub Contents API. Deliberately separate from `RESEARCH_DATA_TOKEN` above (that one is read-only by design). Not yet provisioned as of this job's initial ship — see the job README's "Precondition" section for why the widened candidate set's cost model depends on this being provisioned.
- `UNCOVERED_PLAY_LOOKBACK_DAYS` — Trailing window (days) for the steady-state play arm (`plays.fetchRecentPlays`). Default `30`. Parsed and validated in both modes (a malformed value fails fast even under `--backfill`); its value is only read in steady state — `--backfill` drains every linked album ever played instead.
- `UNCOVERED_MAX_RELEASES_PER_RUN` — Post-anti-join cap, both modes. Default `400`. The one cap knob at one position; an operator raises it per `--backfill` invocation (e.g. `-e UNCOVERED_MAX_RELEASES_PER_RUN=2000`) since the default would collide with the backfill's one-invocation-per-consumer-cycle pacing contract — see the job README.
- `DRY_RUN` — Locked truthy values: `true`, `1`. Runs the fetch + resolve + dedup + both anti-joins and evaluates the run guards, but makes zero writes — and no network calls beyond a Sentry capture if a guard escalates (the guards carry no `DRY_RUN` exemption) — emitting a single locked-schema JSON report line on stdout (see `jobs/uncovered-release-list/README.md`).

### Flowsheet April gap import (`jobs/flowsheet-april-gap-import`, BS#2119)

Expand Down
2 changes: 1 addition & 1 deletion docs/ops-cron-scheduling.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ BS#2218 also made this the first job to write `cronjob_runs.cursor_position` (mi
- `concerts-artist-resolver` (05:15) — pure-SQL strict/alias resolver, no LML. (`concerts-artist-lml-resolver` at 05:35 is the LML-touching one.)
- `concerts-similar-artists-enrichment` (05:55, hits semantic-index not LML), `venue-events-scraper`, `triangle-shows-etl`, `album-reviews-etl`, `legacy-mirror-reconcile` — non-LML.
- `metadata-no-match-digest` (`07 15 * * *` UTC, daily) — reads `flowsheet`/`shows`/`cronjob_runs` directly and sends via SES; no `@wxyc/lml-client` dependency, cannot trip the breaker. Its `:07` past 15:00 UTC slot was picked only to avoid the `:00` slot shared by the `*/30` ETL trio (now a pair — see above) + hourly `artist-identity-etl`, a host-load courtesy unrelated to this policy.
- `album-critic-reviews-etl` (07:10 Sun) and `uncovered-release-list` (07:40 Sun) — non-LML. The latter reads Backend-Service's own Postgres only (rotation + album_critic_reviews + uncovered_release_search_markers) plus an optional GitHub Contents API push to `WXYC/research-data`; scheduled 30 min after the former so its anti-join sees that week's freshly-pulled `album_critic_reviews` rows.
- `album-critic-reviews-etl` (07:10 Sun) and `uncovered-release-list` (07:40 Sun) — non-LML. The latter reads Backend-Service's own Postgres only (rotation + flowsheet + the `album_plays` MV under `--backfill` + album_critic_reviews + uncovered_release_search_markers) plus an optional GitHub Contents API push to `WXYC/research-data`; scheduled 30 min after the former so its anti-join sees that week's freshly-pulled `album_critic_reviews` rows.

## The hourly safety net (BS#895)

Expand Down
Loading
Loading