diff --git a/decisions/README.md b/decisions/README.md new file mode 100644 index 0000000..ff6ed8e --- /dev/null +++ b/decisions/README.md @@ -0,0 +1,112 @@ +# Decisions + +_A decision report mined from your SpecStory coding histories._ + +Teams make dozens of consequential decisions inside coding sessions - which database, +which naming convention, which layout - and they evaporate into scrollback. Decisions +reads the `.specstory/history` transcripts your coding agents already write and +recovers them into an **evidence-cited decision log** (think auto-mined ADRs): + +- **Decided** - firm choices still standing +- **Changed** - decisions later superseded, with the full chain + (e.g. Postgres -> SQLite -> DuckDB) and what replaced them +- **Open** - questions raised but never resolved +- **Entities** - the thing each decision was about (a component, system, UI element, + file, or convention), used to build per-entity decision timelines + +## Insights + +Beyond the log itself, the report surfaces: + +- **Churn hotspots** - entities whose decisions were reversed 2+ times (design thrash, + unstable requirements) +- **Provisional-decision debt** - "for now" / temporary decisions that were never + revisited (each one is latent rework) +- **Re-litigated decisions** - settled entities where a later open question re-opened + the debate +- **Decisions lacking rationale** - flagged in the written report (the ones nobody will + be able to reconstruct later) + +## How it works + +A **deterministic engine** (plain Node, zero dependencies, no LLM or network) does the +high-recall pass: signal grammars find candidate decisions in user turns ("let's go +with...", "actually, switch to...", "should we...?", "for now..."), entities are +extracted from the matching sentence, candidates cluster into per-entity timelines per +project, and supersession/status falls out of timeline order. Duplicated prompt text is +deduped and ubiquitous entities (e.g. a product name that appears everywhere) never +chain unrelated decisions. Output is deterministic - byte-identical across runs. + +The **calling agent** then does the high-precision pass from the engine's JSON: drops +false positives, resolves referents ("I like option 1" -> what option 1 actually was, +by reading the cited evidence), names entities, and writes the final report. + +## Install + +From a clone of this repo: + +```zsh +cd decisions +./install.sh +``` + +That bundles the engine and the skill into `~/.agents/skills/decisions` and symlinks it +into `~/.claude/skills/decisions`, so `/decisions` is available from **any** Claude Code +session in **any** project. It is self-contained - it does not read from any other +skill's directory. Re-run `./install.sh` any time to update. + +Requirements: Node >= 22.5 (for `node:sqlite`), and the SpecStory CLI capturing +histories into `.specstory/history/`. + +## Use + +Start a new Claude Code session (skills load at session start), then: + +``` +/decisions +``` + +or just ask: _"what did we decide this month?"_, _"what's still undecided?"_, +_"what decisions changed?"_. With no arguments it asks three short questions - **Scope** +(which repos), **Window** (all time / 30 / 90 days), **Goal** (full report / open +decisions / changed decisions / insights) - then runs and saves the report to +`.specstory/decisions/-decisions.md`. + +Sample digest shape: + +``` +decisions report - 124 decision(s) across 5 project(s) (window: all time) + +Insights + Churn hotspots (decisions reversed 2+ times) + - EventStore (api): changed 2 time(s) - Postgres -> SQLite -> DuckDB + Provisional decisions never revisited ("for now" debt) + - nav (website, 2026-03-11): "It's ok for now if /specstory isn't linked to from the nav" + +## website + Decided + - root logo: "let's make the logo in the top left be the SpecStory logo" [decided] · 2026-03-11 + .specstory/history/2026-03-11_19-52-23Z.md:80802 + Changed + (none) + Open + - ads config: "should I leave these or do different config in google ads console?" [open] · 2026-04-09 +``` + +Each decision carries its evidence ref (`path:line`) so every claim is checkable. + +## Develop + +The engine is plain Node (ESM, zero dependencies, `node:sqlite`). Run the tests: + +```zsh +cd decisions +npm test +``` + +The engine's own corpus lives at `~/.specstory/decisions.db` (never shared with other +skills); use `--db /tmp/scratch.db` while developing. + +## License + +Apache-2.0. diff --git a/decisions/SKILL.md b/decisions/SKILL.md new file mode 100644 index 0000000..1f9c06e --- /dev/null +++ b/decisions/SKILL.md @@ -0,0 +1,84 @@ +--- +name: decisions +description: SpecStory Decisions - a decision report mined from SpecStory coding histories (any agent - Claude Code, Codex, Cursor, Gemini, and more). It finds the decisions made in a window of sessions, the decisions that were later changed (with the supersession chain), the still-open decisions, and the entities they were about - plus insights like churn hotspots and "for now" provisional-decision debt. Use when someone asks "what did we decide", "what decisions changed", "what's still undecided", "why did we choose X", or wants a decision log / ADR-style report over a .specstory/history corpus. +argument-hint: "Enter = guided setup · or plain English, e.g. 'last 30 days, just the open decisions'" +allowed-tools: Bash, Read, Write, AskUserQuestion +license: Apache-2.0 +metadata: + author: SpecStory + version: "1.0.0" +--- + +# Decisions + +Teams make dozens of small, consequential decisions inside coding sessions - which database, +which naming scheme, which layout - and they evaporate. **Decisions** recovers them from +SpecStory histories into an evidence-cited decision report (a lightweight, auto-mined ADR log): +what was **decided**, what was later **changed** (and what superseded it), what is still +**open**, and **which entities** (components, systems, UI elements) each decision was about. + +The work is split in two, and the split is the design: + +- A **deterministic engine** (`scripts/decisions.mjs`) does high-RECALL extraction: signal + grammars find candidate decisions in user turns, entities cluster them into per-entity + timelines, supersession and insights fall out of timeline order. Every candidate carries its + **quote** and an **evidence ref** (`path:line`). No LLM, no network in the engine. +- **You do the high-PRECISION pass**: drop false positives, resolve referents, name entities, + and write the report. Do not read raw transcripts wholesale - work from the engine's JSON and + open specific evidence refs only when a quote needs context. + +## Default flow + +1. **Index the corpus** (its own DB - nothing shared with other skills): + ```bash + node "${CLAUDE_SKILL_DIR}/scripts/decisions.mjs" index --projects --db + # or a single tree: --scan or a single history dir: --dir + ``` + +2. **Mine decisions** and capture both views: + ```bash + node "${CLAUDE_SKILL_DIR}/scripts/decisions.mjs" decisions --db [--days N] # digest + node "${CLAUDE_SKILL_DIR}/scripts/decisions.mjs" decisions --db [--days N] --json # for your pass + ``` + The JSON has `decisions[]` (project, status decided|changed|open, provisional flag, entity, + summary/quote, date, evidence, supersededBy, chain) and `insights` (churn, provisional + debt, reopened). + +3. **Your precision pass** - for each candidate: + - **Drop non-decisions.** The grammars are deliberately loose; an imperative instruction + with no choice in it ("look at all the places where...") is not a decision. Keep only + genuine choices between alternatives, policies, namings, or commitments. + - **Resolve referents.** Quotes like "I like option 1" or "go with your recommendation" are + real decisions with invisible content. `Read` the evidence file around the ref line to + recover WHAT was chosen, and restate it ("chose option 1: server-side rendering for the + landing page"). + - **Name entities.** Many candidates are `(unnamed)` - infer the entity from the quote and + context (a component, route, system, convention). Merge near-duplicate decisions. + - **Keep the evidence.** Every decision in your report cites its `path:line`. Never invent + a decision that has no engine candidate behind it. + +4. **Write the report** and save it to `.specstory/decisions/-decisions.md`: + - **Summary** - counts, projects, window. + - **Insights** - churn hotspots (entities re-decided 2+ times, with the chain), provisional + "for now" debt (never revisited - each is latent rework), re-litigated decisions (settled, + then re-questioned), and one you judge yourself: **decisions lacking any stated rationale** + (flag them; they are the ones nobody will be able to reconstruct later). + - **Per project:** **Decided** (entity, decision, date, rationale if stated, evidence) · + **Changed** (old -> new, when, and why if stated) · **Open** (the question, how long it has + been open, and your suggested next step for each). + Also offer `--out ` to keep the raw digest beside your written report. + +## Guided start + +Invoked bare, ask three short questions (`AskUserQuestion` or plain chat), then run: + +- **Scope** - which repos / parent directory of `.specstory/history` corpora? +- **Window** - all time (default - supersession needs history), or last 30 / 90 days? +- **Goal** - the full **report**, just **open decisions**, just **changed decisions**, or the + **insights** (churn + debt)? + +## Conventions + +Node ESM only, zero dependencies, Node >= 22.5. No em dashes anywhere (use " - "). The engine +is deterministic (byte-identical across runs on the same corpus); all judgment - filtering, +naming, rationale, next steps - is yours. diff --git a/decisions/fixtures/decisions-alpha/.specstory/history/2026-06-02_09-00-00Z-eventstore-postgres.md b/decisions/fixtures/decisions-alpha/.specstory/history/2026-06-02_09-00-00Z-eventstore-postgres.md new file mode 100644 index 0000000..2606a24 --- /dev/null +++ b/decisions/fixtures/decisions-alpha/.specstory/history/2026-06-02_09-00-00Z-eventstore-postgres.md @@ -0,0 +1,13 @@ +## 2026-06-02 09:00:00Z + + + +_**User**_ + +let's use Postgres for the EventStore persistence layer + +--- + +_**Agent (claude-opus-4-20250514)**_ + +Tool use: **Edit** `src/EventStore.ts` diff --git a/decisions/fixtures/decisions-alpha/.specstory/history/2026-06-09_09-00-00Z-eventstore-sqlite.md b/decisions/fixtures/decisions-alpha/.specstory/history/2026-06-09_09-00-00Z-eventstore-sqlite.md new file mode 100644 index 0000000..c4e4d99 --- /dev/null +++ b/decisions/fixtures/decisions-alpha/.specstory/history/2026-06-09_09-00-00Z-eventstore-sqlite.md @@ -0,0 +1,13 @@ +## 2026-06-09 09:00:00Z + + + +_**User**_ + +actually, let's switch the EventStore to SQLite instead of Postgres + +--- + +_**Agent (claude-opus-4-20250514)**_ + +Tool use: **Edit** `src/EventStore.ts` diff --git a/decisions/fixtures/decisions-alpha/.specstory/history/2026-06-10_09-00-00Z-authclient-fornow.md b/decisions/fixtures/decisions-alpha/.specstory/history/2026-06-10_09-00-00Z-authclient-fornow.md new file mode 100644 index 0000000..cdbbb83 --- /dev/null +++ b/decisions/fixtures/decisions-alpha/.specstory/history/2026-06-10_09-00-00Z-authclient-fornow.md @@ -0,0 +1,13 @@ +## 2026-06-10 09:00:00Z + + + +_**User**_ + +for now, keep the API key handling in AuthClient as a plain environment variable + +--- + +_**Agent (claude-opus-4-20250514)**_ + +Tool use: **Edit** `src/AuthClient.ts` diff --git a/decisions/fixtures/decisions-alpha/.specstory/history/2026-06-15_09-00-00Z-eventstore-duckdb.md b/decisions/fixtures/decisions-alpha/.specstory/history/2026-06-15_09-00-00Z-eventstore-duckdb.md new file mode 100644 index 0000000..74cb19b --- /dev/null +++ b/decisions/fixtures/decisions-alpha/.specstory/history/2026-06-15_09-00-00Z-eventstore-duckdb.md @@ -0,0 +1,13 @@ +## 2026-06-15 09:00:00Z + + + +_**User**_ + +on second thought, switch the EventStore to DuckDB + +--- + +_**Agent (claude-opus-4-20250514)**_ + +Tool use: **Edit** `src/EventStore.ts` diff --git a/decisions/fixtures/decisions-beta/.specstory/history/2026-06-18_09-00-00Z-settingspanel-question.md b/decisions/fixtures/decisions-beta/.specstory/history/2026-06-18_09-00-00Z-settingspanel-question.md new file mode 100644 index 0000000..216e3bc --- /dev/null +++ b/decisions/fixtures/decisions-beta/.specstory/history/2026-06-18_09-00-00Z-settingspanel-question.md @@ -0,0 +1,13 @@ +## 2026-06-18 09:00:00Z + + + +_**User**_ + +should we use tabs or a sidebar for the SettingsPanel layout? + +--- + +_**Agent (claude-opus-4-20250514)**_ + +Tool use: **Edit** `src/SettingsPanel.tsx` diff --git a/decisions/fixtures/decisions-beta/.specstory/history/2026-06-20_09-00-00Z-flagparser-kebab.md b/decisions/fixtures/decisions-beta/.specstory/history/2026-06-20_09-00-00Z-flagparser-kebab.md new file mode 100644 index 0000000..920d16c --- /dev/null +++ b/decisions/fixtures/decisions-beta/.specstory/history/2026-06-20_09-00-00Z-flagparser-kebab.md @@ -0,0 +1,13 @@ +## 2026-06-20 09:00:00Z + + + +_**User**_ + +go with kebab-case for all the CLI flag names in FlagParser + +--- + +_**Agent (claude-opus-4-20250514)**_ + +Tool use: **Edit** `src/FlagParser.ts` diff --git a/decisions/install.sh b/decisions/install.sh new file mode 100755 index 0000000..3c321d8 --- /dev/null +++ b/decisions/install.sh @@ -0,0 +1,27 @@ +#!/usr/bin/env bash +# Install the decisions skill so it is available from any Claude Code session. +# +# Self-contained: bundles decisions' own engine. No dependency on any other skill. +# Re-run any time to update. Pass a target dir to override the default. +set -euo pipefail + +HERE="$(cd "$(dirname "$0")" && pwd)" # .../decisions +DEST="${1:-$HOME/.agents/skills/decisions}" + +if [ ! -f "$HERE/scripts/decisions.mjs" ]; then + echo "error: engine not found at $HERE/scripts - run this from the decisions/ directory of a clone." >&2 + exit 1 +fi + +mkdir -p "$DEST" +rm -rf "$DEST/scripts" +cp -R "$HERE/scripts" "$DEST/scripts" # bundle the self-contained engine +cp "$HERE/SKILL.md" "$DEST/SKILL.md" # SKILL.md calls ${CLAUDE_SKILL_DIR}/scripts/decisions.mjs + +mkdir -p "$HOME/.claude/skills" +ln -sfn "$DEST" "$HOME/.claude/skills/decisions" + +echo "installed decisions:" +echo " skill -> $DEST" +echo " linked -> $HOME/.claude/skills/decisions" +echo "Open a new Claude Code session, then run /decisions (skills load at session start)." diff --git a/decisions/package.json b/decisions/package.json new file mode 100644 index 0000000..eedc6e6 --- /dev/null +++ b/decisions/package.json @@ -0,0 +1,11 @@ +{ + "name": "specstory-decisions", + "version": "1.0.0", + "description": "A decision report mined from SpecStory coding histories. Standalone skill.", + "type": "module", + "license": "Apache-2.0", + "engines": { "node": ">=22.5" }, + "scripts": { + "test": "node --test tests/*.test.mjs" + } +} diff --git a/decisions/scripts/decisions.mjs b/decisions/scripts/decisions.mjs new file mode 100644 index 0000000..fd0a3ad --- /dev/null +++ b/decisions/scripts/decisions.mjs @@ -0,0 +1,71 @@ +#!/usr/bin/env node +// decisions engine - CLI entry. A standalone skill: no dependency on any other skill. +// Modules under lib/ (zero npm deps, Node built-ins only): +// lib/patterns.mjs regexes / classifiers lib/parse.mjs transcript -> beats +// lib/discover.mjs project + file discovery lib/db.mjs SQLite corpus schema +// lib/indexer.mjs incremental indexing lib/decide.mjs decision mining + insights +// +// Subcommands: +// index --dir [--dir ...] | --projects | --scan [--db ] [--force] +// decisions [--db ] [--days N] [--json] [--out ] +// (also accepts the discovery flags above: it will index first, then render) +// +// Default DB: ~/.specstory/decisions.db (its own corpus; shares nothing with other skills). +import { join } from 'node:path' +import { homedir } from 'node:os' +import { writeFileSync } from 'node:fs' +import { openDb } from './lib/db.mjs' +import { indexCorpus } from './lib/indexer.mjs' +import { computeDecisions, renderDigest, decisionsJson } from './lib/decide.mjs' + +function parseArgs(argv) { + const a = { + cmd: '', dirs: [], projects: '', scan: '', + db: join(homedir(), '.specstory', 'decisions.db'), + days: 0, maxBytes: 200_000_000, force: false, json: false, out: '', + } + let i = 0 + if (argv[0] && !argv[0].startsWith('--')) { a.cmd = argv[0]; i = 1 } + for (; i < argv.length; i++) { + const t = argv[i] + if (t === '--dir') a.dirs.push(argv[++i]) + else if (t === '--projects') a.projects = argv[++i] + else if (t === '--scan') a.scan = argv[++i] + else if (t === '--db') a.db = argv[++i] + else if (t === '--days') a.days = +argv[++i] + else if (t === '--max-bytes') a.maxBytes = +argv[++i] + else if (t === '--force') a.force = true + else if (t === '--json' || t === '--emit=json') a.json = true + else if (t === '--out') a.out = argv[++i] + } + if (!a.cmd) a.cmd = 'decisions' + return a +} + +const ARGS = parseArgs(process.argv.slice(2)) +const db = openDb(ARGS.db) + +if (ARGS.cmd === 'index') { + const r = indexCorpus(db, ARGS) + if (r.error) { process.stderr.write(`decisions index: ${r.error}\n`); process.exit(2) } + process.stdout.write(`indexed ${r.indexed} sessions (${r.skippedKnown} unchanged, ${r.skippedBig} too large) across ${r.projects.length} project(s) -> ${ARGS.db}\n`) +} else if (ARGS.cmd === 'decisions' || ARGS.cmd === 'report') { + // Convenience: with discovery flags, index first (unbounded - supersession needs the + // full timeline even for a windowed report), then mine from the same corpus. + if (ARGS.dirs.length || ARGS.projects || ARGS.scan) { + const r = indexCorpus(db, { ...ARGS, days: 0 }) + if (r.error) { process.stderr.write(`decisions: ${r.error}\n`); process.exit(2) } + } + const result = computeDecisions(db, { days: ARGS.days }) + if (ARGS.json) { + process.stdout.write(JSON.stringify(decisionsJson(result), null, 2) + '\n') + } else { + const digest = renderDigest(result, { days: ARGS.days }) + const text = digest.endsWith('\n') ? digest : digest + '\n' + process.stdout.write(text) + if (ARGS.out) writeFileSync(ARGS.out, text) + } +} else { + process.stderr.write('usage: decisions.mjs index|decisions [--dir D | --projects P | --scan R] [--db PATH] [--days N] [--json] [--out FILE]\n') + process.exit(2) +} diff --git a/decisions/scripts/lib/db.mjs b/decisions/scripts/lib/db.mjs new file mode 100644 index 0000000..ae5ef6e --- /dev/null +++ b/decisions/scripts/lib/db.mjs @@ -0,0 +1,76 @@ +// db.mjs - the lore corpus schema (SQLite via node:sqlite, Node >= 22.5; zero deps). + +import { mkdirSync } from 'node:fs' +import { dirname } from 'node:path' +import { DatabaseSync } from 'node:sqlite' + +// Bump whenever the PARSER's extraction behavior changes: sessions indexed under an older +// parser are automatically re-parsed on the next `index` run (no manual purge needed). +export const PARSER_VERSION = 6 // 6 = shell control-flow keywords dropped from command heads + +export function openDb(path) { + mkdirSync(dirname(path), { recursive: true }) + const db = new DatabaseSync(path) + // one-time migration: the unit was renamed episode -> beat (2026-06-10). In-place renames so + // existing corpora keep their expensive artifacts (themes, dossiers) without re-mining. + try { db.exec('ALTER TABLE episodes RENAME TO beats') } catch { /* already migrated or fresh */ } + for (const [t, a, b] of [['commands', 'episode_id', 'beat_id'], ['grams', 'episode_id', 'beat_id'], + ['meta_hits', 'episode_id', 'beat_id'], ['themes', 'episode_keys', 'beat_keys'], ['sessions', 'episodes', 'beats']]) { + try { db.exec(`ALTER TABLE ${t} RENAME COLUMN ${a} TO ${b}`) } catch { /* already migrated or fresh */ } + } + db.exec(` + PRAGMA journal_mode=WAL; + PRAGMA busy_timeout=10000; -- concurrent runs (e.g. a session-end hook + a manual index) wait, not crash + CREATE TABLE IF NOT EXISTS sessions( + id TEXT PRIMARY KEY, project_id TEXT, project_name TEXT, path TEXT, date TEXT, + agent TEXT, size INTEGER, beats INTEGER); + CREATE TABLE IF NOT EXISTS beats( + id INTEGER PRIMARY KEY AUTOINCREMENT, session_id TEXT, ord INTEGER, start_line INTEGER, + intent_raw TEXT, intent_sig TEXT, + n_tools INTEGER, tool_mix TEXT, files TEXT, n_cmds INTEGER, exit_fails INTEGER, + outcome TEXT); + CREATE TABLE IF NOT EXISTS commands( + beat_id INTEGER, ord INTEGER, head TEXT, raw TEXT, line INTEGER); + CREATE TABLE IF NOT EXISTS grams( + beat_id INTEGER, n INTEGER, gram TEXT); + CREATE TABLE IF NOT EXISTS meta_hits( + beat_id INTEGER, meta_id TEXT, quote TEXT, line INTEGER); + CREATE INDEX IF NOT EXISTS idx_ep_session ON beats(session_id); + CREATE INDEX IF NOT EXISTS idx_grams ON grams(gram); + CREATE INDEX IF NOT EXISTS idx_meta ON meta_hits(meta_id); + CREATE INDEX IF NOT EXISTS idx_cmd_ep ON commands(beat_id); + `) + // migrations for corpora created before these columns existed + for (const col of ['mtime INTEGER DEFAULT 0', 'parser INTEGER DEFAULT 0', "author TEXT DEFAULT ''", "uuid TEXT DEFAULT ''"]) { + try { db.exec(`ALTER TABLE sessions ADD COLUMN ${col}`) } catch { /* already there */ } + } + return db +} + +// ---------- the runs journal: Lore's memory of its own activity ---------- +// One row per notable engine invocation (auto) plus one per /lore run (the agent's Step 6 duty). +// This is what makes "what has Lore done?" a query instead of archaeology. + +export function ensureRunsTable(db) { + db.exec(`CREATE TABLE IF NOT EXISTS runs( + id INTEGER PRIMARY KEY AUTOINCREMENT, ts TEXT, cmd TEXT, scope TEXT, summary TEXT)`) +} + +export function addRun(db, cmd, scope, summary) { + ensureRunsTable(db) + db.prepare('INSERT INTO runs(ts, cmd, scope, summary) VALUES(?,?,?,?)') + .run(new Date().toISOString(), cmd, scope || '', summary || '') +} + +export function listRuns(db, limit = 10) { + ensureRunsTable(db) + return db.prepare('SELECT ts, cmd, scope, summary FROM runs ORDER BY id DESC LIMIT ?').all(limit) +} + +// Delete a session's beat children + beats (shared by re-index and prune). +export function deleteSessionRows(db, sessionId) { + for (const r of db.prepare('SELECT id FROM beats WHERE session_id=?').all(sessionId)) { + for (const t of ['commands', 'grams', 'meta_hits']) db.prepare(`DELETE FROM ${t} WHERE beat_id=?`).run(r.id) + } + db.prepare('DELETE FROM beats WHERE session_id=?').run(sessionId) +} diff --git a/decisions/scripts/lib/decide.mjs b/decisions/scripts/lib/decide.mjs new file mode 100644 index 0000000..9f5bd45 --- /dev/null +++ b/decisions/scripts/lib/decide.mjs @@ -0,0 +1,245 @@ +// decide.mjs - the decision-mining engine (query-time lens over the indexed corpus). +// +// Deterministic and high-RECALL: signal grammars find decision candidates in user turns, +// entities are extracted from the matching sentence, candidates cluster into per-entity +// timelines (per project), and supersession/status falls out of the timeline order. Every +// candidate carries its QUOTE and evidence ref so the agent (SKILL.md) can do the +// high-PRECISION pass - dropping false positives and naming decisions - without reading +// raw transcripts. No LLM or network call in this module. +// +// STATUS MODEL (one per decision): +// decided - a firm choice, still standing (the latest firm decision on its entity) +// changed - superseded by a later firm decision on the same entity (chained) +// open - raised but unresolved: a question, or provisional ("for now") with no +// later firm decision on the entity +// Flags: provisional ("for now"/TBD language), reopened (a later open question on an +// entity that already had a firm decision). +// +// INSIGHTS (computed here, deterministically): +// churn - entities whose decisions were reversed >= 2 times (design thrash) +// provisional - "for now" decisions never revisited (decision debt) +// reopened - settled entities re-questioned later (re-litigated decisions) + +import { relative } from 'node:path' +import { redactSecrets } from './patterns.mjs' + +// --- signal grammars (ordered by precedence: CHANGE > OPEN > DECIDE > PROVISIONAL) --- +// A change is itself a decision (it decides the new thing) AND marks its predecessors. +const CHANGE_RE = /\b(?:actually,?\s+(?:let'?s|we|use|go|switch|make)|instead of what|changed?\s+(?:my|our)\s+mind|scrap\s+that|forget\s+that|on\s+second\s+thought|let'?s\s+not\b|go\s+back\s+to|switch(?:ing)?\s+(?:\S+\s+){0,3}?(?:back\s+)?to\b|revert\s+to|no\s+longer\s+(?:use|using)|replace\s+\S+\s+with)/i +const DECIDE_RE = /\b(?:let'?s\s+(?:go\s+with|use|stick\s+with|keep|standardize\s+on|make|call|do)|go(?:ing)?\s+with|we(?:'ll|\s+will)\s+(?:use|go|keep|stick)|decided?\s+(?:on|to|that)|decision\s*:|settle[d]?\s+on|agreed?\s+(?:on|to)\b|standardize\s+on|default\s+to|always\s+use|never\s+use|don'?t\s+use\s+(?:\S+\s+){0,4}?use|rename\s+(?:\S+\s+){0,4}?to\b|call\s+it\s+\S|(?:option|choice)\s+\d|the\s+(?:first|second|third)\s+option|use\s+(?:\S+\s+){0,4}?(?:instead\s+of|rather\s+than))/i +const OPEN_RE = /(?:\bshould\s+(?:we|i|it|this)\b[^.?!\n]*\?|\bwhich\b[^.?!\n]*\?|\bdo\s+we\s+want\b[^.?!\n]*\?|\bwhat\s+about\b[^.?!\n]*\?|\bor\s+should\b[^.?!\n]*\?)/i +// NOTE: "placeholder" is deliberately absent - it is a common UI noun ("placeholder text"), +// not a provisional-decision marker, and it flooded the report with false positives. +const PROVISIONAL_RE = /\b(?:for\s+now|for\s+the\s+moment|temporar(?:y|ily)|as\s+a\s+stopgap|tbd|decide\s+(?:this\s+)?later|punt(?:ing)?\s+on|revisit\s+(?:this\s+)?later)\b/i + +// Entity extraction from the matching sentence: distinctive symbols (camelCase, +// snake_case, ALL_CAPS_WITH_UNDERSCORE), backtick spans, and file-like tokens. Plain +// English words are deliberately NOT entities (too generic to cluster on). +const SYMBOL_RE = /[A-Za-z][A-Za-z0-9]*(?:_[A-Za-z0-9]+)+|[a-z][a-z0-9]*(?:[A-Z][a-z0-9]+)+|[A-Z][a-z0-9]+(?:[A-Z][a-z0-9]+)+/g +const FILE_RE = /(?:\.{0,2}\/)?(?:[\w@.-]+\/)+[\w@-]+\.[A-Za-z][\w]{0,9}|[\w@-]+\.[A-Za-z][\w]{0,9}/g +const TICK_RE = /`([^`\n]{2,60})`/g +const NOISE_ENTITY = new Set(['env.local', '.env.local', 'package.json', 'readme.md', 'claude.md']) + +const matchAll = (re, s) => { const out = []; if (!s) return out; for (const m of s.matchAll(re)) out.push(m[1] ?? m[0]); return out } + +function entitiesOf(sentence) { + const found = new Set() + for (const t of matchAll(TICK_RE, sentence)) for (const s of matchAll(SYMBOL_RE, t)) found.add(s) + for (const s of matchAll(SYMBOL_RE, sentence)) found.add(s) + for (const f of matchAll(FILE_RE, sentence)) if (!NOISE_ENTITY.has(f.toLowerCase())) found.add(f) + return [...found].filter((e) => e.length >= 4 && !NOISE_ENTITY.has(e.toLowerCase())) +} + +// Split a user turn into sentence-ish units (newlines and terminal punctuation). +function sentences(text) { + return (text || '').split(/(?<=[.?!])\s+|\n+/).map((s) => s.trim()).filter(Boolean) +} + +const clip = (s, n = 160) => (s.length > n ? s.slice(0, n - 1) + '…' : s) + +// Disjoint-set for entity clustering. +function makeDSU(n) { + const p = Array.from({ length: n }, (_, i) => i) + const find = (x) => { while (p[x] !== x) { p[x] = p[p[x]]; x = p[x] } return x } + const union = (a, b) => { const ra = find(a), rb = find(b); if (ra !== rb) p[Math.max(ra, rb)] = Math.min(ra, rb) } + return { find, union } +} + +// Pure read of the corpus DB; `now` injectable for deterministic tests. +export function computeDecisions(db, { now = Date.now(), days = 0 } = {}) { + const rows = db.prepare(` + SELECT s.id sid, s.project_id pid, s.project_name pname, s.date date, s.path path, + e.ord ord, e.start_line line, e.intent_raw intent + FROM beats e JOIN sessions s ON s.id = e.session_id + WHERE s.date IS NOT NULL AND s.date != '' AND e.intent_raw IS NOT NULL AND e.intent_raw != '' + ORDER BY s.project_id, s.date, s.id, e.ord + `).all() + + const cutoff = days > 0 ? new Date(now - days * 86400000).toISOString().slice(0, 10) : null + const cwd = process.cwd() + + // 1) Extract candidates: one per (beat, matching sentence). + const cands = [] + for (const r of rows) { + if (cutoff && r.date < cutoff) continue + for (const sent of sentences(r.intent)) { + const change = CHANGE_RE.test(sent) + const open = OPEN_RE.test(sent) + const decide = DECIDE_RE.test(sent) + const provisional = PROVISIONAL_RE.test(sent) + if (!change && !open && !decide && !provisional) continue + // precedence: an explicit question is open even if it contains decision verbs + const kind = change ? 'change' : (open ? 'open' : (decide ? 'decide' : 'provisional')) + cands.push({ + pid: r.pid, project: r.pname, sid: r.sid, date: r.date, ord: r.ord, + evidence: `${relative(cwd, r.path) || r.path}:${r.line}`, + quote: redactSecrets(clip(sent)), + kind, provisional, + entities: entitiesOf(sent), + signals: [change && 'change', open && 'open', decide && 'decide', provisional && 'provisional'].filter(Boolean), + }) + } + } + // 1b) Dedupe: resumed/re-included sessions repeat the same prompt text verbatim, + // which would multiply one decision into many. Keep the EARLIEST occurrence of an + // identical (project, quote) pair - rows arrive in project/date order. + const seen = new Set() + const uniq = [] + for (const c of cands) { + const k = c.pid + '\t' + c.quote.toLowerCase() + if (seen.has(k)) continue + seen.add(k) + uniq.push(c) + } + const list = uniq + if (!list.length) return { decisions: [], insights: { churn: [], provisional: [], reopened: [] } } + + // 2) Cluster candidates by shared entity, per project. Decisions are sparse, so a + // single shared distinctive entity is a legitimate link (unlike work threads) - + // EXCEPT ubiquitous entities (a product name appears everywhere and would chain + // unrelated decisions). An entity in more than ENTITY_MAX sessions is not a key. + const ENTITY_MAX = 5 + const entitySids = new Map() + for (const c of list) for (const e of c.entities) { + const k = c.pid + '\t' + e.toLowerCase() + if (!entitySids.has(k)) entitySids.set(k, new Set()) + entitySids.get(k).add(c.sid) + } + const ubiquitous = (pid, e) => (entitySids.get(pid + '\t' + e.toLowerCase())?.size || 0) > ENTITY_MAX + const dsu = makeDSU(list.length) + const owner = new Map() + list.forEach((c, i) => { + for (const e of c.entities) { + if (ubiquitous(c.pid, e)) continue + const k = c.pid + '\t' + e.toLowerCase() + if (owner.has(k)) dsu.union(i, owner.get(k)) + else owner.set(k, i) + } + }) + const clusters = new Map() + list.forEach((c, i) => { + const root = dsu.find(i) + if (!clusters.has(root)) clusters.set(root, []) + clusters.get(root).push(c) + }) + + // 3) Per-cluster timeline -> statuses, supersession chain, flags, insights. + const decisions = [] + const churn = [], provisionalDebt = [], reopened = [] + for (const members of clusters.values()) { + members.sort((a, b) => a.date.localeCompare(b.date) || a.sid.localeCompare(b.sid) || a.ord - b.ord) + // firm = anything that settles a choice (even provisionally); open questions are not firm + const firm = members.filter((m) => m.kind !== 'open') + // entity display name: most frequent original-case entity across the cluster, + // preferring distinctive (non-ubiquitous) names over product-name noise + const counts = new Map() + for (const m of members) for (const e of m.entities) counts.set(e, (counts.get(e) || 0) + 1) + const ranked = [...counts.entries()].sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0])) + const entity = (ranked.find(([e]) => !ubiquitous(members[0].pid, e)) || ranked[0])?.[0] || '(unnamed)' + + const lastFirm = firm.length ? firm[firm.length - 1] : null + const reversals = Math.max(0, firm.length - 1) + const lastOpen = [...members].reverse().find((m) => m.kind === 'open') + const isReopened = !!(lastFirm && lastOpen && (lastOpen.date > lastFirm.date || (lastOpen.date === lastFirm.date && lastOpen.ord > lastFirm.ord))) + + for (const m of members) { + let status + if (m.kind === 'open') status = 'open' + else if (m === lastFirm) status = 'decided' + else status = 'changed' // an earlier firm decision, superseded + const idx = firm.indexOf(m) + decisions.push({ + project: m.project, status, entity, entities: m.entities, + provisional: m.provisional, reopened: status === 'decided' && isReopened, + summary: m.quote, date: m.date, evidence: m.evidence, signals: m.signals, + supersededBy: status === 'changed' && idx >= 0 && firm[idx + 1] ? firm[idx + 1].quote : (status === 'changed' && lastFirm ? lastFirm.quote : null), + chain: firm.length > 1 ? firm.map((f) => f.quote) : null, + }) + } + if (reversals >= 2) churn.push({ project: members[0].project, entity, reversals, chain: firm.map((f) => clip(f.quote, 80)) }) + if (lastFirm && lastFirm.provisional) provisionalDebt.push({ project: lastFirm.project, entity, date: lastFirm.date, quote: lastFirm.quote, evidence: lastFirm.evidence }) + if (isReopened) reopened.push({ project: members[0].project, entity, decidedOn: lastFirm.date, reopenedOn: lastOpen.date, question: lastOpen.quote }) + } + + // Deterministic order: project, status rank, date, summary. + const rank = { decided: 0, changed: 1, open: 2 } + decisions.sort((a, b) => a.project.localeCompare(b.project) || rank[a.status] - rank[b.status] || + a.date.localeCompare(b.date) || a.summary.localeCompare(b.summary)) + const byName = (a, b) => a.project.localeCompare(b.project) || a.entity.localeCompare(b.entity) + churn.sort(byName); provisionalDebt.sort(byName); reopened.sort(byName) + return { decisions, insights: { churn, provisional: provisionalDebt, reopened } } +} + +const SECTIONS = [['Decided', 'decided'], ['Changed', 'changed'], ['Open', 'open']] + +// Digest: insights first (the "so what"), then per project: Decided / Changed / Open. +export function renderDigest(result, { days = 0 } = {}) { + const { decisions, insights } = result + const projects = new Set(decisions.map((d) => d.project)) + const L = [] + L.push(`decisions report - ${decisions.length} decision(s) across ${projects.size} project(s) (window: ${days > 0 ? `last ${days} days` : 'all time'})`) + L.push('') + L.push('Insights') + L.push(' Churn hotspots (decisions reversed 2+ times)') + if (!insights.churn.length) L.push(' (none)') + for (const c of insights.churn) L.push(` - ${c.entity} (${c.project}): changed ${c.reversals} time(s) - ${c.chain.join(' -> ')}`) + L.push(' Provisional decisions never revisited ("for now" debt)') + if (!insights.provisional.length) L.push(' (none)') + const prov = [...insights.provisional].sort((a, b) => b.date.localeCompare(a.date) || a.entity.localeCompare(b.entity)) + for (const p of prov.slice(0, 12)) L.push(` - ${p.entity} (${p.project}, ${p.date}): "${p.quote}"`) + if (prov.length > 12) L.push(` (+${prov.length - 12} more - see the JSON or full report)`) + L.push(' Re-litigated (settled, then re-questioned)') + if (!insights.reopened.length) L.push(' (none)') + for (const r of insights.reopened) L.push(` - ${r.entity} (${r.project}): decided ${r.decidedOn}, re-opened ${r.reopenedOn}: "${r.question}"`) + L.push('') + + const byProject = new Map() + for (const d of decisions) { + if (!byProject.has(d.project)) byProject.set(d.project, []) + byProject.get(d.project).push(d) + } + const names = [...byProject.keys()].sort() + if (!names.length) L.push('(no decisions found in this window)') + for (const name of names) { + L.push(`## ${name}`) + const list = byProject.get(name) + for (const [label, status] of SECTIONS) { + L.push(` ${label}`) + const items = list.filter((d) => d.status === status) + if (!items.length) { L.push(' (none)'); continue } + for (const d of items) { + const marks = [d.provisional && 'provisional', d.reopened && 're-opened'].filter(Boolean) + L.push(` - ${d.entity}: "${d.summary}" [${d.status}${marks.length ? ' · ' + marks.join(' · ') : ''}] · ${d.date}`) + if (d.status === 'changed' && d.supersededBy) L.push(` superseded by: "${clip(d.supersededBy, 100)}"`) + L.push(` ${d.evidence}`) + } + } + L.push('') + } + return L.join('\n').replace(/\n+$/, '\n') +} + +// JSON view: the full result object (decisions array + insights), agent-consumable. +export function decisionsJson(result) { + return result +} diff --git a/decisions/scripts/lib/discover.mjs b/decisions/scripts/lib/discover.mjs new file mode 100644 index 0000000..45e1ebf --- /dev/null +++ b/decisions/scripts/lib/discover.mjs @@ -0,0 +1,88 @@ +// discover.mjs - find projects and transcripts on disk; resolve stable project identity. + +import { readFileSync, readdirSync, existsSync } from 'node:fs' +import { join, basename, dirname, resolve } from 'node:path' +import { FILE_DATE } from './patterns.mjs' + +// Recursively list every *.md under a history dir (handles specstory-organize year/month layouts). +export function walkMd(dir) { + const out = [] + let ents + try { ents = readdirSync(dir, { withFileTypes: true }) } catch { return out } + for (const e of ents) { + const p = join(dir, e.name) + if (e.isDirectory()) out.push(...walkMd(p)) + else if (e.isFile() && e.name.endsWith('.md')) out.push(p) + } + return out +} + +// Stable project identity from .specstory/.project.json: git_id (hash of normalized git remote - +// globally stable) preferred over workspace_id (path hash - machine-local), else the dir name. +export function readLabel(historyDir) { + const pj = join(historyDir, '..', '.project.json') + let id = null, name = null + try { + const j = JSON.parse(readFileSync(pj, 'utf8')) + id = j.git_id || j.workspace_id || null + name = j.project_name || null + } catch { /* not a .specstory layout */ } + const root = basename(dirname(dirname(historyDir))) || basename(historyDir) + return { id: id || root, name: name || root } +} + +// Find every .specstory/history under a root, at ANY depth (monorepos nest histories in +// sub-packages). Skips dependency/build dirs and other dotdirs; bounded depth as a tripwire. +export function scanForHistories(root, maxDepth = 6) { + const SKIP = new Set(['node_modules', 'build', 'dist', 'out', 'target', 'vendor', 'Pods', 'DerivedData']) + const found = [] + const walk = (dir, depth) => { + if (depth > maxDepth) return + let ents + try { ents = readdirSync(dir, { withFileTypes: true }) } catch { return } + for (const e of ents) { + if (!e.isDirectory()) continue + if (e.name === '.specstory') { + const hd = join(dir, '.specstory', 'history') + if (existsSync(hd)) found.push(hd) + continue + } + if (SKIP.has(e.name) || e.name.startsWith('.')) continue + walk(join(dir, e.name), depth + 1) + } + } + walk(resolve(root), 0) + return found +} + +// Resolve --dir flags, a --projects parent, and/or a --scan root into [{historyDir, id, name}]. +// Dirs are resolved to ABSOLUTE paths before anything is stored: the corpus carries session +// paths that must stay valid from any future working directory (prune, beats export, +// evidence refs). Relative input is a cwd-of-the-moment accident, never a contract. +export function discoverProjects(a) { + const out = [], seen = new Set() + const addHist = (hd) => { + if (!hd) return + hd = resolve(hd) + if (!existsSync(hd) || seen.has(hd)) return + seen.add(hd) + const lab = readLabel(hd) + out.push({ historyDir: hd, id: lab.id, name: lab.name }) + } + if (a.scan) for (const hd of scanForHistories(a.scan)) addHist(hd) + if (a.projects) { + const parent = resolve(a.projects) + let kids = [] + try { kids = readdirSync(parent) } catch { /* ignore */ } + for (const k of kids) addHist(join(parent, k, '.specstory', 'history')) + } + for (const d of a.dirs) addHist(d) + return out +} + +export function fileDate(path, head) { + let m = FILE_DATE.exec(basename(path)) + if (m) return m[1] + m = FILE_DATE.exec(head) + return m ? m[1] : null +} diff --git a/decisions/scripts/lib/indexer.mjs b/decisions/scripts/lib/indexer.mjs new file mode 100644 index 0000000..3a37221 --- /dev/null +++ b/decisions/scripts/lib/indexer.mjs @@ -0,0 +1,136 @@ +// indexer.mjs - walk projects, parse new/changed transcripts (via parse.mjs), persist beats. +// +// IDEMPOTENCY CONTRACT: +// - session identity = project_id + '/' + filename (stable across re-runs and dir reorganizations) +// - fingerprint = size + mtime + PARSER_VERSION: a session is skipped only when the file is +// unchanged AND it was indexed by the current parser. Grown/edited sessions are REPLACED whole +// (beats + children deleted, re-inserted). Engine upgrades re-parse automatically. +// - `--force` re-indexes everything regardless. `prune` removes sessions whose file is gone. + +import { readFileSync, statSync, existsSync, realpathSync } from 'node:fs' +import { basename, relative } from 'node:path' +import { execFileSync } from 'node:child_process' +import { discoverProjects, walkMd, fileDate } from './discover.mjs' +import { parseSessionFile, intentSig, sniffAuthor } from './parse.mjs' +import { PARSER_VERSION, deleteSessionRows } from './db.mjs' + +// Authoritative author per transcript: who ADDED the file to git (one batched call per project). +// Histories committed to a shared repo carry their session owner this way; local-only files fall +// through to path-sniffing, then the machine user (an uncommitted transcript on this machine is +// almost certainly the local user's session). +function gitAuthors(historyDir) { + const map = new Map() + try { + const root = execFileSync('git', ['rev-parse', '--show-toplevel'], { cwd: historyDir, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] }).trim() + const rel = relative(root, realpathSync(historyDir)) // realpath: /tmp vs /private/tmp etc. + const out = execFileSync('git', ['log', '--diff-filter=A', '--format=\x01%an', '--name-only', '--', rel], + { cwd: root, encoding: 'utf8', maxBuffer: 64 * 1024 * 1024, stdio: ['ignore', 'pipe', 'ignore'] }) + let author = '' + for (const line of out.split('\n')) { + if (line.startsWith('\x01')) { author = line.slice(1).trim(); continue } + const f = line.trim() + if (f && author && !map.has(basename(f))) map.set(basename(f), author) + } + } catch { /* not a git repo, or git unavailable - fall through to sniffing */ } + return map +} + +export function indexCorpus(db, args) { + const projects = discoverProjects(args) + if (!projects.length) return { indexed: 0, skippedKnown: 0, skippedBig: 0, projects, error: 'no .specstory history found' } + const cutoff = args.days > 0 ? new Date(Date.now() - args.days * 864e5).toISOString().slice(0, 10) : null + + const known = new Map(db.prepare('SELECT id, size, mtime, parser FROM sessions').all().map(r => [r.id, r])) + const insSession = db.prepare('INSERT OR REPLACE INTO sessions(id,project_id,project_name,path,date,agent,size,beats,mtime,parser,author,uuid) VALUES(?,?,?,?,?,?,?,?,?,?,?,?)') + const insEp = db.prepare('INSERT INTO beats(session_id,ord,start_line,intent_raw,intent_sig,n_tools,tool_mix,files,n_cmds,exit_fails,outcome) VALUES(?,?,?,?,?,?,?,?,?,?,?)') + const insCmd = db.prepare('INSERT INTO commands(beat_id,ord,head,raw,line) VALUES(?,?,?,?,?)') + const insGram = db.prepare('INSERT INTO grams(beat_id,n,gram) VALUES(?,?,?)') + const insMeta = db.prepare('INSERT INTO meta_hits(beat_id,meta_id,quote,line) VALUES(?,?,?,?)') + + let indexed = 0, skippedKnown = 0, skippedBig = 0 + for (const proj of projects) { + // live feedback to stderr: visible in the agent window, never pollutes stdout/--emit json + const t0 = Date.now() + let projNew = 0 + process.stderr.write(`📜 decisions · indexing ${proj.name} …\n`) + const authors = gitAuthors(proj.historyDir) + for (const path of walkMd(proj.historyDir)) { + let st + try { st = statSync(path) } catch { continue } + const size = st.size, mtime = Math.floor(st.mtimeMs) + if (size > args.maxBytes) { skippedBig++; continue } + const sid = proj.id + '/' + basename(path) + const k = known.get(sid) + if (!args.force && k && k.size === size && k.mtime === mtime && k.parser === PARSER_VERSION) { + skippedKnown++; continue // unchanged file, current parser → already indexed + } + + let text + try { text = readFileSync(path, 'utf8') } catch { continue } + const date = fileDate(path, text.slice(0, 600)) + if (cutoff && date && date < cutoff) continue + + const { agent, uuid, beats: eps } = parseSessionFile(text) + indexed++; projNew++ + if (projNew % 100 === 0) process.stderr.write(` … ${projNew} sessions parsed\n`) + // author ladder: git add-author (authoritative) > home-dir sniff > machine user + const author = authors.get(basename(path)) || sniffAuthor(text) || process.env.USER || 'unknown' + + // replace this session's prior rows, then write fresh - ATOMICALLY. Without the transaction, + // a crash mid-write leaves a fingerprint-matched session row with missing children that the + // incremental skip would then preserve forever. (Batching also makes indexing much faster.) + db.exec('BEGIN') + try { + deleteSessionRows(db, sid) + insSession.run(sid, proj.id, proj.name, path, date, agent, size, eps.length, mtime, PARSER_VERSION, author, uuid) + for (let k2 = 0; k2 < eps.length; k2++) { + const e = eps[k2] + const sig = intentSig(e.intent) + const mix = Object.entries(e.tools).map(([t, c]) => `${t}:${c}`).join(',') + const res = insEp.run(sid, k2, e.startLine, (e.firstLine || '').slice(0, 200), sig, + e.nTools, mix, [...e.files].slice(0, 20).join(','), e.cmds.length, e.fails, e.outcome) + const epId = res.lastInsertRowid + e.cmds.forEach((c, ord) => insCmd.run(epId, ord, c.head, c.raw, c.line)) + const heads = e.cmds.map(c => c.head) + const seen = new Set() + for (let n = 2; n <= 4; n++) for (let s = 0; s + n <= heads.length; s++) { + const g = heads.slice(s, s + n).join(' ▸ ') + const key = n + '|' + g + if (!seen.has(key)) { seen.add(key); insGram.run(epId, n, g) } // dedupe within beat + } + for (const m of e.metas) insMeta.run(epId, m.id, m.quote, m.line) + } + db.exec('COMMIT') + } catch (err) { + try { db.exec('ROLLBACK') } catch { /* not in a tx */ } + throw err + } + } + process.stderr.write(` ${proj.name}: +${projNew} new (${((Date.now() - t0) / 1000).toFixed(1)}s)\n`) + } + return { indexed, skippedKnown, skippedBig, projects } +} + +// prune: drop sessions whose transcript no longer exists on disk (deleted/moved corpora), +// and report duplicate filename groups that exist under multiple project_ids (the +// "project gained a git remote → new git_id" drift case) so the caller can decide. +export function pruneCorpus(db) { + const rows = db.prepare('SELECT id, path FROM sessions').all() + let removed = 0 + for (const r of rows) { + if (!existsSync(r.path)) { + deleteSessionRows(db, r.id) + db.prepare('DELETE FROM sessions WHERE id=?').run(r.id) + removed++ + } + } + const dupes = db.prepare(` + SELECT path, COUNT(DISTINCT project_id) np, GROUP_CONCAT(DISTINCT project_id) pids + FROM sessions GROUP BY path HAVING np > 1`).all() + // the SAME session (by provider UUID) indexed from two places - copied/cloned corpora + // double-count every pattern and fake "portable" signals + const contentDupes = db.prepare(` + SELECT uuid, COUNT(*) n, GROUP_CONCAT(id, ' | ') ids + FROM sessions WHERE uuid != '' GROUP BY uuid HAVING n > 1`).all() + return { removed, remaining: rows.length - removed, dupes, contentDupes } +} diff --git a/decisions/scripts/lib/parse.mjs b/decisions/scripts/lib/parse.mjs new file mode 100644 index 0000000..3b415f1 --- /dev/null +++ b/decisions/scripts/lib/parse.mjs @@ -0,0 +1,245 @@ +// parse.mjs - pure transcript parsing: text in, beats out. No I/O, no DB. +// +// All format knowledge lives in patterns.mjs (the grammar); this file is only the walk: +// segment user turns into beats, attach tool activity, label outcomes retroactively. +// +// THE UNIT IS THE BEAT: one user turn (INTENT) + all agent activity until the next user turn +// (METHOD: tool mix, executed commands, files touched, exit codes) + the NEXT user turn's reaction +// (OUTCOME label). Commands come only from executed shell tool blocks, so every command counted +// was genuinely run by the agent. + +import { + SESSION_HDR, USER_MARK, TURN_MARK, TOOLUSE_OPEN, SHELL_TOOLS, SHELL_EXCLUDE, + SUMMARY_CMD, INLINE_CMD, FENCE_LINE, SHELL_FENCE_LANGS, HEREDOC_OPEN, + BULLET_CMD, BULLET_CMD_OPEN, BULLET_CMD_CLOSE, + LEGACY_TOOL, LEGACY_TYPE, LEGACY_SHELL_NAMES, LEGACY_OUTPUT_MARK, + EXIT_CODE, ERROR_HEAD, NOISE, META, VERBS, + tokenize, leadingVerb, classifyOutcome, +} from './patterns.mjs' + +// raw command string -> ordered list of meaningful command heads (project tools kept, recon dropped) +export function headsFrom(cmd) { + const heads = [] + for (let seg of cmd.split(/&&|;/)) { + seg = seg.trim().split('|')[0].trim().replace(/^\$\s+/, '') + if (!seg) continue + seg = seg.replace(/^(?:[A-Z_][A-Z0-9_]*=\S+\s+)+/, '') // strip FOO=bar prefixes + const w = seg.match(/^(?:bash|sh|zsh)\s+-l?c\s+["']?(.+?)["']?$/) // unwrap bash -lc '...' + if (w) seg = w[1].trim() + const toks = seg.split(/\s+/) + let head = toks[0] + // head must look like a command name (lowercase start; optional ./ for project scripts); + // this drops heredoc/prose lines like "Co-Authored-By:" and "EOF" that leak from fences. + if (!head || !/^(?:\.\/)?[a-z][a-z0-9._/+-]*$/.test(head) || NOISE.has(head)) continue + if (toks[1] && /^[a-z][a-z0-9_-]*$/.test(toks[1])) head += ' ' + toks[1] // keep subcommand + if (heads[heads.length - 1] !== head) heads.push(head) + } + return heads +} + +// Extract executed shell commands + failure signals from one MODERN block. +// Command locations (see patterns.mjs §3): (a) in-summary, (b) inline-backtick line, +// (c) shell-language fence (heredocs skipped), (d) "- command:" bullet - single- OR multi-line. +// Output fences are never commands; they are scanned for exit codes / error heads only. +export function extractShellBlock(summary, body, line) { + const cmds = [] + let fails = 0 + const sm = summary.match(SUMMARY_CMD) // (a) + if (sm) cmds.push({ cmd: sm[1], line }) + let inFence = false, lang = '', fbuf = [], sawOutput = false, bulletOpen = false + for (const raw of body) { + if (bulletOpen) { // (d) multi-line bullet tail + if (BULLET_CMD_CLOSE.test(raw.trim())) bulletOpen = false + continue // bullet content ≠ commands + } + const fm = raw.match(FENCE_LINE) + if (fm) { + if (!inFence) { inFence = true; lang = fm[1].toLowerCase(); fbuf = [] } + else { + if (SHELL_FENCE_LANGS.has(lang)) { // (c) + let skipUntil = null + for (const cl of fbuf) { + const t = cl.trim() + if (skipUntil !== null) { if (t === skipUntil) skipUntil = null; continue } + if (!t || t.startsWith('#')) continue + const hd = t.match(HEREDOC_OPEN) + if (hd) skipUntil = hd[1] + cmds.push({ cmd: t, line }) + } + } else { + sawOutput = true + for (const ol of fbuf) { + const xc = EXIT_CODE.exec(ol) + if (xc && xc[1] !== '0') fails++ + else if (ERROR_HEAD.test(ol.trim())) fails++ + } + } + inFence = false + } + continue + } + if (inFence) { fbuf.push(raw); continue } + if (sawOutput) { + const xc = EXIT_CODE.exec(raw) + if (xc && xc[1] !== '0') fails++ + continue + } + const im = raw.match(INLINE_CMD) // (b) + if (im) { cmds.push({ cmd: im[1], line }); continue } + const lm = raw.match(BULLET_CMD) // (d) single-line + if (lm) { cmds.push({ cmd: stripBullet(lm[1]), line }); continue } + const lo = raw.match(BULLET_CMD_OPEN) // (d) multi-line opener: + if (lo) { cmds.push({ cmd: stripBullet(lo[1]), line }); bulletOpen = true } // first line is the command + } + return { cmds, fails } +} +const stripBullet = (s) => s.replace(/^\[\s*/, '').replace(/\s*\]$/, '') + +// Pull file paths from non-shell tool blocks (Read/Edit backticked relpath; apply_patch headers). +export function extractFiles(summary, body) { + const files = new Set() + const grab = (s) => { + for (const m of s.matchAll(/`((?:\.{0,2}\/)?[\w@./-]+\.[a-z]{1,12})`/g)) files.add(m[1]) + for (const m of s.matchAll(/\*\*(?:Add|Modify|Update|Delete):\s*`([^`]+)`\*\*/g)) files.add(m[1]) + } + grab(summary) + for (const l of body.slice(0, 6)) grab(l) // paths appear in the first lines, not in content fences + return [...files] +} + +// Parse one whole transcript: text in -> { agent, uuid, beats } out. Pure function; unit-test me. +export function parseSessionFile(text) { + const head = text.slice(0, 600) + const hdr = SESSION_HDR.exec(head) + const agent = hdr ? hdr[1].toLowerCase().replace(/\s+/g, '-') : 'unknown' // claude-code, codex-cli, cursor, ... + const uuid = hdr ? hdr[2] : '' // provider session id - detects the same session copied into two corpora + + const lines = text.split('\n') + const eps = [] + let cur = null + let i = 0 + while (i < lines.length) { + const line = lines[i] + + // ---- a user turn opens a new beat ---- + if (USER_MARK.test(line)) { + if (cur) eps.push(cur) + const buf = [] + let j = i + 1 + for (; j < lines.length && buf.length < 30; j++) { + const l = lines[j] + if (TURN_MARK.test(l)) break + if (buf.length > 0 && /^#{1,4}\s/.test(l)) break + buf.push(l) + } + const intent = buf.join('\n').trim() + cur = { startLine: i + 1, intent, tools: {}, nTools: 0, cmds: [], files: new Set(), fails: 0, metas: [] } + if (intent) { + const firstLine = (intent.split('\n').find(x => x.trim()) || '').trim() + cur.firstLine = firstLine + const scan = intent.slice(0, 1200) + for (const m of META) { + const hit = m.re.test(firstLine) ? firstLine : (m.re.test(scan) ? (intent.split('\n').find(x => m.re.test(x)) || firstLine) : null) + if (hit) cur.metas.push({ id: m.id, quote: hit.trim().slice(0, 160), line: i + 1 }) + } + } + i = j + continue + } + + // ---- MODERN era: envelope ---- + const tm = TOOLUSE_OPEN.exec(line) + if (tm && cur) { + const ttype = tm[1] || 'generic', name = tm[2] + cur.tools[ttype] = (cur.tools[ttype] || 0) + 1 + cur.nTools++ + let j = i + 1, summary = '' + const body = [] + while (j < lines.length && !lines[j].includes('')) { + if (lines[j].includes('') && !summary) summary = lines[j] + else body.push(lines[j]) + j++ + } + if ((ttype === 'shell' && !SHELL_EXCLUDE.has(name)) || SHELL_TOOLS.has(name)) { + const { cmds, fails } = extractShellBlock(summary, body, i + 1) + cur.fails += fails + for (const c of cmds) for (const h of headsFrom(c.cmd)) cur.cmds.push({ head: h, raw: c.cmd.slice(0, 160), line: c.line }) + } else if (ttype === 'read' || ttype === 'write') { + for (const f of extractFiles(summary, body)) cur.files.add(f) + } + i = j + 1 + continue + } + + // ---- LEGACY era (~2025): bare "Tool use: **Name**" lines ---- + const lt = LEGACY_TOOL.exec(line) + if (lt && cur) { + const name = lt[1] + const ttype = LEGACY_TYPE[name] || 'generic' + cur.tools[ttype] = (cur.tools[ttype] || 0) + 1 + cur.nTools++ + if (ttype === 'read' || ttype === 'write') { + for (const f of extractFiles(lt[2] || '', [])) cur.files.add(f) + } + if (LEGACY_SHELL_NAMES.has(name)) { + // commands follow as inline-backtick lines or shell fences, until Result:/Output:/next marker + let j = i + 1, inFence = false, lang = '', inResult = false + for (; j < lines.length && j < i + 60; j++) { + const l = lines[j] + if (TURN_MARK.test(l) || LEGACY_TOOL.test(l)) break + const fm = l.match(FENCE_LINE) + if (fm) { inFence = !inFence; if (inFence) lang = fm[1].toLowerCase(); continue } + if (inFence) { + if (!inResult && SHELL_FENCE_LANGS.has(lang)) { + const t = l.trim() + if (t && !t.startsWith('#')) for (const h of headsFrom(t)) cur.cmds.push({ head: h, raw: t.slice(0, 160), line: j + 1 }) + } else { + const xc = EXIT_CODE.exec(l) + if (xc && xc[1] !== '0') cur.fails++ + else if (ERROR_HEAD.test(l.trim())) cur.fails++ + } + continue + } + if (LEGACY_OUTPUT_MARK.test(l)) { inResult = true; continue } + if (!inResult) { + const im = l.trim().match(INLINE_CMD) + if (im) for (const h of headsFrom(im[1])) cur.cmds.push({ head: h, raw: im[1].slice(0, 160), line: j + 1 }) + } + } + i = j + continue + } + i++ + continue + } + i++ + } + if (cur) eps.push(cur) + + // outcome labels from the NEXT beat's opening line - the user's reply is free supervision + for (let k = 0; k < eps.length; k++) eps[k].outcome = classifyOutcome(k + 1 < eps.length ? eps[k + 1].firstLine : null) + + return { agent, uuid, beats: eps } +} + +// Intent signature: leading imperative verb + first salient keyword, e.g. "write:commit". +export function intentSig(intent) { + const words = tokenize(intent || '') + const verb = leadingVerb(words) + if (!verb) return null + const kw = words.find(w => w !== verb && !VERBS.has(w)) + return verb + (kw ? ':' + kw : '') +} + +// Author fallback: transcripts leak the session owner's home dir in commands/tool output +// (/Users// on macOS, /home// on Linux). Most frequent name wins. +export function sniffAuthor(text) { + const counts = {} + for (const m of text.matchAll(/\/(?:Users|home)\/([a-z][a-z0-9_-]{1,30})\//gi)) { + const n = m[1].toLowerCase() + if (n === 'shared' || n === 'runner') continue + counts[n] = (counts[n] || 0) + 1 + } + const best = Object.entries(counts).sort((a, b) => b[1] - a[1])[0] + return best ? best[0] : null +} diff --git a/decisions/scripts/lib/patterns.mjs b/decisions/scripts/lib/patterns.mjs new file mode 100644 index 0000000..f021f0e --- /dev/null +++ b/decisions/scripts/lib/patterns.mjs @@ -0,0 +1,219 @@ +// patterns.mjs - THE FORMAT GRAMMAR. +// +// Every regex below is preceded by a verbatim example of the transcript bytes it matches. +// This file is the single source of truth for what SpecStory transcripts look like, across +// providers and eras; parse.mjs consumes these names and contains no inline format knowledge. +// +// Eras (verified against specstory-cli source + a 1,311-session real corpus): +// MODERN (Markdown v2.x, ~Oct 2025+): tool calls wrapped in HTML envelopes. +// LEGACY (~Jun–Oct 2025): bare "Tool use: **Name**" lines, no envelope. +// Both eras share the session header and the _**User/Agent**_ turn markers. + +// ───────────────────────────── 1. SESSION ENVELOPE ───────────────────────────── + +// +// +// (provider names vary, incl. lowercase) +// Capture 1 = provider name (slugified into the agent tag), capture 2 = session UUID. +export const SESSION_HDR = /