diff --git a/.github/extensions/ai-attribution/README.md b/.github/extensions/ai-attribution/README.md index c661011..bed4746 100644 --- a/.github/extensions/ai-attribution/README.md +++ b/.github/extensions/ai-attribution/README.md @@ -1,16 +1,15 @@ # AI attribution canvas A [GitHub Copilot app](https://docs.github.com/en/copilot/how-tos/github-copilot-app/agent-sessions) -canvas for Data and Attribution Model. It examines what is -needed to measure ROI on AI-assisted development across VS Code, Copilot CLI, -the Copilot app and the cloud agent. Live local measurements show reported -usage and work-link coverage, not monetary ROI or proven value. +canvas connecting people, AI usage, work and cost, with facts, estimates and +missing data clearly separated. It does not treat recorded usage as proof of +savings or business value. It renders [the research](../../../docs/research/ai-telemetry-attribution.md) and [the model spec](../../../docs/research/attribution-model.json), and adds live, read-only coverage from this machine. -![AI attribution canvas outcomes tab with coverage bars and PR-reference splits](../../../docs/media/canvas-outcomes.png) +![Overview mission and data-source-to-Recipe map, followed by recorded usage](../../../docs/media/canvas-outcomes.png) The screenshots in this document are generated by `npm run media:canvas` from synthetic session and trace stores, with store discovery pointed at an empty @@ -19,20 +18,26 @@ screenshots uses canned GitHub and Azure Boards answers, not the network. ## Views -Short summaries first; expand a cell, card or section for the full evidence. +Overview is the executive entry point: a mission statement and one hybrid map +of data sources, usage records, Recipe, delivery and the separate bill comparison. +The question about how much usage can be linked to work sits above the local +measurements, not above the mission. The map describes supported data flow, +not which sources are connected on a particular machine. + +Labels use plain English; exact API fields and settings appear in setup or +technical details. Short summaries first; expand a cell, card or section for the full evidence. Expanders support the keyboard and retain their state while the panel updates. Missing values stay unknown, and source errors remain visible. | Tab | Shows | | --- | --- | -| Outcomes | ROI-readiness metrics, reported credits with/without a PR reference, and the cost, value and baseline inputs still needed | +| Overview | Mission and at-a-glance map: sources → usage → Recipe → work/results, with bill comparison alongside; local measurements follow | | Recipe | One delivery cycle per repository branch, from work item through AI work and commits to PR, review, merge and work-item closure, with optional GitHub and Azure Boards verification | -| Surfaces | Concept × surface matrix with native / partial / missing / unverified fields | -| Intersection | Verified concepts available across all five surfaces, with native vs conditional availability and each source's grain/access caveats | -| Data model | Entity map and confidence tiers; expand for fields, provenance and accounting rules | -| Gaps | Impact counts and expandable gaps, verification status and prior art | -| Live coverage | Source cards, sparklines and attribution bars; expand for token tables and breakdowns | -| Research | Section index with the complete document and sources on demand | +| Data sources | What each tool reports; choose all data types or those shared by every source | +| How it connects | Accounts, usage, work, results and bills, with clear explanations of the strength of each link | +| Missing data | Gaps, why they matter and what to do | +| Local usage | Recorded requests and credits by source, with token counts and breakdowns | +| Research | Short inline findings, limits and the cost-allocation explanation, with supporting source links | ![AI attribution canvas surfaces tab showing the concept by surface matrix](../../../docs/media/canvas-surfaces.png) @@ -46,10 +51,10 @@ Missing values stay unknown, and source errors remain visible. | Source | Default location | | --- | --- | | Copilot CLI / app session store | `$COPILOT_HOME/session-store.db` (default `~/.copilot/`) | -| VS Code span store | `globalStorage/github.copilot-chat/agent-traces.db` for Code and Code - Insiders | -| VS Code Copilot harness span store | `/agent-host/otel/agent-host-traces.db`, written only while `chat.agentHost.otel.dbSpanExporter.enabled` is on (VS Code 1.141+) | +| VS Code · Local harness | `globalStorage/github.copilot-chat/agent-traces.db` for Code and Code - Insiders | +| VS Code · Copilot Agent Host harness | `/agent-host/otel/agent-host-traces.db`, written only while `chat.agentHost.otel.dbSpanExporter.enabled` is on (VS Code 1.141+) | -The Copilot harness store also holds Claude and Codex Agent Host traces; only +The Copilot Agent Host harness store also holds Claude and Codex Agent Host traces; only Copilot runtime traces (spans stamped with `github.copilot.*`) are counted. Its credits are the per-turn `github.copilot.nano_aiu` on chat spans, never added to the root invocation total. @@ -57,7 +62,7 @@ the root invocation total. Stores are opened read-only. If the file is locked, a temporary copy is read and then deleted. Only aggregates are produced (counts, token and credit sums, and model, agent, tool, repository and user names); prompt, response and tool content is -never read. Nothing leaves the machine unless you press **Verify** on the Recipe +never read. Nothing leaves the machine unless you press **Check delivery** or **Read billing** on the Recipe tab, and the two sources are shown side by side rather than summed. ### Reading coverage and token values @@ -78,17 +83,18 @@ The surface matrix describes source capabilities, not everything ingested by this canvas. Organisation reports include daily CLI/app tokens and partial skill, plugin, custom-agent and MCP aggregates; MCP counts connections, not tool calls. Task-completion events are self-reports, not correctness signals. -The **Data model** tab carries the billing reconciliation contract: preserve +The **How it connects** tab carries the billing comparison rules: preserve query scope, compare matching gross AI-credit consumption, and keep discounts, -net billed amounts and T3 allocations separate. Billing is not connected here. -Its **Identity provenance** expander distinguishes native identity/resource +final billed amounts and estimated work shares separate. Billing is read only +when you explicitly select **Read billing**. +Its **Where account details come from** expander distinguishes native identity/resource signals, derived actor/device keys, configured organisational attributes and external enrichment. Source capability does not mean that this local store collected those values. -### Intersection +### Shared-surface filter -The **Intersection** tab derives concepts available across VS Code, CLI, app, +The **Data sources** tab's **Shared by all five** filter derives concepts available across VS Code, CLI, app, cloud agent and organisation data from the model. Every cell must be native, partial or derived; missing and unverified fields are excluded. Native across all means directly emitted, not equivalent grain or comparable values. @@ -116,7 +122,7 @@ VS Code 1.140 can add the signed-in GitHub account to Copilot telemetry as resolves each model call's user from its own `user.name`, else from the latest agent span at or before the call in the same chat session, conversation, parent session or trace. It reports the share of credits attributed to a user -(**User identity**) and a **By user** breakdown. `process.user.name` and +(**Linked to an account**) and a **By recorded account** breakdown. `process.user.name` and `host.name` are resource attributes, which `agent-traces.db` does not store. The CLI / app session store records no user, so its actor coverage is not measured. This user name is the join key to organisation data keyed by @@ -129,13 +135,20 @@ measured. This user name is the join key to organisation data keyed by The **Recipe** tab follows one AI-assisted delivery cycle, from picking up a work item to closing it. A cycle is one repository branch. +Start with **Setup & permissions**, then read **Lifecycle**, **Developer → work**, +**Reading the flow**, typical times, **Compare usage with the bill** and cycle timelines. +The guide explains all seven delivery handoffs and their assumptions, followed +by a separate billing handoff covering account/day/model matching, estimated +work costs and unexplained charges. Developer filters +affect the credit breakdown; lifecycle and timelines retain all work in the window. + | Phase | Steps | Local evidence | Verified evidence | | --- | --- | --- | --- | | Ingredients | Work item, branch | Branch convention, `session_refs` issue, session and span branch | GitHub issue state and type; Azure Boards type, state and dates | | Method | AI work, commits | Sessions, model calls and reported credits per source; recorded commits; VS Code head commits moving | PR commit IDs and dates | | Serve | Pull request, review, merge and close | `session_refs` PR | PR by number or head branch, reviews, merge, closing issues and work-item closure | -![Recipe tab with a verified synthetic cycle: recipe strip, KPIs, convention gauge, ingredients and cycle timelines](../../../docs/media/canvas-recipe.png) +![Recipe overview, developer attribution and cycle timelines using synthetic data](../../../docs/media/canvas-recipe.png) The **recipe strip** draws a band per step: the solid core counts cycles that followed the recipe in order up to that step, the lighter band counts cycles @@ -143,14 +156,46 @@ that have the step in any order, and the hatched band is unknown until verified. Dotted tails show where cycles left the in-order recipe. Each step's evidence bar splits **Verified** (GitHub or Azure Boards API), **Recorded** (an agent-recorded reference, not ownership), **Convention** (branch name) and **Observed** (local -telemetry). Below it are serve KPIs (PRs opened, reviewed, merged and closed, -and median lead times), the branch-convention gauge, ingredients by work-item -type, and one timeline per cycle. Select a cycle to see its step-by-step card. +telemetry). Counts appear once in the strip; detailed credits appear once in +the developer table. Select a cycle to see its step-by-step card. Redundant +KPI cards, ingredient summaries and the duplicated surface matrix are removed. +The rightmost **Compare usage and bill** checkpoint is connected with a dashed +line: it follows delivery visually, but runs independently and is not an eighth +delivery funnel stage or a per-PR invoice. Before verification, review, merge and closure are unknown rather than zero: ![Recipe tab before verification, with unknown steps hatched](../../../docs/media/canvas-recipe-local.png) +### Developer → work + +Choose a **developer**, **source** and perspective: + +- **Credits used:** the **User recorded**, **Owner estimate** and **User unknown** + columns separate captured accounts, ownership-based estimates and unknown users. A measured + consumer is never replaced by the assignee or PR author. +- **Work owned:** all contributors' usage on work associated with that + developer, not their personal consumption. + +![Developer usage broken down by work and attribution evidence, synthetic data only](../../../docs/media/canvas-recipe-developer.png) + +Each work group shows its credit breakdown once, even when it has several +linked items. No-repository usage and cycles outside the display cap remain +in the balance. Missing credit reports stay unknown. This is retained local +history for the selected window, not a complete organization or billing total. + +Verify adds GitHub assignment history and the PR author, plus field-limited +Azure Boards assignment snapshots at first AI activity. Current and historical +assignees are separate roles. A sole historical owner / PR author can support +**inference**, never proof of who consumed AI throughout the cycle. Conflicting +owners remain unattributed; missing or truncated history is shown explicitly. + +Azure Boards IDs are organization-qualified. **Link provider identities** +explicitly maps a Boards account to a GitHub login for this panel only. +Display names, emails and branch prefixes are never used to match people. +Links can also be supplied as `identityLinks` when opening the canvas, or set +through `set_recipe_identity_links`; they are not saved to disk. + ### Branch convention and commit hook The work item is the first token after the last slash of the branch name: @@ -165,7 +210,8 @@ The opt-in [`hooks/commit-msg`](hooks/commit-msg) hook adds `Refs: AB#123` or `Refs: #123` from the branch name when the message does not already reference it, so Azure Boards and GitHub create their own links. It skips merges, detached HEADs, empty messages and other branches, never blocks a commit, and needs Node -on `PATH`. For this repository: +on `PATH`. Preserve existing hooks: copy/chain this hook if a hooksPath is already +configured. Otherwise, for this repository: ```sh git config core.hooksPath .github/extensions/ai-attribution/hooks @@ -176,27 +222,67 @@ Elsewhere, copy `hooks/commit-msg` into the repository's `.git/hooks/`. Set ### Verification -**Verify with GitHub & Azure Boards** (or the `verify_recipe` action) is the only -network step, and runs only when asked: +**Check delivery** (`verify_recipe`) reads delivery metadata only when asked. +**Read billing** is a separate explicit network action. - **GitHub.com** through the signed-in `gh` CLI. Each repository gets a GraphQL query, with values passed as variables, that finds the PR by recorded number or head branch. It reads PR state and dates, review states, commit IDs and dates, - closing issues and issue type. A branch name reused by an older PR does not - match. + closing issues, issue type, author login and assignment events. Assignment + history is bounded to 100 events per issue; incomplete history is not used + to establish a past assignee. A branch name reused by an older PR does not match. - **Azure Boards** through `_apis/wit/workitems` with fields limited to type, - state, project and created/activated/resolved/closed dates. State categories + state, project, assignee and created/activated/resolved/closed dates. Field-limited + `asOf` queries retrieve the assignee at first AI activity, not the current owner + retroactively. Historical queries are capped at 60 cycles. State categories decide closure across processes. The organization comes from the tab, the `azureDevOpsOrg` input or `az devops configure --defaults organization=…`. The token comes from `AZURE_DEVOPS_EXT_PAT` or `az account get-access-token`, stays in memory and is never logged or returned. -Only repository, branch and PR, issue or work-item numbers are sent. Titles, -descriptions, commit messages, comments and logins are never requested. +Only repository, branch and PR, issue or work-item numbers plus first-activity +timestamps are sent. Account/assignment metadata is read; titles, +descriptions, commit messages, comments and email fields are never requested. Inaccessible repositories, a missing `gh` or `az`, or an unconfigured organization show as status chips. Credits stay per source, and reported credits per merged PR are consumption, not cost. +### Setup & permissions + +The Recipe section is the operational checklist for local stores and identity +capture, branch conventions, GitHub repository read access, Azure Boards +Work Items (Read) / project access and the paying account's billing permission. +It explains missing history, inferred owners, explicit provider identity links, +network opt-in and how to preserve existing commit hooks. + +![Recipe setup and permission checklist with invented accounts only](../../../docs/media/canvas-recipe-setup.png) + +### Compare usage with the bill + +Choose **organization**, **enterprise** or **personal**, the paying account, a +GitHub login and one completed **UTC day**. Read billing calls the corresponding +GitHub `settings/billing/ai_credit/usage` endpoint through `gh`, filtered to that +user, day and Copilot product. Repository read access alone does not grant billing +access. Personal reports exclude organization-managed seats. + +![Billing comparison and unallocated balance using a synthetic report](../../../docs/media/canvas-recipe-billing.png) + +Compare gross credits to captured native-user credits per exact model and local +source. Gross, discounts and net quantities/amounts remain distinct. Unknown +consumers, unmatched model names and missing local history are not imputed. +Currency is not supplied by the endpoint; amounts are displayed as returned. + +Billing has no PR/session IDs and accrues independently of merge. Work shares +are **estimated shares**: net amount times captured native work credits divided by +billed gross credits for that model. Unlinked work and uncaptured usage leave an +unallocated remainder. A model with more local credits than billed credits is +flagged and not allocated. Failed reads clear the previous report; nothing is +persisted and no token or raw failure body is returned. + +Harness names are consistent across the canvas, status text and API source +labels. Their internal source IDs remain `traces` (Local harness) and `agentHost` +(Copilot Agent Host harness); database paths and settings are unchanged. + ## Use The app loads project extensions from `.github/extensions/`. Ask Copilot to open @@ -212,10 +298,17 @@ open_canvas({ canvasId: "ai-attribution", instanceId: "attribution", input: { vi | `refresh_coverage` | `{ windowDays? }` | Recomputes coverage; returns a summary | | `get_coverage` | — | Full coverage object | | `get_model` | `{ section? }` | Model spec or one section | -| `get_recipe` | — | Recipe stations, KPIs, convention adherence and latest cycles | +| `get_recipe` | `{ actorKey?, sourceId?, perspective? }` | Recipe and developer credit breakdown; perspective is `usage` or `ownership`; `actorKey: "__unknown"` selects unmatched usage/ownership | | `verify_recipe` | `{ azureDevOpsOrg? }` | Verifies cycles with GitHub and Azure Boards; returns the Recipe summary | +| `set_recipe_identity_links` | `{ links: [{ boardsKey, githubLogin }] }` | Explicit cross-provider account links for this panel | +| `reconcile_recipe_billing` | `{ level, account, user, day }` | Explicit one-day AI-credit billing read and per-source reconciliation; no per-PR invoices | -Open input also accepts `sessionStorePath`, `tracesDbPath` and `agentHostTracesDbPath` overrides, and `azureDevOpsOrg`. +Open input also accepts `sessionStorePath`, `tracesDbPath` and `agentHostTracesDbPath` +overrides, `azureDevOpsOrg`, and `identityLinks: [{ boardsKey, githubLogin }]`. +API view IDs remain `outcomes`, `recipe`, `surfaces`, `model`, `gaps`, `coverage` +and `research`; the display-name changes do not break callers. +Model schema 2 exposes descriptive `linkMethods`; estimated-cost rows use +`method: "estimated-share"`. The former research-only code system is removed. ## Layout @@ -224,17 +317,22 @@ Open input also accepts `sessionStorePath`, `tracesDbPath` and `agentHostTracesD | `extension.mjs` | Canvas declaration, actions and per-panel state | | `lib/coverage.mjs` | Store discovery and coverage computation | | `lib/recipe.mjs` | Recipe cycles, steps, evidence and KPIs (pure) | +| `lib/actors.mjs` | Provider-qualified identities, role attribution and per-source developer balances (pure) | +| `lib/billing.mjs` | Explicit GitHub billing reader, scope validation, per-source comparison and estimated work shares | | `lib/convention.mjs` | Branch-name work-item convention | | `lib/verify.mjs` | On-demand GitHub and Azure Boards verification | | `hooks/commit-msg` | Opt-in hook adding the branch's work item to commit messages | | `lib/tokenCoverage.cjs`, `lib/tokenCoverage.d.cts` | Shared paired-count and reporting-coverage logic; the declaration supplies the VS Code extension's type contract | | `lib/server.mjs` | Loopback HTTP server, JSON API and server-sent events | | `lib/model.mjs` | Loads the committed research files | +| `ui/format.mjs` | Shared display formatting and harness labels used by the canvas, source reports and status text | | `ui/` | Static front end (no build step, no dependencies) | After editing, reload extensions in the app. Tests live in [`test/attributionCanvas.test.js`](../../../test/attributionCanvas.test.js) and -[`test/recipeCanvas.test.js`](../../../test/recipeCanvas.test.js). +[`test/recipeCanvas.test.js`](../../../test/recipeCanvas.test.js) and +[`test/recipeActors.test.js`](../../../test/recipeActors.test.js) and +[`test/recipeBilling.test.js`](../../../test/recipeBilling.test.js). ## Reuse diff --git a/.github/extensions/ai-attribution/extension.mjs b/.github/extensions/ai-attribution/extension.mjs index f2a7df7..b808b9c 100644 --- a/.github/extensions/ai-attribution/extension.mjs +++ b/.github/extensions/ai-attribution/extension.mjs @@ -10,6 +10,9 @@ import { EXTENSION_DIR, MODEL_SECTIONS, loadModel, loadResearch } from './lib/mo import { recipeView } from './lib/recipe.mjs'; import { startCanvasServer } from './lib/server.mjs'; import { verifyRecipe } from './lib/verify.mjs'; +import { validateIdentityLinks } from './lib/actors.mjs'; +import { fetchBilling, billingView } from './lib/billing.mjs'; +import { sourceLabels as SOURCE_NAMES } from './ui/format.mjs'; const VIEWS = ['outcomes', 'recipe', 'surfaces', 'intersection', 'model', 'gaps', 'coverage', 'research']; const DEFAULT_WINDOW_DAYS = 30; @@ -62,7 +65,6 @@ function summarize(coverage) { }; } -const SOURCE_NAMES = { sessionStore: 'CLI/app', traces: 'VS Code', agentHost: 'VS Code Copilot harness' }; function statusLine(coverage) { const window = coverage.window.days ? `${coverage.window.days}-day window` : 'All history'; @@ -84,7 +86,30 @@ function instanceFor(ctx) { } async function recipeFor(instance) { - return recipeView((await coverageFor(instance.options)).recipe, instance.verification); + const recipe = (await coverageFor(instance.options)).recipe; + return { + ...recipeView(recipe, instance.verification, { identityLinks: instance.identityLinks }), + billing: billingView(instance.billing, recipe), + }; +} + +async function readBilling(instance, scope) { + if (instance.readingBilling) throw new CanvasError('busy', 'A billing request is already running.'); + instance.readingBilling = true; + instance.billing = null; + try { + instance.billing = await fetchBilling(scope); + instance.server.broadcast('recipe', {}); + return recipeFor(instance); + } finally { + instance.readingBilling = false; + } +} + +async function setIdentityLinks(instance, links) { + instance.identityLinks = validateIdentityLinks(links); + instance.server.broadcast('recipe', {}); + return recipeFor(instance); } /** Explicit network step: asks GitHub and Azure Boards about the cycles already found locally. */ @@ -103,8 +128,26 @@ async function verifyInstance(instance, organization) { return recipeFor(instance); } -function summarizeRecipe(view) { +function summarizeRecipe(view, filter = {}) { + const developerSources = view.developers.sources + .filter((source) => !filter.sourceId || source.id === filter.sourceId) + .map((source) => { + const owned = filter.perspective === 'ownership'; + const selected = filter.actorKey === '__unknown' ? source[owned ? 'unowned' : 'unknown'] : + filter.actorKey ? source.actors.find((actor) => actor.key === filter.actorKey)?.[owned ? 'ownership' : 'usage'] : + owned ? source.ownership : source; + return { id: source.id, summary: selected ? { + total: selected.total, measured: selected.measured, inferred: selected.inferred, unattributed: selected.unattributed, + work: selected.work.map((row) => { + const cycle = view.cycles.find((c) => c.key === row.cycleKey); + return { ...row, repository: cycle?.repository ?? null, branch: cycle?.branch ?? null, + workItems: cycle?.workItems.map((w) => w.reference) ?? [], pullRequest: cycle?.pullRequest?.number ?? null }; + }), + } : null }; + }); return { + billing: view.billing, + developers: { actors: view.developers.actors, identityLinks: view.developers.identityLinks, sources: developerSources, perspective: filter.perspective ?? 'usage' }, verification: view.verification, totals: view.totals, stations: view.stations.map(({ id, any, strict, unknown }) => ({ id, any, strict, unknown })), @@ -114,6 +157,7 @@ function summarizeRecipe(view) { repository: c.repository, branch: c.branch, workItems: c.workItems.map((w) => w.reference), + ownership: c.ownership, pullRequest: c.pullRequest?.number ?? null, reached: c.reached, steps: Object.fromEntries(Object.entries(c.steps).map(([id, s]) => [id, s.state])), @@ -130,6 +174,9 @@ async function openInstance(ctx) { options: { windowDays: DEFAULT_WINDOW_DAYS }, verification: null, verifying: null, + identityLinks: [], + billing: null, + readingBilling: false, }; instance.server = await startCanvasServer({ uiDir: UI_DIR, @@ -145,6 +192,8 @@ async function openInstance(ctx) { }, recipe: () => recipeFor(instance), verify: ({ azureDevOpsOrg } = {}) => verifyInstance(instance, azureDevOpsOrg), + identityLinks: (links) => setIdentityLinks(instance, links), + billing: (scope) => readBilling(instance, scope), getView: () => instance.view, setView: (view) => { if (VIEWS.includes(view)) instance.view = view; @@ -163,6 +212,7 @@ async function openInstance(ctx) { if (input.tracesDbPath !== undefined) options.tracesDbPath = input.tracesDbPath || undefined; if (input.agentHostTracesDbPath !== undefined) options.agentHostTracesDbPath = input.agentHostTracesDbPath || undefined; if (input.azureDevOpsOrg !== undefined) options.azureDevOpsOrg = input.azureDevOpsOrg || undefined; + if (input.identityLinks !== undefined) instance.identityLinks = validateIdentityLinks(input.identityLinks); instance.options = options; // Every open recomputes, so a reopened panel never shows a stale snapshot; @@ -200,13 +250,21 @@ await joinSession({ tracesDbPath: { type: 'string', description: "Override for VS Code's agent-traces.db." }, agentHostTracesDbPath: { type: 'string', - description: "Override for VS Code's Copilot-harness agent-host-traces.db.", + description: "Override for VS Code's Copilot Agent Host harness agent-host-traces.db.", }, azureDevOpsOrg: { type: 'string', description: 'Azure DevOps organization for Recipe verification (name, https://dev.azure.com/ or https://.visualstudio.com). Defaults to az devops configure.', }, + identityLinks: { + type: 'array', maxItems: 32, + description: 'Explicit Azure Boards account-key to GitHub-login links. No display-name matching.', + items: { + type: 'object', additionalProperties: false, required: ['boardsKey', 'githubLogin'], + properties: { boardsKey: { type: 'string' }, githubLogin: { type: 'string' } }, + }, + }, }, }, actions: [ @@ -250,11 +308,36 @@ await joinSession({ inputSchema: { type: 'object', additionalProperties: false, properties: {} }, handler: async (ctx) => withoutRecipe(await coverageFor(instanceFor(ctx).options)), }, + { + name: 'reconcile_recipe_billing', + description: 'Explicitly read one completed UTC day of GitHub Copilot AI-credit billing for a user and paying account. Compare native captured consumption per source; work amounts are estimated allocations, never PR invoices.', + inputSchema: { + type: 'object', additionalProperties: false, required: ['level', 'account', 'user', 'day'], + properties: { + level: { type: 'string', enum: ['organization', 'enterprise', 'user'] }, + account: { type: 'string' }, user: { type: 'string' }, + day: { type: 'string', pattern: '^\\d{4}-\\d{2}-\\d{2}$' }, + }, + }, + handler: async (ctx) => summarizeRecipe(await readBilling(instanceFor(ctx), ctx.input)), + }, { name: 'get_recipe', - description: 'Return the Recipe summary: cycle stations, KPIs, branch-convention adherence and the latest cycles.', - inputSchema: { type: 'object', additionalProperties: false, properties: {} }, - handler: async (ctx) => summarizeRecipe(await recipeFor(instanceFor(ctx))), + description: 'Return Recipe and developer credit attribution. Filter by actorKey/sourceId; usage separates measured from inferred, ownership shows all contributors to owned work.', + inputSchema: { type: 'object', additionalProperties: false, properties: { + actorKey: { type: 'string' }, sourceId: { type: 'string', enum: ['sessionStore', 'traces', 'agentHost'] }, + perspective: { type: 'string', enum: ['usage', 'ownership'] }, + } }, + handler: async (ctx) => summarizeRecipe(await recipeFor(instanceFor(ctx)), ctx.input ?? {}), + }, + { + name: 'set_recipe_identity_links', + description: 'Set explicit Azure Boards account-key to GitHub-login links for this panel. Never infer them from names or emails.', + inputSchema: { type: 'object', additionalProperties: false, required: ['links'], properties: { + links: { type: 'array', maxItems: 32, items: { type: 'object', additionalProperties: false, required: ['boardsKey', 'githubLogin'], + properties: { boardsKey: { type: 'string' }, githubLogin: { type: 'string' } } } }, + } }, + handler: async (ctx) => summarizeRecipe(await setIdentityLinks(instanceFor(ctx), ctx.input.links)), }, { name: 'verify_recipe', diff --git a/.github/extensions/ai-attribution/lib/actors.mjs b/.github/extensions/ai-attribution/lib/actors.mjs new file mode 100644 index 0000000..7cf2581 --- /dev/null +++ b/.github/extensions/ai-attribution/lib/actors.mjs @@ -0,0 +1,139 @@ +// Provider-qualified identities. Device names, branch prefixes and display names +// never become account keys; cross-provider links must be supplied explicitly. +export function githubActor(login, kind = 'User') { + if (typeof login !== 'string' || !/^[a-z\d](?:[a-z\d-]{0,37}[a-z\d])?(?:\[bot\])?$/i.test(login)) return null; + const bot = kind === 'Bot' || login.endsWith('[bot]'); + return { key: `github:${login.toLowerCase()}`, label: login, provider: 'github', bot }; +} + +export function telemetryActor(value) { + if (typeof value !== 'string') return null; + const name = value.trim(); + if (!name || name.length > 128 || /[\u0000-\u001f\u007f]/.test(name)) return null; + return githubActor(name) ?? { key: `telemetry:${name}`, label: name, provider: 'telemetry', bot: false }; +} + +export function boardsActor(value, organization) { + if (!value || typeof value.id !== 'string' || !/^[a-f\d-]{8,64}$/i.test(value.id)) return null; + const label = typeof value.displayName === 'string' && value.displayName.length <= 128 && !/[\u0000-\u001f\u007f]/.test(value.displayName) + ? value.displayName : value.id; + return { key: `boards:${organization.toLowerCase()}:${value.id.toLowerCase()}`, label, provider: 'azure-boards', bot: false }; +} + +export function validateIdentityLinks(links) { + if (!Array.isArray(links) || links.length > 32) throw new Error('Provide at most 32 identity links.'); + const seen = new Set(); + return links.map((link) => { + if (!link || typeof link.boardsKey !== 'string' || + !/^boards:https:\/\/(?:dev\.azure\.com\/[a-z\d-]+|[a-z\d-]+\.visualstudio\.com):[a-f\d-]{8,64}$/i.test(link.boardsKey) || + !githubActor(link.githubLogin) || seen.has(link.boardsKey.toLowerCase())) { + throw new Error('Each identity link needs a unique Azure Boards account key and a GitHub login.'); + } + const boardsKey = link.boardsKey.toLowerCase(); + seen.add(boardsKey); + return { boardsKey, githubLogin: link.githubLogin }; + }); +} + +function resolve(actor, links) { + if (!actor) return null; + const link = links.find((l) => l.boardsKey === actor.key); + return link ? { ...githubActor(link.githubLogin), linkedFrom: actor.key } : actor; +} + +/** Current assignment is displayed as a role, never backdated to infer usage. */ +export function ownershipFor(workItems, pr, links) { + const roles = []; + if (pr?.author) roles.push({ role: 'PR author', actor: resolve(pr.author, links), identity: pr.author, at: pr.createdAt }); + for (const item of workItems) { + for (const [field, role, at] of [ + ['assignees', 'Assigned now', item.assignmentCheckedAt], + ['assigneesAtStart', 'Assigned at pickup', item.assignmentAt], + ]) { + for (const actor of item[field] ?? []) roles.push({ role, actor: resolve(actor, links), identity: actor, at, reference: item.reference }); + } + } + const historical = roles.filter((r) => r.role !== 'Assigned now' && !r.actor.bot); + const candidates = [...new Map(historical.map((r) => [r.actor.key, r.actor])).values()]; + return { + roles, + inferred: candidates.length === 1 ? candidates[0] : null, + inference: candidates.length === 1 + ? [...new Set(historical.map((r) => r.role))].join(' + ') + : candidates.length > 1 ? 'Conflicting owners' : 'No historical owner', + }; +} + +function newTally() { + return { calls: 0, creditedCalls: 0, credits: 0 }; +} + +function add(tally, usage) { + tally.calls += usage.calls; + tally.creditedCalls += usage.creditedCalls; + tally.credits += usage.credits ?? 0; +} + +function reported(tally) { + return { ...tally, credits: tally.creditedCalls ? tally.credits : null }; +} + +/** Partitions every source into disjoint measured, inferred and unattributed usage. */ +export function attributeUsage(sources, ownership, links) { + return sources.flatMap((source) => { + const actors = source.actors ?? [{ actor: null, calls: source.calls, creditedCalls: source.creditedCalls, credits: source.credits }]; + return actors.map((usage) => ({ + source: source.id, + ...reported(usage), + actor: resolve(usage.actor, links) ?? ownership?.inferred ?? null, + method: usage.actor ? 'measured' : ownership?.inferred ? 'inferred' : 'unattributed', + evidence: usage.actor ? 'user.name' : ownership?.inference ?? 'No user identity', + })); + }); +} + +export function developerReport(cycles, unmatched, identityLinks) { + const actors = new Map(); + const entries = [ + ...cycles.flatMap((cycle) => cycle.attribution.map((entry) => ({ ...entry, cycleKey: cycle.key }))), + ...attributeUsage(unmatched, null, identityLinks).map((entry) => ({ ...entry, cycleKey: null })), + ]; + for (const entry of entries) if (entry.actor) actors.set(entry.actor.key, entry.actor); + // Owners without attributed usage remain selectable as work owners, not consumers. + for (const cycle of cycles) for (const role of cycle.ownership.roles) actors.set(role.actor.key, role.actor); + const summarize = (list, withWork = true) => { + const tally = { total: newTally(), measured: newTally(), inferred: newTally(), unattributed: newTally() }; + const work = new Map(); + for (const entry of list) { + add(tally.total, entry); + add(tally[entry.method], entry); + const group = work.get(entry.cycleKey) ?? []; + group.push(entry); + work.set(entry.cycleKey, group); + } + return { + ...Object.fromEntries(Object.entries(tally).map(([key, value]) => [key, reported(value)])), + ...(withWork ? { work: [...work.entries()].map(([cycleKey, group]) => ({ cycleKey, ...summarize(group, false) })) + .sort((a, b) => (b.total.credits ?? -1) - (a.total.credits ?? -1)) } : {}), + }; + }; + const sourceIds = [...new Set(entries.map((entry) => entry.source))]; + const ownedCycles = new Set(cycles.filter((cycle) => cycle.ownership.roles.length).map((cycle) => cycle.key)); + return { + sources: sourceIds.map((id) => { + const list = entries.filter((entry) => entry.source === id); + return { + id, ...summarize(list), + unknown: summarize(list.filter((entry) => !entry.actor)), + ownership: summarize(list.filter((entry) => ownedCycles.has(entry.cycleKey))), + unowned: summarize(list.filter((entry) => !ownedCycles.has(entry.cycleKey))), + actors: [...actors.keys()].map((key) => { + const owned = new Set(cycles.filter((cycle) => cycle.ownership.roles.some((r) => r.actor.key === key)).map((cycle) => cycle.key)); + return { key, usage: summarize(list.filter((entry) => entry.actor?.key === key)), ownership: summarize(list.filter((entry) => owned.has(entry.cycleKey))) }; + }), + }; + }), + actors: [...actors.values()].sort((a, b) => a.label.localeCompare(b.label)), + identityLinks, + }; +} diff --git a/.github/extensions/ai-attribution/lib/billing.mjs b/.github/extensions/ai-attribution/lib/billing.mjs new file mode 100644 index 0000000..d5cef5b --- /dev/null +++ b/.github/extensions/ai-attribution/lib/billing.mjs @@ -0,0 +1,155 @@ +// Explicit GitHub billing reads. Usage reconciliation is by native consumer, +// UTC day and exact model, per source. Work-level cost is an estimated share. +import { runCommand } from './verify.mjs'; +import { githubActor } from './actors.mjs'; + +const AMOUNTS = ['grossQuantity', 'grossAmount', 'discountQuantity', 'discountAmount', 'netQuantity', 'netAmount']; +const SOURCE_IDS = ['sessionStore', 'traces', 'agentHost']; + +function utcDay(value) { + return typeof value === 'string' && /^\d{4}-\d{2}-\d{2}$/.test(value) && + Number.isFinite(Date.parse(value)) && new Date(value).toISOString().slice(0, 10) === value; +} + +export function billingScope(input, now = Date.now()) { + const { level, account, user, day } = input ?? {}; + if (!['organization', 'enterprise', 'user'].includes(level) || + typeof account !== 'string' || !/^[a-z\d][a-z\d-]{0,99}$/i.test(account) || + !githubActor(user) || !utcDay(day)) { + throw new Error('Choose a billing level, account, GitHub login and valid UTC day.'); + } + if (day >= new Date(now).toISOString().slice(0, 10)) throw new Error('Choose a completed UTC day, not today or a future date.'); + if (level === 'user' && account.toLowerCase() !== user.toLowerCase()) throw new Error('Personal billing can only report the selected account itself.'); + return { level, account: account.toLowerCase(), user: user.toLowerCase(), day, product: 'Copilot' }; +} + +function finiteAmount(value) { + return typeof value === 'number' && Number.isFinite(value) && value >= 0; +} + +function safeWord(value) { + return typeof value === 'string' && value.length > 0 && value.length <= 128 && !/[\u0000-\u001f\u007f]/.test(value); +} + +export function parseBillingReport(body, scope, retrievedAt) { + const [year, month, day] = scope.day.split('-').map(Number); + if (body?.timePeriod?.year !== year || body.timePeriod.month !== month || body.timePeriod.day !== day || + !Array.isArray(body.usageItems) || body.usageItems.length > 1000) throw new Error('GitHub returned a different billing period or an invalid report.'); + const accountField = scope.level === 'organization' ? 'organization' : scope.level === 'enterprise' ? 'enterprise' : 'user'; + for (const [field, expected] of [[accountField, scope.account], ['user', scope.user], ['product', scope.product]]) { + if (body[field] !== undefined && (typeof body[field] !== 'string' || body[field].toLowerCase() !== expected.toLowerCase())) { + throw new Error('GitHub returned a different billing account, user or product.'); + } + } + const items = body.usageItems.map((row) => { + if (!safeWord(row.product) || row.product.toLowerCase() !== 'copilot' || !safeWord(row.sku) || + !safeWord(row.model) || !safeWord(row.unitType) || + !/^(ai[\s_-]?)?credits?$/i.test(row.unitType) || + !finiteAmount(row.pricePerUnit) || !AMOUNTS.every((key) => finiteAmount(row[key]))) { + throw new Error('Unsupported billing units or missing amounts. No comparison was made.'); + } + return { + product: row.product, sku: row.sku, model: row.model, unitType: row.unitType, pricePerUnit: row.pricePerUnit, + ...Object.fromEntries(AMOUNTS.map((key) => [key, row[key]])), + }; + }); + return { status: 'ok', scope, retrievedAt, items }; +} + +export async function fetchBilling(input, { run = runCommand, now = Date.now(), env = process.env } = {}) { + const scope = billingScope(input, now); + const prefix = { organization: 'organizations', enterprise: 'enterprises', user: 'users' }[scope.level]; + const endpoint = `/${prefix}/${scope.account}/settings/billing/ai_credit/usage`; + const [year, month, day] = scope.day.split('-').map(Number); + const args = ['api', '--hostname', 'github.com', '--method', 'GET', endpoint, + '-H', 'Accept: application/vnd.github+json', '-H', 'X-GitHub-Api-Version: 2026-03-10', + '-f', `year=${year}`, '-f', `month=${month}`, '-f', `day=${day}`, '-f', 'product=Copilot']; + if (scope.level !== 'user') args.push('-f', `user=${scope.user}`); + const retrievedAt = new Date(now).toISOString(); + try { + const response = await run('gh', args, { env, timeout: 30_000 }); + if (response.missing) return { status: 'error', scope, retrievedAt, reason: 'Install gh and sign in with access to the paying account.' }; + if (response.code !== 0) { + const denied = /401|403|404|auth login|bad credentials/i.test(response.stderr); + return { status: 'error', scope, retrievedAt, + reason: denied ? 'Billing access denied. Use a supported credential with billing permissions on the paying account.' : 'GitHub billing request failed or timed out. Retry explicitly.' }; + } + let body; + try { body = JSON.parse(response.stdout); } catch { throw new Error('GitHub returned invalid billing JSON.'); } + return parseBillingReport(body, scope, retrievedAt); + } catch (err) { + return { status: 'error', scope, retrievedAt, + reason: err instanceof Error && /^(GitHub returned|Unsupported billing)/.test(err.message) ? err.message : 'Billing request failed. No billing data was retained.' }; + } +} + +function sum(rows, field) { + return rows.reduce((value, row) => value + row[field], 0); +} + +/** A billing balance is not proof that the selected account paid for local calls. */ +export function reconcileBilling(report, recipe, sourceId) { + if (report?.status !== 'ok') return null; + if (!SOURCE_IDS.includes(sourceId)) throw new Error('Choose a supported local usage source.'); + const scope = report.scope; + const rows = (recipe.billingUsage ?? []).filter((row) => row.source === sourceId && row.day === scope.day); + const userRows = rows.filter((row) => row.actor?.key === `github:${scope.user}`); + const modelRows = new Map(); + for (const item of report.items) { + const key = item.model.toLowerCase(); + const group = modelRows.get(key) ?? { model: item.model, items: [] }; + group.items.push(item); + modelRows.set(key, group); + } + const allocations = new Map(); + let matched = 0; + let observed = 0; + let reportingCalls = 0; + let modelDifference = false; + const models = [...modelRows.entries()].map(([model, group]) => { + const usage = userRows.filter((row) => row.model?.toLowerCase() === model); + const credited = usage.filter((row) => row.creditedCalls > 0); + const local = sum(credited, 'credits'); + const gross = sum(group.items, 'grossQuantity'); + const netAmount = sum(group.items, 'netAmount'); + observed += local; + reportingCalls += sum(usage, 'creditedCalls'); + matched += Math.min(local, gross); + if (local > gross) modelDifference = true; + // Never distribute more than the bill. No allocation for model mismatches, + // unknown consumers or unlinked work; those amounts remain unallocated. + if (gross > 0 && local <= gross) { + for (const row of credited.filter((row) => row.cycleKey)) { + const allocation = allocations.get(row.cycleKey) ?? { cycleKey: row.cycleKey, grossQuantity: 0, netAmount: 0, method: 'estimated-share' }; + allocation.grossQuantity += row.credits; + allocation.netAmount += netAmount * row.credits / gross; + allocations.set(row.cycleKey, allocation); + } + } + return { + model: group.model, grossQuantity: gross, observed: credited.length ? local : null, + difference: credited.length ? gross - local : null, + }; + }); + const unseenModels = userRows.filter((row) => !row.model || !modelRows.has(row.model.toLowerCase())); + const totals = Object.fromEntries(AMOUNTS.map((field) => [field, sum(report.items, field)])); + const work = [...allocations.values()].sort((a, b) => b.grossQuantity - a.grossQuantity); + return { + sourceId, ...totals, models, + observed: reportingCalls ? observed : null, + difference: reportingCalls ? totals.grossQuantity - observed : null, + unmatchedBilling: Math.max(0, totals.grossQuantity - matched), + outsideModelScope: sum(unseenModels, 'credits'), + unknownIdentity: sum(rows.filter((row) => !row.actor), 'credits'), + localExceedsBilling: modelDifference, + work, + unallocatedNetAmount: Math.max(0, totals.netAmount - sum(work, 'netAmount')), + }; +} + +export function billingView(report, recipe) { + if (!report) return { status: 'disconnected' }; + return { ...report, + sources: report.status === 'ok' ? SOURCE_IDS.map((id) => reconcileBilling(report, recipe, id)) : [], + }; +} diff --git a/.github/extensions/ai-attribution/lib/coverage.mjs b/.github/extensions/ai-attribution/lib/coverage.mjs index ce1011b..3ac454b 100644 --- a/.github/extensions/ai-attribution/lib/coverage.mjs +++ b/.github/extensions/ai-attribution/lib/coverage.mjs @@ -2,8 +2,8 @@ // // Sources: // - Copilot CLI / GitHub Copilot app session store ($COPILOT_HOME/session-store.db) -// - VS Code Copilot Chat span store (globalStorage/github.copilot-chat/agent-traces.db) -// - VS Code Agent Host span store for the Copilot harness (/agent-host/otel/agent-host-traces.db) +// - VS Code Local harness span store (globalStorage/github.copilot-chat/agent-traces.db) +// - Copilot Agent Host harness span store (/agent-host/otel/agent-host-traces.db) // // Only aggregates leave this module: counts, token and credit sums, model, // agent and repository names. Prompt, response and tool content are never read. @@ -15,6 +15,7 @@ import os from 'node:os'; import path from 'node:path'; import { TokenTally } from './tokenCoverage.cjs'; import { CycleCollector, buildRecipe } from './recipe.mjs'; +import { sourceLabels } from '../ui/format.mjs'; export const NANO_AIU_PER_CREDIT = 1_000_000_000; const DAY_MS = 86_400_000; @@ -398,6 +399,7 @@ export function sessionStoreCoverage(db, { sinceDay = null, sinceMs = 0, cycles repository: sessionId ? repository : null, branch: session?.branch, session: sessionId, + model: row.model, at: t.ms ?? (t.dateOnly ? Date.parse(`${t.day}T00:00:00Z`) : null), nano, credited: hasCredit, @@ -743,6 +745,8 @@ export function tracesCoverage(db, { sinceMs = 0, copilotRuntime = false, cycles branch: ctx?.branch, commit: ctx?.commit, session: stageKey, + user, + model: chat.response_model || chat.request_model, at, nano, credited: hasCredit, @@ -840,7 +844,7 @@ export function tracesCoverage(db, { sinceMs = 0, copilotRuntime = false, cycles hasAttributes ? null : 'span_attributes not found: credits and repository context are unavailable.', hasAttributes && actor.calls === 0 ? copilotRuntime - ? 'No user.name on agent spans: Copilot-harness identity needs the managed telemetry.capture.identity setting (VS Code 1.141+).' + ? 'No user.name on agent spans: the Copilot Agent Host harness needs the managed telemetry.capture.identity setting (VS Code 1.141+).' : 'No user.name on agent spans: turn on github.copilot.chat.otel.captureIdentity (VS Code 1.140+, Local harness) to attribute calls to a GitHub account.' : null, actor.calls > 0 @@ -917,7 +921,7 @@ export async function computeCoverage(options = {}) { ? await readDatabase(agentHostPath, (db) => tracesCoverage(db, { sinceMs, copilotRuntime: true, cycles })) : { status: 'missing', - reason: 'No agent-host-traces.db. VS Code 1.141+ stores Copilot-harness traces locally only while chat.agentHost.otel.dbSpanExporter.enabled is on.', + reason: 'No agent-host-traces.db. VS Code 1.141+ stores Copilot Agent Host harness traces locally only while chat.agentHost.otel.dbSpanExporter.enabled is on.', }; return { @@ -926,15 +930,15 @@ export async function computeCoverage(options = {}) { sources: [ { id: 'sessionStore', - label: 'Copilot CLI / app session store', + label: sourceLabels.sessionStore, surfaces: ['cli', 'app'], path: storePath, ...store, }, - { id: 'traces', label: 'VS Code agent-traces.db', surfaces: ['vscode'], path: tracesPath ?? null, ...traces }, + { id: 'traces', label: sourceLabels.traces, surfaces: ['vscode'], path: tracesPath ?? null, ...traces }, { id: 'agentHost', - label: 'VS Code Copilot harness agent-host-traces.db', + label: sourceLabels.agentHost, surfaces: ['vscode'], path: agentHostPath ?? null, ...agentHost, diff --git a/.github/extensions/ai-attribution/lib/model.mjs b/.github/extensions/ai-attribution/lib/model.mjs index ffd0d0b..d5612f0 100644 --- a/.github/extensions/ai-attribution/lib/model.mjs +++ b/.github/extensions/ai-attribution/lib/model.mjs @@ -14,7 +14,7 @@ export const MODEL_SECTIONS = [ 'concepts', 'entities', 'relationships', - 'tiers', + 'linkMethods', 'invariants', 'reconciliation', 'recipe', diff --git a/.github/extensions/ai-attribution/lib/recipe.mjs b/.github/extensions/ai-attribution/lib/recipe.mjs index b36cab9..43a2aa4 100644 --- a/.github/extensions/ai-attribution/lib/recipe.mjs +++ b/.github/extensions/ai-attribution/lib/recipe.mjs @@ -9,6 +9,7 @@ // Credits stay per source and are never summed across sources. import { branchName, parseBranchWorkItem } from './convention.mjs'; +import { telemetryActor, ownershipFor, attributeUsage, developerReport, validateIdentityLinks } from './actors.mjs'; export const RECIPE_STEPS = ['workItem', 'branch', 'aiWork', 'commits', 'pullRequest', 'review', 'close']; const STEP_PHASE = { @@ -66,6 +67,29 @@ function earliest(map, key, at) { if (known === undefined || (Number.isFinite(at) && (!Number.isFinite(known) || at < known))) map.set(key, at); } +function usageTally() { + return { calls: 0, creditedCalls: 0, nano: 0, actors: new Map() }; +} + +function addUsage(tally, actor, nano, credited) { + tally.calls += 1; + tally.creditedCalls += credited ? 1 : 0; + tally.nano += credited ? nano : 0; + const key = actor?.key ?? ''; + const part = tally.actors.get(key) ?? { actor, calls: 0, creditedCalls: 0, nano: 0 }; + part.calls += 1; + part.creditedCalls += credited ? 1 : 0; + part.nano += credited ? nano : 0; + tally.actors.set(key, part); +} + +function usageView(tally) { + return { + calls: tally.calls, creditedCalls: tally.creditedCalls, credits: credits(tally.nano), + actors: [...tally.actors.values()].map(({ nano, ...actor }) => ({ ...actor, credits: actor.creditedCalls ? credits(nano) : null })), + }; +} + // ------------------------------------------------------------- collector --- /** Accumulates local evidence per (repository, branch) while coverage scans its stores. */ @@ -74,6 +98,7 @@ export class CycleCollector { this.cycles = new Map(); this.unlinked = new Map(); this.droppedRefs = 0; + this.billingUsage = new Map(); } #key(repository, branch) { @@ -102,20 +127,26 @@ export class CycleCollector { } /** One model call. Calls without a repository are tallied as unlinked AI work. */ - addCall(source, { repository, branch, commit, session, at, nano = 0, credited = false }) { + addCall(source, { repository, branch, commit, session, at, user, model, nano = 0, credited = false }) { + const actor = telemetryActor(user); + const day = Number.isFinite(at) && at > 0 ? iso(at).slice(0, 10) : null; + const modelName = typeof model === 'string' && model.length <= 128 && !/[\u0000-\u001f\u007f]/.test(model) ? model.trim() || null : null; + const cycleKey = repository ? `${repository}#${branchName(branch) ?? ''}` : null; + const key = JSON.stringify([source, actor?.key ?? null, day, modelName, cycleKey]); + const billing = this.billingUsage.get(key) ?? { source, actor, day, model: modelName, cycleKey, calls: 0, creditedCalls: 0, nano: 0 }; + billing.calls += 1; + billing.creditedCalls += credited ? 1 : 0; + billing.nano += credited ? nano : 0; + this.billingUsage.set(key, billing); if (!repository) { - const u = this.unlinked.get(source) ?? { calls: 0, creditedCalls: 0, nano: 0 }; - u.calls += 1; - u.creditedCalls += credited ? 1 : 0; - u.nano += nano; + const u = this.unlinked.get(source) ?? usageTally(); + addUsage(u, actor, nano, credited); this.unlinked.set(source, u); return; } const cycle = this.#cycle(repository, branchName(branch)); - const s = cycle.sources.get(source) ?? { sessions: new Set(), calls: 0, creditedCalls: 0, nano: 0 }; - s.calls += 1; - s.creditedCalls += credited ? 1 : 0; - s.nano += nano; + const s = cycle.sources.get(source) ?? { ...usageTally(), sessions: new Set() }; + addUsage(s, actor, nano, credited); if (session) s.sessions.add(String(session)); cycle.sources.set(source, s); if (Number.isFinite(at) && at > 0) { @@ -164,13 +195,13 @@ export function buildRecipe(collector, { generatedAt, window } = {}) { return { generatedAt, window, + billingUsage: [...collector.billingUsage.values()].map(({ nano, ...row }) => ({ ...row, credits: credits(nano) })), truncated: Math.max(0, all.length - MAX_CYCLES), droppedRefs: collector.droppedRefs, + omitted: all.slice(MAX_CYCLES).flatMap((cycle) => [...cycle.sources.entries()].map(([id, s]) => ({ id, ...usageView(s) }))), unlinked: [...collector.unlinked.entries()].map(([id, u]) => ({ id, - calls: u.calls, - creditedCalls: u.creditedCalls, - credits: credits(u.nano), + ...usageView(u), })), cycles: all.slice(0, MAX_CYCLES).map((c) => ({ key: `${c.repository}#${c.branch ?? ''}`, @@ -183,9 +214,7 @@ export function buildRecipe(collector, { generatedAt, window } = {}) { sources: [...c.sources.entries()].map(([id, s]) => ({ id, sessions: s.sessions.size, - calls: s.calls, - creditedCalls: s.creditedCalls, - credits: credits(s.nano), + ...usageView(s), })), hours: [...c.hours.entries()] .sort((a, b) => a[0] - b[0]) @@ -284,6 +313,11 @@ function workItemsFor(cycle, checked) { state: verified?.state ?? null, category: verified?.category ?? null, stateReason: verified?.stateReason ?? null, + assignees: verified?.assignees ?? [], + assigneesAtStart: verified?.assigneesAtStart ?? [], + assignmentAt: verified?.assignmentAt ?? null, + assignmentCheckedAt: verified?.assignmentCheckedAt ?? null, + assignmentStatus: verified?.assignmentStatus ?? null, createdAt: verified?.createdAt ?? null, activatedAt: verified?.activatedAt ?? null, closedAt: verified?.closedAt ?? verified?.resolvedAt ?? null, @@ -309,35 +343,34 @@ function cycleSteps(cycle, checked, workItems) { : step('no', null, null, workItems.length ? `${workItems.map((w) => w.reference).join(', ')} not found` : 'No work item linked'); steps.branch = cycle.branch - ? step('yes', 'observed', cycle.first, cycle.convention ? `Follows the convention (${cycle.convention.reference})` : 'Does not follow the convention') + ? step('yes', 'observed', cycle.first, cycle.convention ? `Branch names work item ${cycle.convention.reference}` : 'No work-item number in the branch name') : step('no', null, null, 'No branch recorded'); - const calls = cycle.sources.reduce((n, s) => n + s.calls, 0); - steps.aiWork = step('yes', 'observed', cycle.first, `${calls} model call${calls === 1 ? '' : 's'} across ${cycle.sources.length} source${cycle.sources.length === 1 ? '' : 's'}`); + steps.aiWork = step('yes', 'observed', cycle.first, `AI requests recorded in ${cycle.sources.length} source${cycle.sources.length === 1 ? '' : 's'}; totals stay separate`); const prCommits = pr?.commits?.total ?? 0; const headMoves = Math.max(0, cycle.heads.length - 1); if (pr && prCommits > 0) steps.commits = step('yes', 'verified', pr.commits.items[0]?.at ?? null, `${prCommits} commit${prCommits === 1 ? '' : 's'} on the PR`); else if (cycle.commits.length) steps.commits = step('yes', 'recorded', cycle.commits[0].at, `${cycle.commits.length} recorded commit${cycle.commits.length === 1 ? '' : 's'}`); - else if (headMoves) steps.commits = step('yes', 'observed', cycle.heads[1].at, `Head moved ${headMoves} time${headMoves === 1 ? '' : 's'} during AI work`); + else if (headMoves) steps.commits = step('yes', 'observed', cycle.heads[1].at, `The branch's recorded commit changed ${headMoves} time${headMoves === 1 ? '' : 's'} during AI work`); else if (pr) steps.commits = step('no', 'verified', null, 'The PR has no commits'); - else steps.commits = step('unknown', null, null, github && !verifiedGithub ? 'Verify to read commits from the PR' : 'No commits recorded locally'); + else steps.commits = step('unknown', null, null, github && !verifiedGithub ? 'Check delivery to load pull request commits' : 'No commits recorded locally'); const mentioned = checked?.mentioned ?? []; if (pr) steps.pullRequest = step('yes', 'verified', pr.createdAt, `#${pr.number} (${pr.isDraft ? 'draft' : pr.state.toLowerCase()})${checked.candidates > 1 ? `, ${checked.candidates} PRs on this branch` : ''}`); else if (verifiedGithub) { - steps.pullRequest = step('no', mentioned.length ? 'verified' : null, null, mentioned.length ? `Referenced ${mentioned.map((p) => `#${p.number}`).join(', ')} on another branch` : 'No PR for this branch'); + steps.pullRequest = step('no', mentioned.length ? 'verified' : null, null, mentioned.length ? `Mentioned ${mentioned.map((p) => `#${p.number}`).join(', ')} on another branch` : 'No pull request for this branch'); } else if (cycle.pullRequests.length) { steps.pullRequest = step('yes', 'recorded', cycle.pullRequests[0].at, cycle.pullRequests.map((p) => `#${p.number}`).join(', ')); - } else steps.pullRequest = step('unknown', null, null, github ? 'Verify to find the PR' : 'Only github.com repositories can be verified'); + } else steps.pullRequest = step('unknown', null, null, github ? 'Check delivery to find the pull request' : 'Only github.com repositories can be checked online'); if (pr) { const reviews = pr.reviews?.items ?? []; steps.review = (pr.reviews?.total ?? 0) > 0 ? step('yes', 'verified', minIso(reviews.map((r) => r.at)), `${pr.reviews.total} review${pr.reviews.total === 1 ? '' : 's'}${pr.reviewDecision ? ` · ${pr.reviewDecision.toLowerCase().replace(/_/g, ' ')}` : ''}`) : step('no', 'verified', null, 'No reviews'); - } else if (steps.pullRequest.state === 'no') steps.review = step('no', null, null, 'No PR'); - else steps.review = step('unknown', null, null, 'Verify to read reviews'); + } else if (steps.pullRequest.state === 'no') steps.review = step('no', null, null, 'No pull request'); + else steps.review = step('unknown', null, null, 'Check delivery to load reviews'); if (pr) { const merged = pr.state === 'MERGED'; @@ -348,8 +381,8 @@ function cycleSteps(cycle, checked, workItems) { else if (open.length) steps.close = step('no', 'verified', null, `Merged; ${open.map((w) => `${w.reference} ${w.state ?? 'open'}`).join(', ')}`); else if (unchecked.length) steps.close = step('unknown', 'verified', pr.mergedAt, `Merged; ${unchecked.map((w) => w.reference).join(', ')} not verified`); else steps.close = step('yes', 'verified', maxIso([pr.mergedAt, ...items.map((w) => w.closedAt)]), items.length ? `Merged; ${items.map((w) => w.reference).join(', ')} closed` : 'Merged; no work item to close'); - } else if (steps.pullRequest.state === 'no') steps.close = step('no', null, null, 'No PR'); - else steps.close = step('unknown', null, null, 'Verify to read merge and closure'); + } else if (steps.pullRequest.state === 'no') steps.close = step('no', null, null, 'No pull request'); + else steps.close = step('unknown', null, null, 'Check delivery to load merge and work-item status'); return steps; } @@ -412,12 +445,15 @@ function sourceCredits(cycles) { * The Recipe view model. `verification` is verify.mjs output or null; cycles * it did not cover stay unverified, and unknown steps stay unknown, not zero. */ -export function recipeView(recipe, verification = null) { +export function recipeView(recipe, verification = null, { identityLinks = [] } = {}) { + const links = validateIdentityLinks(identityLinks); const cycles = (recipe?.cycles ?? []).map((cycle) => { - const checked = verification?.cycles?.[cycle.key] ?? null; + const candidate = verification?.cycles?.[cycle.key] ?? null; + const checked = candidate?.activityFirst && candidate.activityFirst !== cycle.first ? null : candidate; const workItems = workItemsFor(cycle, checked); const steps = cycleSteps(cycle, checked, workItems); const pr = checked?.pullRequest ?? null; + const ownership = ownershipFor(workItems, pr, links); let strict = 0; for (const id of RECIPE_STEPS) { if (steps[id].state !== 'yes') break; @@ -436,6 +472,8 @@ export function recipeView(recipe, verification = null) { first: cycle.first, last: cycle.last, sources: cycle.sources, + ownership, + attribution: attributeUsage(cycle.sources, ownership, links), verified: Boolean(checked?.github), workItems, pullRequest: pr, @@ -498,6 +536,7 @@ export function recipeView(recipe, verification = null) { droppedRefs: recipe?.droppedRefs ?? 0, unlinked: recipe?.unlinked ?? [], }, + developers: developerReport(cycles, [...(recipe?.unlinked ?? []), ...(recipe?.omitted ?? [])], links), stations, adherence: { cycles: cycles.length, diff --git a/.github/extensions/ai-attribution/lib/server.mjs b/.github/extensions/ai-attribution/lib/server.mjs index 749348d..e9baf99 100644 --- a/.github/extensions/ai-attribution/lib/server.mjs +++ b/.github/extensions/ai-attribution/lib/server.mjs @@ -52,6 +52,8 @@ async function readJson(req) { * model(): Promise, research(): Promise, * coverage(): Promise, refresh(windowDays?: number): Promise, * recipe?(): Promise, verify?(options: { azureDevOpsOrg?: string }): Promise, + * identityLinks?(links: unknown): Promise, + * billing?(scope: unknown): Promise, * getView(): string, setView(view: string): string, windowDays(): number, * }, * }} options @@ -126,6 +128,16 @@ export async function startCanvasServer({ uiDir, api }) { send(res, 200, await api.verify({ azureDevOpsOrg: org })); return; } + if (req.method === 'POST' && url.pathname === '/api/recipe/identity-links' && api.identityLinks) { + const body = await readJson(req); + send(res, 200, await api.identityLinks(body.links)); + return; + } + if (req.method === 'POST' && url.pathname === '/api/recipe/billing' && api.billing) { + const body = await readJson(req); + send(res, 200, await api.billing(body)); + return; + } if (req.method === 'POST' && url.pathname === '/api/view') { const body = await readJson(req); send(res, 200, { view: api.setView(String(body.view ?? '')) }); diff --git a/.github/extensions/ai-attribution/lib/verify.mjs b/.github/extensions/ai-attribution/lib/verify.mjs index 450dd9d..bf80c72 100644 --- a/.github/extensions/ai-attribution/lib/verify.mjs +++ b/.github/extensions/ai-attribution/lib/verify.mjs @@ -3,8 +3,8 @@ // // Only identifiers already recorded locally are sent: repository, branch and // PR, issue or work-item numbers. Queries select states, dates, types, review -// states and commit IDs; titles, descriptions, commit messages, comments and -// logins are never requested. GitHub calls go through the signed-in `gh` CLI +// states, commit IDs and actor/assignment metadata. Titles, descriptions, +// commit messages and comments are never requested. GitHub uses the signed-in CLI // (no shell; values are GraphQL variables). Azure Boards uses a token from // `az account get-access-token` or AZURE_DEVOPS_EXT_PAT, held only in memory. @@ -14,6 +14,7 @@ import os from 'node:os'; import path from 'node:path'; import { branchName } from './convention.mjs'; import { isGitHubRepository } from './recipe.mjs'; +import { githubActor, boardsActor } from './actors.mjs'; /** Azure DevOps' Entra resource ID, used to request a token from the Azure CLI. */ export const AZURE_DEVOPS_RESOURCE = '499b84ac-1321-427f-aa17-267ca6975798'; @@ -32,6 +33,7 @@ const WORK_ITEM_FIELDS = [ 'System.WorkItemType', 'System.State', 'System.TeamProject', + 'System.AssignedTo', 'System.CreatedDate', 'Microsoft.VSTS.Common.ActivatedDate', 'Microsoft.VSTS.Common.ResolvedDate', @@ -71,12 +73,22 @@ function firstLine(value) { // ---------------------------------------------------------------- GitHub --- const ISSUE_TYPE = 'issueType { name }'; +const ASSIGNMENT_FIELDS = `assignees(first: 20) { totalCount nodes { login } } + timelineItems(first: 100, itemTypes: [ASSIGNED_EVENT, UNASSIGNED_EVENT]) { + pageInfo { hasNextPage } + nodes { + __typename + ... on AssignedEvent { createdAt assignee { __typename ... on User { login } ... on Bot { login } } } + ... on UnassignedEvent { createdAt assignee { __typename ... on User { login } ... on Bot { login } } } + } + }`; function prFields(issueType) { return `number state isDraft createdAt mergedAt closedAt headRefName baseRefName reviewDecision + author { login __typename } reviews(first: 50) { totalCount nodes { state submittedAt } } commits(first: 100) { totalCount nodes { commit { oid committedDate } } } - closingIssuesReferences(first: 10) { nodes { number state stateReason createdAt closedAt repository { nameWithOwner } ${issueType ? ISSUE_TYPE : ''} } }`; + closingIssuesReferences(first: 10) { nodes { number state stateReason createdAt closedAt repository { nameWithOwner } ${issueType ? ISSUE_TYPE : ''} ${ASSIGNMENT_FIELDS} } }`; } /** One GraphQL document per repository batch. Values are variables, never interpolated. */ @@ -92,7 +104,7 @@ export function githubQuery(lookups, { issueType = true } = {}) { return `${l.alias}: pullRequests(headRefName: $${l.alias}, first: 5, orderBy: { field: CREATED_AT, direction: DESC }) { totalCount nodes { ${prFields(issueType)} } }`; } vars.push(`$${l.alias}: Int!`); - return `${l.alias}: issueOrPullRequest(number: $${l.alias}) { __typename ... on Issue { number state stateReason createdAt closedAt ${issueType ? ISSUE_TYPE : ''} } }`; + return `${l.alias}: issueOrPullRequest(number: $${l.alias}) { __typename ... on Issue { number state stateReason createdAt closedAt ${issueType ? ISSUE_TYPE : ''} ${ASSIGNMENT_FIELDS} } }`; }); return `query(${vars.join(', ')}) { repository(owner: $owner, name: $name) { ${fields.join(' ')} } }`; } @@ -104,7 +116,7 @@ function githubArgs(repository, lookups, issueType) { return args; } -function issueFrom(node) { +function issueFrom(node, pickup, checkedAt) { if (!node || node.__typename !== 'Issue') return { found: false }; return { found: true, @@ -114,6 +126,35 @@ function issueFrom(node) { createdAt: date(node.createdAt), closedAt: date(node.closedAt), type: text(node.issueType?.name, 64), + ...githubAssignments(node, pickup, checkedAt), + }; +} + +export function githubAssignments(node, pickup, checkedAt) { + const current = (node.assignees?.nodes ?? []).map((a) => githubActor(a?.login)).filter(Boolean); + const events = node.timelineItems?.nodes; + const complete = node.timelineItems?.pageInfo?.hasNextPage === false && Array.isArray(events); + const at = Date.parse(pickup ?? ''); + const assignees = new Map(); + let valid = complete && Number.isFinite(at); + for (const event of events ?? []) { + const actor = githubActor(event?.assignee?.login, event?.assignee?.__typename); + const time = Date.parse(event?.createdAt ?? ''); + if (!actor || !Number.isFinite(time) || !['AssignedEvent', 'UnassignedEvent'].includes(event.__typename)) { + valid = false; + continue; + } + if (time <= at) { + if (event.__typename === 'AssignedEvent') assignees.set(actor.key, actor); + else assignees.delete(actor.key); + } + } + return { + assignees: current, + assigneesAtStart: valid ? [...assignees.values()] : [], + assignmentAt: valid ? date(pickup) : null, + assignmentCheckedAt: date(checkedAt), + assignmentStatus: valid ? 'history' : complete ? 'unknown' : 'history unavailable or truncated', }; } @@ -121,6 +162,7 @@ export function prFrom(node) { if (!node || !Number.isInteger(node.number)) return null; return { number: node.number, + author: githubActor(node.author?.login, node.author?.__typename), state: word(node.state), isDraft: node.isDraft === true, createdAt: date(node.createdAt), @@ -155,6 +197,7 @@ export function prFrom(node) { closedAt: date(i.closedAt), repository: text(i.repository?.nameWithOwner, 140), type: text(i.issueType?.name, 64), + assignment: { assignees: i.assignees, timelineItems: i.timelineItems }, })), }; } @@ -247,10 +290,16 @@ async function verifyGitHub(cycles, { run, env, now }) { const recorded = plan.prs.map((l) => prFrom(data[l.alias])).filter(Boolean); const matching = new Map([...onBranch, ...recorded.filter((pr) => pr.headRefName === cycle.branch)].map((pr) => [pr.number, pr])); const issues = {}; - for (const { id, lookup } of plan.issues) issues[id] = issueFrom(data[lookup.alias]); + const checkedAt = new Date(now).toISOString(); + for (const { id, lookup } of plan.issues) issues[id] = issueFrom(data[lookup.alias], cycle.first, checkedAt); + const chosen = choosePullRequest([...matching.values()], cycle, now); + if (chosen) chosen.closingIssues = chosen.closingIssues.map(({ assignment, ...issue }) => ({ + ...issue, ...githubAssignments(assignment ?? {}, cycle.first, checkedAt), + })); results[cycle.key] = { github: true, - pullRequest: choosePullRequest([...matching.values()], cycle, now), + activityFirst: cycle.first, + pullRequest: chosen, candidates: matching.size, mentioned: recorded.filter((pr) => pr.headRefName !== cycle.branch).map((pr) => ({ number: pr.number, headRefName: pr.headRefName })), issues, @@ -358,10 +407,15 @@ async function azureGet(fetchImpl, url, authorization) { await res.body?.cancel?.(); throw new Error(`Azure DevOps request failed (HTTP ${res.status})`); } - return JSON.parse(await readBounded(res)); + const body = await readBounded(res); + try { + return JSON.parse(body); + } catch { + throw new Error('Azure DevOps returned invalid JSON.'); + } } -async function verifyBoards(ids, { organization, run, fetch: fetchImpl, env, home, platform }) { +async function verifyBoards(ids, { organization, run, fetch: fetchImpl, env, home, platform, cycles, now }) { const org = normalizeOrganization(organization ?? '') ?? configuredOrganization({ env, home }); if (!ids.length) return { status: 'skipped', reason: 'No branch follows the AB123 convention.', organization: org, checked: 0, items: {} }; if (!org) { @@ -377,6 +431,8 @@ async function verifyBoards(ids, { organization, run, fetch: fetchImpl, env, hom if (!auth.header) return { status: 'unavailable', reason: auth.reason, organization: org, checked: 0, items: {} }; const items = {}; + const snapshots = {}; + const warnings = []; try { const wanted = ids.slice(0, MAX_WORK_ITEMS); const url = `${org}/_apis/wit/workitems?ids=${wanted.join(',')}&fields=${WORK_ITEM_FIELDS.join(',')}&errorPolicy=omit&api-version=7.1`; @@ -394,6 +450,8 @@ async function verifyBoards(ids, { organization, run, fetch: fetchImpl, env, hom type, state: text(f['System.State'], 64), category: null, + assignees: [boardsActor(f['System.AssignedTo'], org)].filter(Boolean), + assignmentCheckedAt: new Date(now).toISOString(), createdAt: date(f['System.CreatedDate']), activatedAt: date(f['Microsoft.VSTS.Common.ActivatedDate']), resolvedAt: date(f['Microsoft.VSTS.Common.ResolvedDate']), @@ -413,15 +471,52 @@ async function verifyBoards(ids, { organization, run, fetch: fetchImpl, env, hom stateCategories.set(key, new Map((states?.value ?? []).map((s) => [s?.name, text(s?.category, 32)]))); } catch (err) { if (err.auth) throw err; + warnings.push('Some work-item state categories could not be read.'); } } for (const item of Object.values(items)) { if (item.found) item.category = stateCategories.get(`${item.project}\u0000${item.type}`)?.get(item.state) ?? null; } + // fields + asOf avoids the content-bearing revision/history endpoints. + // Snapshot at first AI activity: never backdate the current assignee. + const times = new Map(); + for (const cycle of cycles.slice(0, MAX_VERIFY_CYCLES)) { + const at = date(cycle.first); + const id = cycle.convention.id; + if (!at || !items[id]?.found) continue; + const group = times.get(at) ?? []; + group.push({ key: cycle.key, id }); + times.set(at, group); + } + const groups = [...times.entries()]; + let cursor = 0; + await Promise.all(Array.from({ length: Math.min(4, groups.length) }, async () => { + while (cursor < groups.length) { + const [at, group] = groups[cursor++]; + const snapshotIds = [...new Set(group.map((c) => c.id))]; + try { + const past = await azureGet(fetchImpl, + `${org}/_apis/wit/workitems?ids=${snapshotIds.join(',')}&fields=System.Id,System.AssignedTo&asOf=${encodeURIComponent(at)}&errorPolicy=omit&api-version=7.1`, + auth.header); + if (!Array.isArray(past?.value)) throw new Error('Invalid assignment snapshot.'); + const rows = new Map(past.value.map((row) => [row.id, row])); + for (const { key, id } of group) { + const row = rows.get(id); + snapshots[key] = { + assigneesAtStart: row ? [boardsActor(row.fields?.['System.AssignedTo'], org)].filter(Boolean) : [], + assignmentAt: at, + assignmentStatus: row ? 'snapshot at pickup' : 'not present at pickup', + }; + } + } catch { + warnings.push('Some historical assignments could not be read; current assignees are not used as historical owners.'); + } + } + })); } catch (err) { return { status: 'error', reason: err instanceof Error ? err.message : String(err), organization: org, checked: 0, items: {} }; } - return { status: 'ok', reason: null, organization: org, checked: Object.keys(items).length, items }; + return { status: warnings.length ? 'partial' : 'ok', reason: [...new Set(warnings)].join(' ') || null, organization: org, checked: Object.keys(items).length, items, snapshots }; } // ------------------------------------------------------------ entry point --- @@ -452,14 +547,18 @@ export async function verifyRecipe(recipe, options = {}) { const boardsCycles = cycles.filter((c) => c.convention?.provider === 'azure-boards').slice(0, MAX_WORK_ITEMS); const ids = [...new Set(boardsCycles.map((c) => c.convention.id))]; - const boards = await verifyBoards(ids, { organization, run, fetch: fetchImpl, env, home, platform }); + const boards = await verifyBoards(ids, { organization, run, fetch: fetchImpl, env, home, platform, cycles: boardsCycles, now }); const result = {}; for (const [key, value] of Object.entries(github.cycles)) result[key] = value; for (const cycle of boardsCycles) { const item = boards.items[cycle.convention.id]; if (!item) continue; - result[cycle.key] = { ...(result[cycle.key] ?? { github: false }), workItems: { [cycle.convention.id]: item } }; + result[cycle.key] = { + ...(result[cycle.key] ?? { github: false }), + activityFirst: cycle.first, + workItems: { [cycle.convention.id]: { ...item, ...(boards.snapshots?.[cycle.key] ?? { assignmentStatus: 'history unavailable' }) } }, + }; } return { diff --git a/.github/extensions/ai-attribution/ui/app.js b/.github/extensions/ai-attribution/ui/app.js index 8cdb12a..87a8825 100644 --- a/.github/extensions/ai-attribution/ui/app.js +++ b/.github/extensions/ai-attribution/ui/app.js @@ -1,46 +1,44 @@ import { h, s, disclosure } from './dom.js'; -import { renderMarkdown } from './markdown.js'; import { intersectConcepts } from './intersection.mjs'; -import { coverageText, formatTokens } from './format.mjs'; +import { coverageText, formatTokens, entityLabels, sourceLabels as SOURCE_SHORT } from './format.mjs'; import { renderRecipe } from './recipe.mjs'; import { initializeTheme } from './theme.mjs'; const VIEWS = [ - { id: 'outcomes', label: 'Outcomes' }, + { id: 'outcomes', label: 'Overview' }, { id: 'recipe', label: 'Recipe' }, - { id: 'surfaces', label: 'Surfaces' }, - { id: 'intersection', label: 'Intersection' }, - { id: 'model', label: 'Data model' }, - { id: 'gaps', label: 'Gaps' }, - { id: 'coverage', label: 'Live coverage' }, + { id: 'surfaces', label: 'Data sources' }, + { id: 'model', label: 'How it connects' }, + { id: 'gaps', label: 'Missing data' }, + { id: 'coverage', label: 'Local usage' }, { id: 'research', label: 'Research' }, ]; const WINDOWS = [ { days: 7, label: 'Last 7 days' }, { days: 30, label: 'Last 30 days' }, { days: 90, label: 'Last 90 days' }, - { days: 0, label: 'All retained history' }, + { days: 0, label: 'All available history' }, ]; -const AVAILABILITY_TEXT = { - local: 'Local', - 'needs-vcs-join': 'Needs delivery data', - 'needs-org-api': 'Needs billing', - 'needs-baseline': 'Needs baseline', -}; -const SOURCE_SHORT = { sessionStore: 'CLI / app', traces: 'VS Code', agentHost: 'VS Code Copilot harness' }; const state = { view: 'outcomes', model: null, coverage: null, - research: null, group: 'all', + sharedOnly: false, busy: false, recipe: null, verifying: false, org: '', expanded: new Set(), showAllCycles: false, + developerSource: '', + developerActor: '', + developerPerspective: 'usage', + allDeveloperWork: false, + recipeError: null, + readingBilling: false, + billingForm: {}, }; const main = document.getElementById('view'); @@ -48,7 +46,10 @@ const main = document.getElementById('view'); async function api(path, init) { const res = await fetch(path, init); - if (!res.ok) throw new Error(`${path}: ${res.status}`); + if (!res.ok) { + const body = await res.json(); + throw new Error(typeof body.error === 'string' ? body.error : `${path}: ${res.status}`); + } return res.json(); } @@ -86,24 +87,76 @@ function statTiles(items) { )); } -function outcomeTable(m) { - return h('table', {}, - h('thead', {}, h('tr', {}, h('th', {}, 'Outcome'), h('th', {}, 'Measure'), h('th', {}, 'Status'))), - h('tbody', {}, m.outcomes.map((o) => h('tr', {}, - h('td', {}, o.name), h('td', {}, o.measure), h('td', {}, badge(o.availability, AVAILABILITY_TEXT[o.availability])), - ))), +// --------------------------------------------------------------- views --- + +function overviewMap() { + const box = ({ id, x, y, w = 320, h = 60, title, lines, kind = '' }) => + s('g', { class: `overview-node ${kind}`, 'data-overview-node': id }, + s('rect', { x, y, width: w, height: h, rx: 10 }), + s('text', { x: x + 14, y: y + 24, class: 'overview-node-title' }, title), + lines.map((line, index) => s('text', { x: x + 14, y: y + 44 + index * 17, class: 'overview-node-detail' }, line)), + ); + const edges = [ + ['source-sessionStore', 'usage', 'M340 95 H380 V90 H420', false], + ['source-traces', 'usage', 'M340 175 H380 V90 H420', false], + ['source-agentHost', 'usage', 'M340 255 H380 V90 H420', false], + ['usage', 'recipe', 'M570 123 V190', false], + ['work-platforms', 'recipe', 'M340 365 H395 V280 H420', true], + ['recipe', 'work-summary', 'M720 230 H765 V105 H800', false], + ['recipe', 'delivery-summary', 'M720 285 H780 V250 H800', false], + ['usage', 'billing-summary', 'M720 105 H740 V400 H780 V460 H800', false], + ['billing', 'billing-summary', 'M340 460 H800', true], + ]; + return h('figure', { class: 'card overview-map', 'aria-label': 'Executive summary: data sources, usage records, Recipe and bill comparison' }, + s('svg', { viewBox: '0 0 1200 525', role: 'img', 'aria-label': 'Local Copilot records feed account and usage records, then Recipe links them to work and delivery. Requested GitHub and Azure Boards checks add work details. GitHub billing is compared separately using recorded accounts, dates and models.' }, + s('defs', {}, s('marker', { id: 'overview-arrow', viewBox: '0 0 10 10', refX: 9, refY: 5, markerWidth: 6, markerHeight: 6, orient: 'auto' }, + s('path', { d: 'M0 0 L10 5 L0 10 Z' }), + )), + s('text', { x: 20, y: 34, class: 'overview-group' }, 'LOCAL USAGE RECORDS'), + s('text', { x: 800, y: 34, class: 'overview-group' }, 'QUESTIONS WE CAN ANSWER'), + s('text', { x: 20, y: 320, class: 'overview-group' }, 'CHECKS YOU REQUEST'), + edges.map(([from, to, d, requested]) => + s('path', { d, class: `overview-edge${requested ? ' requested' : ''}`, 'marker-end': 'url(#overview-arrow)', + 'data-overview-from': from, 'data-overview-to': to })), + [ + ['sessionStore', 'Sessions, credits and branches', 65], + ['traces', 'Requests and account details', 145], + ['agentHost', 'Requests and account details', 225], + ].map(([id, detail, y]) => box({ id: `source-${id}`, x: 20, y, title: SOURCE_SHORT[id], lines: [detail] })), + box({ id: 'usage', x: 420, y: 58, w: 300, h: 65, title: 'Keep each source separate', lines: ['Account, session, requests and credits'] }), + s('g', { class: 'overview-node recipe', 'data-overview-node': 'recipe' }, + s('rect', { x: 420, y: 190, width: 300, height: 175, rx: 12 }), + s('text', { x: 438, y: 221, class: 'overview-node-title' }, 'Recipe'), + s('text', { x: 438, y: 253, class: 'overview-step' }, 'Work item → branch'), + s('text', { x: 438, y: 282, class: 'overview-step' }, 'AI work → commits'), + s('text', { x: 438, y: 311, class: 'overview-step' }, 'Pull request → review → merge & close'), + s('text', { x: 438, y: 344, class: 'overview-node-detail' }, 'One cycle per branch'), + ), + box({ id: 'work-platforms', x: 20, y: 335, h: 65, kind: 'requested', title: 'GitHub + Azure Boards', + lines: ['Work owners and delivery records'] }), + box({ id: 'billing', x: 20, y: 430, h: 65, kind: 'requested', title: 'GitHub billing', + lines: ['Daily credits, discounts and charges'] }), + box({ id: 'work-summary', x: 800, y: 65, w: 380, h: 80, kind: 'answer', title: 'Who worked on what?', + lines: ['Known accounts, estimates and unknowns'] }), + box({ id: 'delivery-summary', x: 800, y: 210, w: 380, h: 80, kind: 'answer', title: 'Was the work delivered?', + lines: ['Review, merge and closure'] }), + box({ id: 'billing-summary', x: 800, y: 420, w: 380, h: 80, kind: 'answer', title: 'Usage vs bill', + lines: ['Same account, UTC day and model', 'Estimated costs, not per-change bills'] }), + ), + h('figcaption', { class: 'muted small' }, 'Solid: local records. Dashed: checks you request. A source shown here may not be recording data.'), ); } -// --------------------------------------------------------------- views --- - function renderOutcomes() { const m = state.model; const sources = okSources(); - const live = m.outcomes.filter((o) => o.liveMetric); - const names = { coverage: 'PR references', repo_coverage: 'Repository', credit_coverage: 'Credit reporting', actor_coverage: 'User identity', efficiency: 'Cache read' }; + const live = m.outcomes.filter((o) => o.liveMetric && !['coverage', 'efficiency'].includes(o.id)); + const names = { repo_coverage: 'Linked to a repository', credit_coverage: 'Requests with credit counts', actor_coverage: 'Linked to an account' }; return [ - h('p', { class: 'lede' }, 'Usage linked to work. Coverage, not financial return.'), + h('h2', { class: 'overview-title' }, 'From AI usage to work and cost'), + h('p', { class: 'lede overview-mission' }, 'Connect people, AI usage, work and cost—with facts, estimates and missing data clearly separated.'), + overviewMap(), + h('h2', {}, 'How much recorded AI usage can we link to work?'), sources.length ? h( 'div', @@ -127,19 +180,12 @@ function renderOutcomes() { ), ), ), - disclosure(`outcome-${o.id}`, 'Definition', h('p', {}, o.measure), - o.liveMetric === 'cacheReadRatio' - ? h('ul', {}, sources.map((source) => h('li', {}, - `${SOURCE_SHORT[source.id] ?? source.label}: ${coverageText(source.metricCoverage?.cacheReadRatio, source.totals.calls)}`, - ))) - : null, - ), ), ), ) - : h('p', { class: 'muted' }, 'No local store could be read. See Live coverage for details.'), - h('p', { class: 'muted small' }, 'Sources stay separate. Missing = unknown; a PR reference does not prove delivery.'), - h('h2', {}, 'Credits with a PR reference'), + : h('p', { class: 'muted' }, 'No local usage file could be read. See Local usage for details.'), + h('p', { class: 'muted small' }, 'Sources stay separate. Missing is unknown; a mention is not proof of delivery.'), + h('h2', {}, 'Credits linked to a mentioned pull request'), sources.length ? h( 'div', @@ -155,23 +201,16 @@ function renderOutcomes() { ? [ ratioBar(s.totals.credits > 0 ? pr.credits / s.totals.credits : null), h('div', { class: 'split-labels small' }, - h('span', {}, `${fmtCredits.format(pr.credits)} cr linked`), - h('span', { class: 'muted' }, `${fmtCredits.format(s.totals.credits - pr.credits)} cr unlinked`), + h('span', {}, `${fmtCredits.format(pr.credits)} credits linked`), + h('span', { class: 'muted' }, `${fmtCredits.format(s.totals.credits - pr.credits)} credits not linked`), ), ] : h('p', { class: 'muted' }, 'Not measured'), - disclosure(`outcome-funnel-${s.id}`, `Attribution · ${creditWeighted(s) ? 'credits' : 'calls'}`, funnel(s), funnelCounts(s)), ); }), ) : null, - h('h2', {}, 'Before ROI'), - h('div', { class: 'flow-strip', 'aria-label': 'Inputs still needed for ROI' }, - ['Verify delivery', 'Reconcile billed cost', 'Measure human effort', 'Compare a baseline'].map((label, i) => - h('div', {}, h('span', { class: 'flow-number' }, i + 1), label), - ), - ), - disclosure('roi-contract', 'Measures & limits', h('p', {}, m.summary), outcomeTable(m)), + h('p', { class: 'callout' }, 'Savings still need evidence of business value, people’s time and similar work without AI.'), ]; } @@ -195,11 +234,11 @@ function renderSurfaces(intersection = false) { { class: 'concept' }, disclosure(`concept-${c.id}`, h('strong', {}, c.name), h('span', { class: 'mono' }, c.canonical), - h('span', { class: 'mono' }, `OTel: ${c.otel}`), + h('span', { class: 'mono' }, `OpenTelemetry field: ${c.otel}`), c.note ? h('p', { class: 'small muted' }, c.note) : null, ), intersection - ? badge(c.intersectionAvailability, c.intersectionAvailability === 'native' ? 'All native' : 'Conditional') + ? badge(c.intersectionAvailability, c.intersectionAvailability === 'native' ? 'Reported by all' : 'Some limits') : null, ), m.surfaces.map((s) => { @@ -223,19 +262,19 @@ function renderSurfaces(intersection = false) { 'p', { class: 'lede' }, intersection - ? `${available.length} concepts shared across five surfaces. Availability, not equivalent data.` - : 'What each surface emits. Expand a cell for fields and conditions.', + ? `${available.length} types of data appear in all five tools or reports. The detail can still differ.` + : 'What each Copilot tool or report provides. Expand a cell for details.', ), intersection ? statTiles([ - ['Native everywhere', available.filter((c) => c.intersectionAvailability === 'native').length, 'native'], - ['Conditional', available.filter((c) => c.intersectionAvailability !== 'native').length, 'partial'], - ['Not shared', m.concepts.length - available.length, 'missing'], + ['Reported by all', available.filter((c) => c.intersectionAvailability === 'native').length, 'native'], + ['Some limits', available.filter((c) => c.intersectionAvailability !== 'native').length, 'partial'], + ['Not in every source', m.concepts.length - available.length, 'missing'], ]) : null, - h('p', { class: 'muted small' }, 'Per-call, daily and panel-only data differ. No cross-source join or sum.'), + h('p', { class: 'muted small' }, 'Individual requests, daily totals and screen-only values are not interchangeable. Do not add overlapping sources.'), intersection && !concepts.length - ? h('p', { class: 'muted' }, 'No verified shared concepts in this group.') + ? h('p', { class: 'muted' }, 'No checked data is shared by every source in this group.') : null, h( 'div', @@ -248,6 +287,14 @@ function renderSurfaces(intersection = false) { ), ), ), + h('div', { class: 'chips', role: 'group', 'aria-label': 'Which data to show' }, + [[false, 'All data types'], [true, 'Shared by all five']].map(([shared, label]) => + h('button', { + 'aria-pressed': String(state.sharedOnly === shared), 'data-focus-key': `scope-${shared}`, + onclick: () => { state.sharedOnly = shared; render(); }, + }, label), + ), + ), h( 'div', { class: 'chips', role: 'group', 'aria-label': 'Filter by group' }, @@ -278,7 +325,7 @@ function renderSurfaces(intersection = false) { h( 'tr', {}, - h('th', {}, 'Concept'), + h('th', {}, 'Data'), m.surfaces.map((s) => h('th', { title: s.collection }, s.name), ), @@ -287,9 +334,9 @@ function renderSurfaces(intersection = false) { h('tbody', {}, rows), ), ), - disclosure('matrix-guide', 'How to read this matrix', + disclosure('matrix-guide', 'How to read this table', h('ul', {}, m.availabilityLevels.map((l) => h('li', {}, h('strong', {}, `${l.label}: `), l.description))), - h('p', {}, 'Capabilities are not proof of collection. Native means directly emitted, not identical grain. Missing and unverified fields are excluded from Intersection.'), + h('p', {}, 'A tool may support a field without recording it here. “Reported” means the source supplies it, not that every report has the same detail. Shared data excludes fields that are not reported or not checked.'), h('ul', {}, m.surfaces.map((surface) => h('li', {}, h('strong', {}, `${surface.name}: `), surface.collection))), ), ]; @@ -298,38 +345,47 @@ function renderSurfaces(intersection = false) { function modelGraph(m) { const positions = { actor: [30, 28], ai_session: [300, 28], model_call: [570, 28], tool_call: [570, 150], - cost_ledger: [30, 150], work_item: [30, 272], attribution: [300, 272], outcome: [840, 272], + cost_ledger: [30, 410], work_item: [30, 272], attribution: [300, 272], outcome: [840, 272], + billing_reconciliation: [300, 410], work_allocation: [570, 410], + }; + const hints = { + actor: 'Who used AI', ai_session: 'One coding session', model_call: 'One AI request', tool_call: 'One tool action', + work_item: 'Issue, branch or pull request', attribution: 'Evidence connecting the work', outcome: 'What happened to the change', + cost_ledger: 'Usage and actual charges', billing_reconciliation: 'Same account, day and model', work_allocation: 'A share of the bill, not proof', }; - const labels = { actor: 'Actor', ai_session: 'AI session', model_call: 'Model call', tool_call: 'Tool call', cost_ledger: 'Billing', work_item: 'Work item', attribution: 'Attribution', outcome: 'Outcome' }; const routes = { 'actor:ai_session': 'M220 57 H300', 'ai_session:model_call': 'M490 57 H570', 'ai_session:tool_call': 'M490 70 H530 V179 H570', 'ai_session:attribution': 'M395 86 V272', 'work_item:attribution': 'M220 301 H300', - 'work_item:outcome': 'M125 330 V375 H935 V330', + 'work_item:outcome': 'M125 330 V365 H935 V330', 'ai_session:outcome': 'M490 42 H800 V301 H840', - 'cost_ledger:actor': 'M125 150 V86', + 'cost_ledger:actor': 'M30 439 H10 V100 H125 V86', + 'cost_ledger:billing_reconciliation': 'M220 439 H300', + 'model_call:billing_reconciliation': 'M665 86 V118 H790 V390 H395 V410', + 'billing_reconciliation:work_allocation': 'M490 439 H570', + 'work_allocation:work_item': 'M665 468 V525 H255 V301 H220', }; return h('figure', { class: 'card model-graph' }, - s('svg', { viewBox: '0 0 1060 400', role: 'img', 'aria-label': 'Attribution data model: actors start AI sessions, which contain model and tool calls, link to work items, and produce outcomes. Billing reconciles separately.' }, + s('svg', { viewBox: '0 0 1060 550', role: 'img', 'aria-label': 'AI usage links to work and results. Separately, compare the bill using the recorded account, day in UTC and model. Costs assigned to work are estimates.' }, s('defs', {}, s('marker', { id: 'model-arrow', viewBox: '0 0 10 10', refX: 9, refY: 5, markerWidth: 6, markerHeight: 6, orient: 'auto-start-reverse' }, s('path', { d: 'M0 0 L10 5 L0 10 Z' }), )), m.relationships.map((r) => s('path', { d: routes[`${r.from}:${r.to}`], class: 'edge', 'marker-end': 'url(#model-arrow)' }, - s('title', {}, `${r.from} ${r.label} ${r.to}`), + s('title', {}, `${entityLabels[r.from]} ${r.label} ${entityLabels[r.to]}`), )), m.entities.map((e) => { const [x, y] = positions[e.id]; return s('g', { class: `model-node node-${e.id}`, 'data-entity': e.id }, - s('title', {}, `${e.name}: ${e.grain}`), + s('title', {}, `${entityLabels[e.id]}: ${hints[e.id]}`), s('rect', { x, y, width: 190, height: 58, rx: 9 }), - s('text', { x: x + 14, y: y + 24, class: 'node-label' }, labels[e.id]), - s('text', { x: x + 14, y: y + 43, class: 'node-caption' }, e.id), + s('text', { x: x + 14, y: y + 24, class: 'node-label' }, entityLabels[e.id]), + s('text', { x: x + 14, y: y + 43, class: 'node-caption' }, hints[e.id]), ); }), ), - h('figcaption', { class: 'muted small' }, 'Usage → attribution → outcomes. Billing reconciles separately; it is never added to local usage.'), + h('figcaption', { class: 'muted small' }, 'Compare bills by account, UTC day and model. Unexplained charges stay separate; no per-pull-request bills.'), ); } @@ -339,51 +395,46 @@ function renderModel() { h( 'p', { class: 'lede' }, - 'How usage connects to work and outcomes.', + 'How AI usage connects to people, work, results and charges.', ), modelGraph(m), - h('h2', {}, 'Attribution confidence'), + h('h2', {}, 'How strong is the link?'), h( 'div', - { class: 'grid tier-cards' }, - m.tiers.map((t) => h('div', { class: 'card' }, - h('h3', {}, badge(`conf-${t.confidence.replace(/[^a-z]/g, '')}`, t.id), ' ', t.name), - h('p', { class: 'small muted' }, t.confidence), - disclosure(`tier-${t.id}`, 'Evidence', h('p', {}, t.method), h('p', {}, t.example)), + { class: 'grid link-methods' }, + m.linkMethods.map((method) => h('div', { class: 'card' }, + h('h3', {}, method.name), + h('p', { class: 'small muted' }, method.confidence), + disclosure(`link-${method.id}`, 'Why', h('p', {}, method.method), h('p', {}, method.example)), )), ), - disclosure('entity-fields', 'Entities & fields', + disclosure('entity-fields', 'Records & technical fields', h('div', { class: 'grid cards' }, m.entities.map((e) => - h('div', { class: 'card entity' }, h('h3', {}, e.name), h('p', { class: 'muted small' }, e.grain), + h('div', { class: 'card entity' }, h('h3', {}, entityLabels[e.id]), h('code', {}, e.name), h('p', { class: 'muted small' }, e.grain), h('ul', {}, e.fields.map((f) => h('li', {}, f))), ), )), ), - disclosure('model-relationships', 'Relationships', - h('ul', { class: 'relations' }, m.relationships.map((r) => - h('li', {}, h('code', {}, r.from), h('span', { class: 'arrow' }, `— ${r.label} →`), h('code', {}, r.to)), - )), - ), - disclosure('actor-provenance', 'Identity provenance', - h('p', {}, 'Source capability does not mean the local store collected it. Device identity is not person identity.'), + disclosure('actor-provenance', 'Where account details come from', + h('p', {}, 'Supported fields may not have been recorded here. A computer or operating-system account is not proof of who used AI.'), h('div', { class: 'matrix-wrap' }, h('table', {}, - h('thead', {}, h('tr', {}, h('th', {}, 'Field'), h('th', {}, 'Origin'), h('th', {}, 'Read path'), h('th', {}, 'Conditions'))), + h('thead', {}, h('tr', {}, h('th', {}, 'Field'), h('th', {}, 'How supplied'), h('th', {}, 'Where to find it'), h('th', {}, 'Limits'))), h('tbody', {}, m.entities.find((e) => e.id === 'actor').fieldProvenance.map((p) => h('tr', {}, h('td', {}, h('code', {}, p.field)), - h('td', {}, badge(p.kind === 'native' || p.kind === 'derived' ? p.kind : 'partial', p.kind)), + h('td', {}, badge(p.kind === 'native' || p.kind === 'derived' ? p.kind : 'partial', { native: 'Reported', derived: 'Calculated', configured: 'Set up manually', enriched: 'Added from another system' }[p.kind])), h('td', {}, p.readPath), h('td', {}, p.conditions), ), ), ), )), ), - h('p', { class: 'muted small' }, 'Missing stays unknown. Count each call once. Preserve the unattributed remainder.'), - disclosure('accounting-rules', 'Accounting rules', + h('p', { class: 'muted small' }, 'Missing is unknown, not zero. Count each AI request once. Keep usage we cannot link.'), + disclosure('accounting-rules', 'Counting rules', h('ol', {}, m.invariants.map((i) => h('li', {}, i))), ), - disclosure('billing-contract', 'Billing reconciliation', + disclosure('billing-contract', 'How we compare usage with the bill', h('p', {}, m.reconciliation.source), - h('p', {}, h('strong', {}, 'Grain: '), m.reconciliation.grain), + h('p', {}, h('strong', {}, 'Compare like for like: '), m.reconciliation.grain), h('ul', {}, m.reconciliation.requirements.map((rule) => h('li', {}, rule))), h('p', { class: 'muted small' }, m.reconciliation.allocationPolicy), ), @@ -410,40 +461,12 @@ function renderGaps() { { class: 'chips' }, g.surfaces.map((s) => badge('neutral', surfaceName(s))), ), - h('p', {}, h('strong', {}, 'Consequence: '), g.consequence), - h('p', {}, h('strong', {}, 'Mitigation: '), g.mitigation), - ), - ), - ), - disclosure('verification-status', `Verification status · ${m.verification.filter((v) => v.status === 'verified').length}/${m.verification.length} verified`, - h( - 'table', - {}, - h('thead', {}, h('tr', {}, h('th', {}, 'Claim'), h('th', {}, 'Status'), h('th', {}, 'Note'))), - h( - 'tbody', - {}, - m.verification.map((v) => - h('tr', {}, h('td', {}, v.claim), h('td', {}, badge(v.status)), h('td', { class: 'muted' }, v.note)), - ), - ), - ), - ), - disclosure('prior-art', 'Prior art', - h( - 'div', - { class: 'grid cards' }, - m.priorArt.map((p) => - h( - 'div', - { class: 'card' }, - h('h3', {}, p.name), - h('p', {}, h('strong', {}, 'Proved: '), p.proved), - h('p', {}, h('strong', {}, 'Learned: '), p.learned), - ), + h('p', {}, h('strong', {}, 'Why it matters: '), g.consequence), + h('p', {}, h('strong', {}, 'What to do: '), g.mitigation), ), ), ), + h('p', { class: 'muted small' }, 'Research summarizes the evidence and its limits.'), ]; } @@ -455,7 +478,7 @@ function creditWeighted(source) { function funnel(source) { const top = source.funnel[0]; const byCredits = creditWeighted(source); - const labels = { observed: 'Observed', session: 'Session', repository: 'Repository', branch: 'Branch / commit', workRef: 'Work reference', pullRequest: 'PR reference' }; + const labels = { observed: 'Recorded requests', session: 'Linked to a session', repository: 'Linked to a repository', branch: 'Branch / commit', workRef: 'Work mentioned', pullRequest: 'Pull request mentioned' }; return h( 'div', { class: 'funnel' }, @@ -472,7 +495,7 @@ function funnel(source) { ), h( 'span', - { class: 'stats', title: f.emitted ? `${fmtCredits.format(f.credits)} credits · ${fmtInt.format(f.calls)} calls · ${fmtInt.format(f.sessions)} sessions` : 'Not emitted by this source' }, + { class: 'stats', title: f.emitted ? `${fmtCredits.format(f.credits)} credits · ${fmtInt.format(f.calls)} requests · ${fmtInt.format(f.sessions)} sessions` : 'Not reported by this source' }, f.emitted ? pct(ratio) : 'Unavailable', @@ -484,7 +507,7 @@ function funnel(source) { function funnelCounts(source) { return h('div', { class: 'matrix-wrap' }, h('table', {}, - h('thead', {}, h('tr', {}, ...['Stage', 'Sessions', 'Calls', 'Credits'].map((label) => h('th', {}, label)))), + h('thead', {}, h('tr', {}, ...['Step', 'Sessions', 'Requests', 'Credits'].map((label) => h('th', {}, label)))), h('tbody', {}, source.funnel.map((stage) => h('tr', {}, h('td', {}, stage.label), ...[stage.sessions, stage.calls, stage.credits].map((value, i) => @@ -503,8 +526,8 @@ function breakdownTable(title, rows) { h('h3', {}, title), h('div', { class: 'breakdown-bars' }, rows.map((r) => h('div', {}, - h('div', { class: 'metric-label' }, h('span', {}, r.key), - h('span', { class: 'muted small' }, `${fmtCredits.format(r.credits)} cr · ${fmtInt.format(r.calls)} calls`), + h('div', { class: 'metric-label' }, h('span', {}, r.key === '(none)' ? 'Not recorded' : r.key), + h('span', { class: 'muted small' }, `${fmtCredits.format(r.credits)} credits · ${fmtInt.format(r.calls)} requests`), ), ratioBar(max > 0 ? r.credits / max : 0, `${r.key}: ${fmtCredits.format(r.credits)} credits; scaled to the largest row`), ), @@ -532,7 +555,8 @@ function sourceCard(s) { const head = h( 'div', { class: 'head' }, - h('h2', { title: s.label }, SOURCE_SHORT[s.id] ?? s.label, ' ', badge(s.status)), + h('h2', { title: s.label }, SOURCE_SHORT[s.id] ?? s.label, ' ', + badge(s.status, { ok: 'Ready', missing: 'Not found', schema: 'Format changed', unavailable: 'Not available', error: 'Read failed' }[s.status] ?? s.status)), ); if (s.status !== 'ok') return h('section', { class: 'card' }, head, h('p', {}, s.reason ?? 'Unavailable.')); const t = s.totals; @@ -550,15 +574,15 @@ function sourceCard(s) { 'div', { class: 'grid kpis' }, [ - ['Model calls', fmtInt.format(t.calls)], + ['AI requests', fmtInt.format(t.calls)], ['Reported credits', fmtCredits.format(t.credits)], - ['Credit reporting', pct(s.metrics.creditCoverage)], + ['Requests with credit counts', pct(s.metrics.creditCoverage)], ['Sessions', fmtInt.format(t.sessions)], ['Input / output tokens', `${formatTokens(t.inputTokens)} / ${formatTokens(t.outputTokens)}`], - ['Cache-read ratio', pct(s.metrics.cacheReadRatio)], - s.metrics.subAgentShare !== null ? ['Sub-agent credit share', pct(s.metrics.subAgentShare)] : null, + ['Reused input', pct(s.metrics.cacheReadRatio)], + s.metrics.subAgentShare !== null ? ['Credits used by helper agents', pct(s.metrics.subAgentShare)] : null, s.metrics.creditsToActorShare !== null && s.metrics.creditsToActorShare !== undefined - ? ['Credits with a user', pct(s.metrics.creditsToActorShare)] + ? ['Credits linked to an account', pct(s.metrics.creditsToActorShare)] : null, ] .filter(Boolean) @@ -566,19 +590,23 @@ function sourceCard(s) { h('div', { class: 'kpi' }, h('div', { class: 'label' }, label), h('div', { class: 'value' }, value)), ), ), - h('p', { class: 'muted small' }, `${t.creditedCalls}/${t.calls} calls report credits. Cache pairs: ${coverageText(s.metricCoverage?.cacheReadRatio, t.calls)}.`), + h('p', { class: 'muted small' }, `${t.creditedCalls}/${t.calls} requests include credit counts. `, + s.metricCoverage?.cacheReadRatio?.share === null || !s.metricCoverage?.cacheReadRatio + ? 'Input reuse is not measured.' + : `Input reuse can be measured for ${coverageText(s.metricCoverage.cacheReadRatio, t.calls)}.`, + ), sparkline(s.daily), - h('h3', {}, `Attribution · ${creditWeighted(s) ? 'credits' : 'calls'}`), + h('h3', {}, `Where ${creditWeighted(s) ? 'credits' : 'requests'} can be linked`), funnel(s), - disclosure(`tokens-${s.id}`, 'Token reporting', - h('p', { class: 'muted small' }, 'Reported subtotals only. Missing is unknown, not zero.'), + disclosure(`tokens-${s.id}`, 'Token counts', + h('p', { class: 'muted small' }, 'Tokens are pieces of text. Totals include only requests that report each count; missing is unknown, not zero.'), h('table', {}, - h('thead', {}, h('tr', {}, h('th', {}, 'Field'), h('th', { class: 'num' }, 'Tokens'), h('th', {}, 'Reporting coverage'))), + h('thead', {}, h('tr', {}, h('th', {}, 'Count'), h('th', { class: 'num' }, 'Tokens'), h('th', {}, 'Requests reporting this count'))), h('tbody', {}, [ ['inputTokens', 'Input'], ['outputTokens', 'Output'], - ['cacheReadTokens', 'Cache read'], - ['cacheWriteTokens', 'Cache write'], + ['cacheReadTokens', 'Reused input'], + ['cacheWriteTokens', 'Input saved for reuse'], ['reasoningTokens', 'Reasoning'], ].map(([key, label]) => h('tr', {}, h('td', {}, label), @@ -592,16 +620,16 @@ function sourceCard(s) { 'div', { class: 'grid two' }, breakdownTable('By model', b.model), - breakdownTable(b.agent ? 'By agent' : 'By initiator', b.agent ?? b.initiator), - breakdownTable('By reasoning effort', b.reasoningEffort), + breakdownTable(b.agent ? 'By AI agent' : 'Who started the request', b.agent ?? b.initiator), + breakdownTable('By reasoning setting', b.reasoningEffort), breakdownTable('By repository', b.repository), - b.user ? breakdownTable('By user (user.name)', b.user) : null, - b.link ? breakdownTable('By repository link method', b.link) : null, + b.user ? breakdownTable('By recorded account', b.user) : null, + b.link ? breakdownTable('How the repository was linked', b.link) : null, s.tools?.length ? h( 'div', {}, - h('h3', {}, 'Tool calls · top 8 by calls'), + h('h3', {}, 'Tool actions · eight most used'), h( 'table', {}, @@ -612,11 +640,11 @@ function sourceCard(s) { 'tr', {}, h('th', {}, 'Tool'), - h('th', { class: 'num' }, 'Calls'), + h('th', { class: 'num' }, 'Actions'), h( 'th', - { class: 'num', title: 'Failed calls / calls that reported an OK or ERROR status' }, - 'Failed / with status', + { class: 'num', title: 'Failed actions / actions with a reported success or failure' }, + 'Failed / known results', ), ), ), @@ -642,7 +670,7 @@ function sourceCard(s) { : null, ), ), - disclosure(`source-${s.id}`, 'Source & caveats', + disclosure(`source-${s.id}`, 'Source & limits', h('p', {}, s.label), h('p', { class: 'mono' }, shortPath(s.path)), funnelCounts(s), @@ -658,7 +686,7 @@ function renderCoverage() { h( 'p', { class: 'lede' }, - 'Local usage, by source. No prompts or responses; no cross-source totals.', + 'Usage recorded on this machine, shown separately for each source. No prompts or responses are read.', ), h( 'div', @@ -677,7 +705,7 @@ function renderCoverage() { c ? h('div', { class: 'grid coverage-sources' }, c.sources.map(sourceCard)) : h('p', { class: 'muted' }, 'Loading coverage…'), - c ? disclosure('coverage-rules', 'Accounting & scope', h('ul', {}, c.caveats.map((x) => h('li', {}, x)))) : null, + c ? disclosure('coverage-rules', 'What these numbers include', h('ul', {}, c.caveats.map((x) => h('li', {}, x)))) : null, ]; } @@ -701,6 +729,28 @@ function renderRecipeView() { org: state.org, expanded: state.expanded, showAll: state.showAllCycles, + sourceId: state.developerSource, + actorKey: state.developerActor, + perspective: state.developerPerspective, + allWork: state.allDeveloperWork, + error: state.recipeError, + readingBilling: state.readingBilling, + billingForm: state.billingForm, + lastCompleteDay: new Date(Date.now() - 86_400_000).toISOString().slice(0, 10), + setBilling: (key, value) => { state.billingForm[key] = value; }, + readBilling: (scope) => readBilling(scope), + setDeveloper: (key, value) => { + if (key === 'developer-source') state.developerSource = value; + if (key === 'developer-actor') state.developerActor = value; + if (key === 'developer-perspective') state.developerPerspective = value; + render(); + }, + toggleWork: () => { state.allDeveloperWork = !state.allDeveloperWork; render(); }, + linkIdentity: (boardsKey, githubLogin) => saveIdentityLinks([ + ...state.recipe.developers.identityLinks.filter((link) => link.boardsKey !== boardsKey), + { boardsKey, githubLogin }, + ]), + unlinkIdentity: (boardsKey) => saveIdentityLinks(state.recipe.developers.identityLinks.filter((link) => link.boardsKey !== boardsKey)), refresh: () => refresh(), verify: () => verify(), setOrg: (value) => { @@ -718,28 +768,35 @@ function renderRecipeView() { } function renderResearch() { - if (!state.research) return h('p', { class: 'muted' }, 'Loading research…'); - const document = renderMarkdown(state.research); - const index = h('div', { class: 'md research-index' }); - let section = disclosure('research-about', 'About this research'); - index.append(section); - let number = 0; - for (const node of [...document.childNodes]) { - if (node.nodeName === 'H3') { - section = disclosure(`research-${number++}`, node.textContent); - index.append(section); - } else { - section.append(node); - } - } + const shared = intersectConcepts(state.model); + const native = shared.filter((concept) => concept.intersectionAvailability === 'native'); + const findings = [ + ['Different sources', + 'VS Code has a Local harness and a Copilot Agent Host harness. Copilot CLI and the app share an engine. Cloud usage is a preview feature; organization reports contain daily totals.', + 'Keep source and reporting period. Never sum overlapping usage.'], + ['Shared fields are limited', + `${shared.length} types of data appear in all five tools or reports; ${native.length} are reported directly by all five.`, + 'The same type of data can have different detail. It may not contain the IDs needed to link records.'], + ['People need evidence', + 'A captured account identifies who used AI. Work-item assignees and pull request authors tell us who owns or submitted the work.', + 'Usage assigned from ownership is an estimate. A device, branch prefix or matching name does not prove who used AI.'], + ['Work needs verification', + 'Repository, branch and commit records can link usage to work. Online checks add pull requests, reviews, merges and closure.', + 'A mention does not prove who owns the work. A merge or review is not proof of quality or value.'], + ['Billing is separate', + 'GitHub reports credits, discounts and final charges by account, day and model, but supplies no PR/session billing IDs.', + 'Estimated work cost = final charge × recorded account’s work credits / billed credits, per model. Charges we cannot explain stay separate.'], + ]; return [ - h('p', { class: 'lede' }, 'The evidence behind the model. Read one section at a time.'), - h('div', { class: 'flow-strip' }, - h('div', {}, h('strong', {}, 'Usage'), 'Captured locally'), - h('div', {}, h('strong', {}, 'Delivery'), 'Verify ownership'), - h('div', {}, h('strong', {}, 'Return'), 'Needs cost & baseline'), - ), - index, + h('p', { class: 'lede' }, 'Research in brief · what we can measure and what we cannot conclude.'), + h('div', { class: 'matrix-wrap research-summary' }, h('table', {}, + h('thead', {}, h('tr', {}, h('th', {}, 'Finding'), h('th', {}, 'Evidence'), h('th', {}, 'Limit'))), + h('tbody', {}, findings.map(([finding, evidence, limit]) => h('tr', {}, + h('th', { scope: 'row' }, finding), h('td', {}, evidence), h('td', {}, limit), + ))), + )), + h('p', { class: 'callout' }, h('strong', {}, 'Before claiming a return on investment: '), + 'measure people’s time, business value and similar work done without AI. Credits alone show usage, not savings.'), disclosure('research-sources', `Sources · ${state.model.sources.length}`, h( 'ul', @@ -761,8 +818,7 @@ function renderResearch() { const RENDERERS = { outcomes: renderOutcomes, recipe: renderRecipeView, - surfaces: () => renderSurfaces(), - intersection: () => renderSurfaces(true), + surfaces: () => renderSurfaces(state.sharedOnly), model: renderModel, gaps: renderGaps, coverage: renderCoverage, @@ -824,6 +880,10 @@ function render() { } async function selectView(view, fromUser) { + if (view === 'intersection') { + state.sharedOnly = true; + view = 'surfaces'; + } if (!RENDERERS[view]) return; state.view = view; render(); @@ -835,10 +895,6 @@ async function selectView(view, fromUser) { body: JSON.stringify({ view }), }).catch(() => {}); } - if (view === 'research' && !state.research) { - state.research = (await api('/api/research')).markdown; - render(); - } if (view === 'recipe' && !state.recipe) await loadRecipe(); } @@ -856,6 +912,7 @@ async function loadRecipe() { async function verify() { if (state.verifying) return; state.verifying = true; + state.recipeError = null; render(); try { state.recipe = await api('/api/recipe/verify', { @@ -863,12 +920,44 @@ async function verify() { headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(state.org.trim() ? { azureDevOpsOrg: state.org.trim() } : {}), }); + } catch (err) { + state.recipeError = `Verification failed: ${err.message}`; } finally { state.verifying = false; render(); } } +async function saveIdentityLinks(links) { + state.recipeError = null; + try { + state.recipe = await api('/api/recipe/identity-links', { + method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ links }), + }); + } catch (err) { + state.recipeError = `Identity link failed: ${err.message}`; + } + render(); +} + +async function readBilling(scope) { + if (state.readingBilling) return; + state.readingBilling = true; + state.billingForm = { ...scope }; + state.recipe = { ...state.recipe, billing: { status: 'loading' } }; + render(); + try { + state.recipe = await api('/api/recipe/billing', { + method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(scope), + }); + } catch (err) { + state.recipe = { ...state.recipe, billing: { status: 'error', reason: `Billing failed: ${err.message}` } }; + } finally { + state.readingBilling = false; + render(); + } +} + async function loadCoverage() { state.coverage = await api('/api/coverage'); render(); diff --git a/.github/extensions/ai-attribution/ui/format.mjs b/.github/extensions/ai-attribution/ui/format.mjs index 7e84494..3e3e284 100644 --- a/.github/extensions/ai-attribution/ui/format.mjs +++ b/.github/extensions/ai-attribution/ui/format.mjs @@ -6,5 +6,18 @@ export function formatTokens(value) { export function coverageText(coverage, calls) { if (!coverage || coverage.share === null) return 'Not measured'; - return `${coverage.reportedCalls} / ${calls} calls (${Math.round(coverage.share * 100)}%)`; + return `${coverage.reportedCalls} / ${calls} requests (${Math.round(coverage.share * 100)}%)`; } + +export const usageLabels = { measured: 'User recorded', inferred: 'Owner estimate', unattributed: 'User unknown' }; +export const evidenceLabels = { verified: 'Checked online', recorded: 'Mentioned', convention: 'Branch name', observed: 'Local record' }; +export const sourceLabels = { + sessionStore: 'Copilot CLI / app', + traces: 'VS Code · Local harness', + agentHost: 'VS Code · Copilot Agent Host harness', +}; +export const entityLabels = { + actor: 'Account', ai_session: 'AI session', model_call: 'AI request', tool_call: 'Tool action', + work_item: 'Work item', attribution: 'Link to work', outcome: 'Result', + cost_ledger: 'GitHub bill', billing_reconciliation: 'Compare with bill', work_allocation: 'Estimated work cost', +}; diff --git a/.github/extensions/ai-attribution/ui/index.html b/.github/extensions/ai-attribution/ui/index.html index e1c969a..0b455ef 100644 --- a/.github/extensions/ai-attribution/ui/index.html +++ b/.github/extensions/ai-attribution/ui/index.html @@ -10,7 +10,7 @@

AI attribution

-

Usage, work and delivery. Not an ROI score.

+

What AI was used for, what it cost and what is still unknown.