diff --git a/.env.example b/.env.example index 7fb0a747f..eaa1eb468 100644 --- a/.env.example +++ b/.env.example @@ -41,3 +41,20 @@ BUDGET_WINDOW_MS=86400000 # Extra root CA (PEM content) trusted for the Postgres connection, for providers that pin a private root. Optional; verification stays on. (core) # DATABASE_CA_CERT= + +# Procedural memory (Memorable). Off unless MEMORABLE is a true value; any false value of +# QM_MEMORABLE forces it off regardless. Both are read once at startup. MEMORABLE_BIN names the binary to spawn (split on spaces, +# so "npx memorable" works); a missing binary is a silent no-op. +# See docs/procedural-memory.md. +#MEMORABLE=1 +#QM_MEMORABLE=0 +#MEMORABLE_BIN=memorable +# Where per-scope device sign-in is performed. Defaults to the public service. +#MEMORABLE_API_URL= +# Fallback key for scopes with no connected account of their own. +#MEMORABLE_API_KEY= +# The spawned CLI is given MEMORABLE_BACKEND=qm and MEMORABLE_DB_URL=DATABASE_URL +# unless you set them here, so procedures land in this deployment's own Postgres +# rather than a per-machine file. Set them only to override that. +#MEMORABLE_BACKEND=qm +#MEMORABLE_DB_URL= diff --git a/README.md b/README.md index 145a890e4..db3757156 100644 --- a/README.md +++ b/README.md @@ -171,11 +171,47 @@ qm, cutting the branch from `upstream/main` and checking the outgoing diff, comm messages, and screenshots for organization identifiers before it pushes. Nothing under `deploy/layers/` ever travels upstream. +## Procedural memory (Memorable) + +Optional, off by default, additive. QM can record _how_ a task was done and replay that +later, so a recurring job is not re-diagnosed from scratch every time. A prompt and the +tool calls that followed it become one _memorable_: which files changed, which commands +verified the work, in what order, with the real exit codes. A later prompt that one of +them already answers gets a short pointer injected into the system prompt. + +`MEMORABLE=1` turns it on; unset, empty, or any false value means off, and +any false value of `QM_MEMORABLE` forces off even when `MEMORABLE=1`. Both are read once at +startup, so a change takes effect on restart. With it off, `buildApp` installs no +hook and passes no dependency, and the orchestrator's one added check short-circuits on +`undefined`. QM itself opens no socket and makes no HTTP call for this: it spawns the +[Memorable](https://memorable.sh) CLI, which does the network +work outside the QM process, writes only to scopes explicitly consented `read-write`, and +stores into QM's own `DATABASE_URL` Postgres rather than a second store. + +Setup is `npm i -g memorable-cli` and `MEMORABLE=1`. From there, each person can connect +their own Memorable account: `POST /v1/memorable/connect` starts a browser sign-in for +their own scope, and the key that comes back is stored encrypted against it, so their +procedures land in their own organization. A sign-in can only ever be started for the +caller, because whoever opens the URL is whoever the key belongs to. Anything with no +account of its own, a shared channel included, uses the single `MEMORABLE_API_KEY` in the +server's environment, which is all a deployment that connects nobody ever needs. +Connecting is not consent: nothing is captured for a scope until someone sets it to +`read-write` through `POST /v1/memorable/consent`. + +Five `memorable_*` tables appear in the database `DATABASE_URL` already points at, three +created by the CLI and two by QM's own `artifactMap`. QM ships no migration for any of +them. + +[`docs/procedural-memory.md`](./docs/procedural-memory.md) has the rest: exactly which +guarantees are enforced by code in this repository and which are enforced by the binary, +what the injected block is allowed to contain, and where a model is and is not involved. + ## Going deeper - [`docs/getting-started.md`](./docs/getting-started.md) — first run, end to end - [`cli/README.md`](./cli/README.md) — the `qm` CLI and the deployment directory contract - [`docs/deploy-directory.md`](./docs/deploy-directory.md) — the deployment directory in full +- [`docs/procedural-memory.md`](./docs/procedural-memory.md) — the optional Memorable integration in full - [`.env.example`](./.env.example) — every knob, documented in place - [`plugins/`](./plugins) — the surfaces (Slack, web UI, admin, portal) diff --git a/adrs/procedural-memory.md b/adrs/procedural-memory.md new file mode 100644 index 000000000..112cc4ebc --- /dev/null +++ b/adrs/procedural-memory.md @@ -0,0 +1,59 @@ +# Procedural memory: let QM remember how it did something + +QM remembers what was said. It does not remember how anything was done. So the same +recurring job gets re-diagnosed from scratch every time: find the file again, run the +same three commands again, discover the same exit code again. + +What we would like to add: at the end of a run, take a prompt and the tool calls that +followed it, and store that as a small record. Which files changed, which commands +verified the work, what order, real exit codes. Later, when a prompt looks like one of +those records already answers it, put a short pointer in the system prompt saying where +the fix landed last time. + +The reason it is worth doing at the harness level rather than in a skill: the tool calls +and their exit codes are already in `session_entries`. Nothing else has to be captured, +and nothing has to be inferred by a model to get the record. A skill can only tell an +agent a method; this replays work that actually ran and passed. + +We have this working against QM already and can share the branch. Rough shape, so you +can tell us if it is the wrong shape before anyone writes more: + +- One env flag, off by default. With it off, no hook is registered and the orchestrator + check short-circuits on an undefined dependency. +- Roughly 30 lines added to existing files (`config.ts`, `wiring.ts`, `orchestrator.ts`, + `orchestrator/types.ts`), nothing deleted or edited in place. The rest is new files + under `src/memorable/` and `test/`. +- QM opens no socket. It spawns a local binary; that binary does the network work and + holds the consent check. Storage lands in QM's own `DATABASE_URL` Postgres, not a + second store. +- The injected block is treated as untrusted input: envelope-checked, escape-stripped, + size-capped, dropped whole rather than truncated, and appended outside the prompt-cache + boundary. + +Two things we already know are not free, so they should be part of the decision rather +than a surprise later. Recall sits on the turn's critical path behind a subprocess call +with a 15 second bound. And the relay holds no state, so a long session re-offers its +earlier work at each run end; the binary skips what it has already stored, but that is a +dedupe rather than an absence of the call. + +The parts of this that are enforced by code you can read, and the parts that are +enforced by the binary and therefore taken on trust, are separated explicitly in +`docs/procedural-memory.md`, because that is the line we would want drawn if we were +reviewing it. + +One account per deployment was the obvious first shape and it is the wrong one. QM is +multi-tenant; a single key means every scope's procedures land in one organization, and +whoever holds that key can read all of them. So each scope can connect its own account +instead, through a device authorization: QM asks the sign-in service for a code, hands the +human a URL, and stores whatever key comes back, encrypted under the same key material the +keychain already uses. QM never sees a password, and it cannot create an account for +someone who has not signed in themselves. + +That does add the one outbound call this integration otherwise avoids, to two endpoints +that carry a scope label and an opaque code and nothing else. We think that is the right +trade against a shared credential, but it is the part of this change most obviously open +to argument, so it is called out here rather than buried. A deployment that connects +nobody keeps the single-key behavior and makes no such call. + +Happy to cut it down, split it, or move any of it out of core if the answer is that it +does not belong here. diff --git a/docs/procedural-memory.md b/docs/procedural-memory.md new file mode 100644 index 000000000..4f53d6994 --- /dev/null +++ b/docs/procedural-memory.md @@ -0,0 +1,318 @@ +# Procedural memory (Memorable) + +An optional, off-by-default integration that lets QM record _how_ a task was done and +replay it later. This page is the full contract. The [README section](../README.md#procedural-memory-memorable) +is the summary. + +The integration is deliberately split. A small amount of code runs inside the QM process +and is readable in this repository. Everything else runs in a separate binary, the +`memorable` CLI, and in the service behind it. The distinction matters more than any +individual guarantee, so this page is organized around it: what QM enforces, and what QM +trusts. + +## The unit is one prompt + +A prompt and the tool calls that followed it become one _memorable_: which files changed, +which commands verified the work, in what order, with the real exit codes. + +A session with four prompts produces up to four memorables, each keyed to its own prompt. +A prompt that produced no tool calls produces nothing. Tool calls that precede the first +prompt of a session are kept as their own memorable with an empty prompt. + +This matters for recall quality. A session-wide record of four unrelated pieces of work +matches badly against a later prompt about one of them. + +## What QM enforces + +All of the following is in `src/config.ts`, `src/wiring.ts`, `src/core/orchestrator.ts`, +`src/api/routes/memorable.ts`, and the four files under `src/memorable/`. + +### The switch + +| Variable | Default | Effect | +| ---------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `MEMORABLE` | unset (off) | `1`/`true`/`yes`/`on` enables both capture and recall. Parsed by `boolEnvStrict`, so an unrecognized value is a startup error rather than a silent default. | +| `QM_MEMORABLE` | unset | Any false value (`0`/`false`/`no`/`off`/`none`, case and padding insensitive) forces the integration off even when `MEMORABLE=1`. Same `boolEnvStrict` parser, so an unrecognized value is a startup error. | +| `MEMORABLE_BIN` | `memorable` | The binary to spawn. Split on spaces, so `npx memorable` works. Spawned without a shell. | +| `MEMORABLE_API_URL` | the public service | Where device sign-in is performed. Point it at your own deployment of the extraction service to keep sign-in inside your perimeter. | +| `CONNECTOR_SECRET_KEY` | unset | Already required for the keychain. Without it, per-scope accounts are off and only the single `MEMORABLE_API_KEY` is used, because there would be nothing to encrypt a stored key with. | + +### The binary + +The published `memorable` CLI, unmodified. There is no QM-specific build of it and no +vendored copy of it in this repository. + +``` +npm i -g memorable-cli # provides `memorable` +npm i pg # the qm backend's Postgres driver; it is not bundled +``` + +#### Accounts + +Each scope can hold its own Memorable account, so a person's procedures land in their own +organization rather than in a shared one. There is no form to fill in and no key to +request: QM starts a device authorization (RFC 8628), hands the human a URL, and collects +whatever key comes back. QM never sees a password and never creates an account on +anyone's behalf. + +``` +POST /v1/memorable/connect start a device authorization; returns a URL and a code +GET /v1/memorable/connect poll it; `connected` once the human has approved +DELETE /v1/memorable/connect forget the key and any authorization still in flight +POST /v1/memorable/consent set this scope's consent; nothing is captured until you do +GET /v1/memorable/accounts what is connected, admin-only, never with keys +``` + +**These act on the caller's own `personal:` scope and nothing else.** Naming any +other scope is refused, with no admin override, for the same reason +`src/api/routes/connectors.ts` refuses it: whoever opens the URL is whoever the key +belongs to, so binding it to a channel would file the first person who clicked under +everyone else's name, and binding it to another person's scope would let an admin route +their transcripts to an organization they do not control. Post the URL where only that +person sees it. + +A consequence to plan around: a session in a shared channel runs under `channel:`, +which nobody can connect, so it uses the deployment's own key. Per-person accounts cover +personal-scope sessions. That is the conservative reading of who a channel's procedures +belong to, and it is deliberate rather than an oversight. + +The flow is: `POST` the connect, show the human `verificationUriComplete`, and `GET` +until the status stops being `pending`. The window is ten minutes. A `503` means the +sign-in service could not be reached and nothing was stored. A transient failure while +polling reports `pending` rather than discarding a code the human may already have +approved. + +#### Which key a spawn uses + +| Order | Source | +| ----- | ---------------------------------------- | +| 1 | the scope's own connected account | +| 2 | `MEMORABLE_API_KEY` from the environment | + +A key is never borrowed by a scope that did not connect it. A deployment where nobody has +connected keeps working exactly as before on the single environment key. + +To get that environment key by hand instead, run `memorable login` once on a machine with +a browser (`--code` on a container or CI runner) and copy the `api_key` out of +`~/.memorable/config.json`. + +#### What the CLI reads from QM's environment + +| Variable | Effect | +| ------------------- | --------------------------------------------------------- | +| `MEMORABLE_BACKEND` | `qm` — store in QM's Postgres rather than on this machine | +| `MEMORABLE_DB_URL` | Where. Falls back to `DATABASE_URL`, which is QM's own | +| `MEMORABLE_API_URL` | The extraction service | +| `MEMORABLE_API_KEY` | The connected scope's key, or the deployment's; see above | + +#### What lands in your database + +Five tables, all on QM's DurableMap row shape (`id TEXT PRIMARY KEY, json JSONB NOT NULL`), +all in the database `DATABASE_URL` already points at. No new database is created. + +Three are the CLI's, created on first write. QM ships no migration for these and the +schema is not QM's: + +```sql +CREATE TABLE IF NOT EXISTS memorable_procedures (...); +CREATE TABLE IF NOT EXISTS memorable_mode (...); +CREATE TABLE IF NOT EXISTS memorable_stats (...); +``` + +Two are QM's own, created lazily by `artifactMap` exactly as `consent_links` and +`secret_drops` are, so there is no migration for these either: + +```sql +CREATE TABLE IF NOT EXISTS memorable_accounts (...); +CREATE TABLE IF NOT EXISTS memorable_device_codes (...); +``` + +`memorable_accounts` holds one row per connected scope. **The key is encrypted at rest** +with `deriveConnectorKey(CONNECTOR_SECRET_KEY, "memorable-accounts")`, the same AES-256-GCM +path `model_credentials` uses; nothing in the row is the key in the clear, and a row that +will not decrypt reads as no key rather than throwing. `memorable_device_codes` holds +in-flight authorizations, one row per scope, and carries no key. A row is replaced when +the same scope starts again and cleared when it is polled after expiry; a scope that +starts a sign-in and never polls again leaves its row until it does. There is no sweeper. + +Removing the integration leaves all five behind; drop them if you want the data gone. + +QM calls exactly two of its subcommands: + +``` +memorable inject --scope # task on stdin, injected block on stdout +memorable record --scope - # capture JSON on stdin +``` + +`-` is the CLI's own convention for "the input is on stdin"; without it `record` goes +looking for a session receipt on disk, which a server process does not leave. + +Consent is per scope and falls back to the org that owns it, so +`memorable enable --scope org:` once answers for every channel and person under +it, and a narrower scope answered for itself overrides that. Two absent answers are still +`unset`, and `unset` is deny. + +With the flag off, `buildApp` registers no `onTerminal` hook and passes no `memorable` +dependency to the orchestrator. The single check added to the turn path is +`useMemory && deps.memorable && input.text.trim()`, which short-circuits on an undefined +dependency before any work happens. The three modules under `src/memorable/` are imported +at boot either way; they are 221 lines that never run. + +Both variables are read once, by `loadConfig` at startup. Setting `QM_MEMORABLE=0` in a +running process does nothing until the process restarts. If you want a gate that answers +mid-session, say so: the idiom is already here in `src/resolution/config-store.ts`, next +to `getIndividualModelAuthDurable`, and we will move to it. + +### Egress + +QM makes exactly one class of outbound call of its own, and only to sign someone in. +`src/memorable/accounts.ts` posts to two endpoints and nothing else: + +| Endpoint | When | Carries | +| ---------------------------- | ------------------------ | --------------------------------------------------------------------------------------------------- | +| `POST /v1/device/code` | someone starts a connect | a label like `qm channel a1b2c3d4`: the scope kind plus a truncated hash, never the scope id itself | +| `POST /v1/device/token` | polling that connect | the opaque device code | + +Neither carries a prompt, a tool call, a file, or a transcript. `` is +`MEMORABLE_API_URL`. The `fetch` is injected through `createMemorableAccounts`, so no +test in this repository reaches the network. + +Everything on the recall and capture paths is still spawn-only: a `spawn` of a local +binary for recall, and a detached `spawn` of the same binary at run end for capture. Any +network traffic carrying your data originates from that binary, on the machine QM is +running on, after its own consent checks. Verify the split: + +``` +grep -rnE 'fetch\(|https?://|node:https?|net\.|WebSocket' src/memorable/ +``` + +Only `accounts.ts` answers. + +### The child's environment + +Spawned children get an allow-list, built once in `loadConfig` as `memorableProcessEnv`, +never the whole process environment: + +``` +PATH TMPDIR LANG LC_ALL SSL_CERT_FILE SSL_CERT_DIR NODE_EXTRA_CA_CERTS +HTTP_PROXY HTTPS_PROXY NO_PROXY ALL_PROXY HOME +MEMORABLE_BACKEND MEMORABLE_DB_URL MEMORABLE_API_URL MEMORABLE_API_KEY MEMORABLE_HOME +``` + +`MEMORABLE_BACKEND` defaults to `qm` and `MEMORABLE_DB_URL` to `DATABASE_URL`, so the CLI +writes into this deployment's Postgres without the operator having to know that; an +explicit value for either wins. `DATABASE_URL` itself is deliberately **not** forwarded: +the child gets the connection string under the one name that is meant for it. + +The relay honours QM's own memory policy. With `MEMORABLE_CAPTURE=off` no session-end +relay is registered, and with recall off no injection is attempted, so procedural memory +cannot outlive the switch operators already use. + +This mirrors what `codexProcessEnv` and `claudeProcessEnv` already do for the harnesses. +When the scope has a connected account, its key replaces `MEMORABLE_API_KEY` for that one +spawn; nothing else about the environment changes. + +### What the injected block may contain + +`src/memorable/inject.ts` treats the subprocess's stdout as untrusted. A block is used +only if all of the following hold: + +- The process exited `0`. +- The text opens with the literal data-not-instructions envelope prefix. +- After stripping, the text is at most 8,000 characters. + +Stripping removes escape sequences by family, each matched as a whole sequence (CSI, OSC, +DCS/SOS/PM/APC, the enumerated two-character escapes, and a stray `ESC` last), then bare +control bytes other than tab and newline. Carriage return is stripped because it rewrites +a terminal line, which is a spoofing primitive in any surface that later prints a stored +command. + +An over-length block is dropped whole, never truncated. The sentence that marks the block +as inert data sits at the end of it, so slicing to fit would remove exactly the part that +makes the block safe. + +A recall attempt is bounded at 15 seconds, after which the child is killed and the turn +proceeds with no injected block. Any failure at any point yields no block rather than a +turn error. + +### Prompt cache + +The recalled block is appended to the system prompt _after_ `stableSystemBytes` is +measured, and that value is what gets passed as `systemCacheBoundary`. An injection +therefore does not invalidate the cached prefix. + +### Capture + +At the end of a run with no other active run on the thread, the session's entries are +read and split into workflows at each user prompt. Each tool call carries its name, its +input payload, and, joined by `callId`, whether it succeeded and its exit code when the +tool was `execute`. A quarantined result counts as a failure; success is never inferred. +The JSON goes to the binary's stdin. + +Note what the input payload means in practice: it is the tool call's arguments minus +`tool` and `callId` — whatever `recordCall` in `src/harness/pi-tools.ts` chose to record. +Anything a tool call carried into that payload is what the subprocess receives. The +subprocess then drops all but an allow-listed handful of those fields before anything +leaves the machine, but that is its guarantee to keep, not QM's. + +## What QM trusts + +None of the following is enforced by code in this repository. It is enforced by the +`memorable` binary and the service behind it, and it is listed here so an operator knows +what is being taken on trust. + +- **Consent is per scope and fail-closed.** Writes happen only for scopes explicitly set + to `read-write` via `memorable enable`. Unset means deny, and deny suppresses recall as + well as writes. +- **Steps are deterministic.** The steps, commands, exit codes and postconditions in a + stored memorable are built from the trace by a parser with no model call. +- **Two things do use a small model.** The human-readable title is written by one, and + the second stage of the admission gate that decides whether a memorable is kept at all + is another. Neither can change what a step says. The gate's first stage is a + deterministic prefilter that refuses one-step traces, read-only traces, traces where + every step is the same verb, and work with no checkable ending; only what survives it + reaches the model. +- **Storage rides QM's own database.** The `memorable_procedures` and `memorable_mode` + tables live in the same `DATABASE_URL` Postgres. There is no second store. +- **The trace is minimized before it leaves the machine.** The binary forwards a tool + call's name, its outcome, and an allow-listed set of argument fields; a field not on + that list is dropped rather than sent, home paths collapse to `~`, and + credential-shaped strings are redacted. So the file contents QM's `write` tool carries + in its payload do not travel. + +## Plans + +A prompt that is several things at once can be answered with a plan instead of a single +memorable: several stored memorables in dependency order, each naming the files its +verified run wrote and the command that proved it. The order is derived rather than +guessed. A memorable that writes a file and one that reads it are a dependency, in that +direction, and both facts are already in the recorded steps. Steps sharing no dependency +are marked safe to run in parallel, and anything memory cannot cover is stated as such +rather than filled with the nearest vaguely similar memorable. + +Opt in per call with `memorable inject --chain`, or `MEMORABLE_CHAIN=1`. A plan uses the +same envelope and the same size cap, so nothing in the harness changes to accept one. + +## Known costs + +- **Recall is on the turn's critical path.** When enabled, every turn that is not + `skipMemory` awaits the subprocess before the model is called, bounded at 15 seconds. +- **The relay holds no state.** `onTerminal` fires once per run and the relay re-reads the + session's entries each time, so a long session re-sends its earlier workflows on every + later run. The binary skips workflows it has already stored, but a workflow the + admission gate _refused_ has no stored row and is re-offered on each later run of that + session. + +```mermaid +flowchart LR + subgraph QM["QM host process"] + LOOP["agent loop"] -->|emits tool_call / tool_result| SE[("session_entries")] + SE -->|"cut at each prompt"| RELAY["relay: one workflow per prompt"] + end + RELAY -->|"workflows JSON, stdin"| CLI["memorable CLI"] + CLI -->|"POST /v1/extract, once per prompt"| API["extraction worker
steps deterministic;
title and admission gate use a small model"] + API -->|"draft + admission verdict"| CLI + CLI -->|"write iff admitted AND consent read-write"| DB[("memorable_* tables
in QM's own Postgres")] + DB -->|recall top hit| CLI + CLI -->|"a short pointer, a multi-step plan, or nothing"| LOOP +``` diff --git a/src/api/agent-api-catalog.ts b/src/api/agent-api-catalog.ts index d4db19a51..fd9dc4ea9 100644 --- a/src/api/agent-api-catalog.ts +++ b/src/api/agent-api-catalog.ts @@ -50,6 +50,43 @@ const FAMILIES: AgentApiFamily[] = [ }, ], }, + { + match: (m, p) => + (p === "/v1/memorable/connect" && (m === "POST" || m === "GET" || m === "DELETE")) || + (p === "/v1/memorable/consent" && m === "POST") || + (p === "/v1/memorable/accounts" && m === "GET"), + guidance: + "Procedural memory stores a scope's procedures in that scope's own Memorable organization. Connecting is a person signing in themselves: start it, give them the URL, and poll until the status stops being pending. Connecting is not consent: capture stays off until someone sets consent to read-write, and asking for that is a separate question you put to the person, never a default you pick for them. These act on your caller's own scope only, and naming any other scope is refused, because whoever opens the URL is whoever the key belongs to. Post the URL where only that person will see it.", + routes: [ + { + method: "POST", + path: "/v1/memorable/connect", + summary: + "start a browser sign-in for this scope; returns verificationUriComplete and userCode to show the person, or already_connected", + }, + { + method: "GET", + path: "/v1/memorable/connect", + summary: "poll the sign-in: pending, connected, denied, expired, or none", + }, + { + method: "DELETE", + path: "/v1/memorable/connect", + summary: "forget this scope's stored key and any sign-in still in flight", + }, + { + method: "POST", + path: "/v1/memorable/consent", + summary: + 'set this scope\'s consent with {mode}: "read-write" turns capture on, "read-only" keeps recall but stores nothing new, "deny" turns both off. Nothing is ever captured until someone chooses read-write', + }, + { + method: "GET", + path: "/v1/memorable/accounts", + summary: "org admins only: which scopes have connected an account, never the keys", + }, + ], + }, { match: (m, p) => p === "/v1/channel-header-pin" && (m === "GET" || m === "PUT"), guidance: diff --git a/src/api/deps.ts b/src/api/deps.ts index d0dadd549..2e74ebdf2 100644 --- a/src/api/deps.ts +++ b/src/api/deps.ts @@ -7,6 +7,7 @@ import type { McpToolService } from "../mcp/mcp-tool-service.ts"; import type { ReplayDedupe } from "../auth/replay-dedupe.ts"; import type { FetchLike, OAuthClientResolver } from "../connectors/oauth.ts"; import type { ConsentLinkStore } from "../connectors/consent-link.ts"; +import type { MemorableAccounts } from "../memorable/accounts.ts"; import type { OrgBranding, ScopedConfigStore } from "../resolution/config-store.ts"; import type { AclStore } from "../acl/acl-store.ts"; import type { CredentialUsageSink } from "../admin/credential-usage-sink.ts"; @@ -74,6 +75,9 @@ export interface ServerDeps { oauthEnv?: NodeJS.ProcessEnv; resolveClient?: OAuthClientResolver; consentLinks?: ConsentLinkStore; + memorableAccounts?: MemorableAccounts; + memorableBin?: string; + memorableProcessEnv?: NodeJS.ProcessEnv; apiBaseUrl?: string; publicUrl?: string; portalUrl?: string; diff --git a/src/api/routes/index.ts b/src/api/routes/index.ts index 28503004f..870ab590c 100644 --- a/src/api/routes/index.ts +++ b/src/api/routes/index.ts @@ -26,6 +26,7 @@ import { deploymentLayerRoutes } from "./deployment-layer.ts"; import { egressAuditRoutes } from "./egress-audit.ts"; import { authBrokerRoutes } from "./auth-broker.ts"; import { userModelAuthRoutes } from "./user-model-auth.ts"; +import { memorableRoutes } from "./memorable.ts"; export const rawRoutes: ReadonlyArray> = [ { method: "GET", path: "/healthz", auth: "public", handle: ({ res }) => sendJson(res, 200, { ok: true }) }, @@ -44,6 +45,7 @@ export const rawRoutes: ReadonlyArray> = [ export const apiRoutes: ReadonlyArray> = [ ...deploymentLayerRoutes, + ...memorableRoutes, ...turnRoutes, ...credentialRoutes, ...keychainRoutes, diff --git a/src/api/routes/memorable.ts b/src/api/routes/memorable.ts new file mode 100644 index 000000000..5854a4772 --- /dev/null +++ b/src/api/routes/memorable.ts @@ -0,0 +1,154 @@ +import { sendJson } from "../http.ts"; +import { audit, authorizeAdmin, isObj, orgScope } from "./shared.ts"; +import { parseConsentMode, setConsent } from "../../memorable/consent.ts"; +import { personalScope } from "../../types.ts"; +import type { ApiCtx, Route } from "./route.ts"; + +type Resolved = { scope: string; actorId: string } | null; + +async function resolveScope(ctx: ApiCtx, requested: string): Promise { + const { res, capability } = ctx; + if (!capability) { + sendJson(res, 401, { error: "unauthorized", message: "agent capability token required" }); + return null; + } + const own = personalScope(capability.actorId); + if (requested && requested !== own) { + sendJson(res, 400, { + error: "bad_request", + message: + "a Memorable sign-in can only be started for yourself: whoever opens the URL is whoever the key belongs to, so binding it to another scope would file their account under someone else's name. Ask that person to run it from their own session.", + }); + return null; + } + return { scope: own, actorId: capability.actorId }; +} + +function requestedScope(ctx: ApiCtx): string { + const body = isObj(ctx.body) ? ctx.body : {}; + const fromBody = typeof body.scope === "string" ? body.scope : ""; + return fromBody || ctx.url.searchParams.get("scope") || ""; +} + +async function startConnect(ctx: ApiCtx): Promise { + const { res, deps } = ctx; + if (!deps.memorableAccounts) { + return sendJson(res, 404, { + error: "not_supported", + message: + "per-scope Memorable accounts are unavailable: the integration is off, or CONNECTOR_SECRET_KEY is unset so there is nothing to encrypt a stored key with", + }); + } + const resolved = await resolveScope(ctx, requestedScope(ctx)); + if (!resolved) return; + const body = isObj(ctx.body) ? ctx.body : {}; + const result = await deps.memorableAccounts.start(resolved.scope, { force: body.force === true }); + if (result.status === "unavailable") return sendJson(res, 503, result); + audit(deps, { + principalId: resolved.actorId, + action: "memorable.connect.start", + resource: resolved.scope, + scopeLabel: resolved.scope, + }); + return sendJson(res, 200, { scope: resolved.scope, ...result }); +} + +async function connectStatus(ctx: ApiCtx): Promise { + const { res, deps } = ctx; + if (!deps.memorableAccounts) { + return sendJson(res, 404, { + error: "not_supported", + message: + "per-scope Memorable accounts are unavailable: the integration is off, or CONNECTOR_SECRET_KEY is unset so there is nothing to encrypt a stored key with", + }); + } + const resolved = await resolveScope(ctx, requestedScope(ctx)); + if (!resolved) return; + const result = await deps.memorableAccounts.poll(resolved.scope); + if (result.status === "unavailable") return sendJson(res, 503, result); + if (result.status === "connected") { + audit(deps, { + principalId: resolved.actorId, + action: "memorable.connect.complete", + resource: result.orgId, + scopeLabel: resolved.scope, + }); + } + return sendJson(res, 200, { scope: resolved.scope, ...result }); +} + +async function disconnect(ctx: ApiCtx): Promise { + const { res, deps } = ctx; + if (!deps.memorableAccounts) { + return sendJson(res, 404, { + error: "not_supported", + message: + "per-scope Memorable accounts are unavailable: the integration is off, or CONNECTOR_SECRET_KEY is unset so there is nothing to encrypt a stored key with", + }); + } + const resolved = await resolveScope(ctx, requestedScope(ctx)); + if (!resolved) return; + const removed = await deps.memorableAccounts.disconnect(resolved.scope); + audit(deps, { + principalId: resolved.actorId, + action: "memorable.disconnect", + resource: resolved.scope, + scopeLabel: resolved.scope, + }); + return sendJson(res, 200, { scope: resolved.scope, disconnected: removed }); +} + +async function consent(ctx: ApiCtx): Promise { + const { res, deps } = ctx; + if (!deps.memorableAccounts || !deps.memorableBin || !deps.memorableProcessEnv) { + return sendJson(res, 404, { + error: "not_supported", + message: + "per-scope Memorable accounts are unavailable: the integration is off, or CONNECTOR_SECRET_KEY is unset so there is nothing to encrypt a stored key with", + }); + } + const body = isObj(ctx.body) ? ctx.body : {}; + const mode = parseConsentMode(body.mode); + if (!mode) { + return sendJson(res, 400, { + error: "bad_request", + message: 'mode must be "read-write" (capture on), "read-only" (recall only), or "deny"', + }); + } + const resolved = await resolveScope(ctx, requestedScope(ctx)); + if (!resolved) return; + const apiKey = await deps.memorableAccounts.keyFor(resolved.scope).catch(() => null); + const result = await setConsent(deps.memorableBin, resolved.scope, mode, { + env: deps.memorableProcessEnv, + ...(apiKey ? { apiKey } : {}), + }); + audit(deps, { + principalId: resolved.actorId, + action: "memorable.consent", + resource: mode, + scopeLabel: resolved.scope, + }); + return sendJson(res, result.ok ? 200 : 502, { scope: resolved.scope, ...result }); +} + +async function listAccounts(ctx: ApiCtx): Promise { + const { res, deps } = ctx; + if (!deps.memorableAccounts) { + return sendJson(res, 404, { + error: "not_supported", + message: + "per-scope Memorable accounts are unavailable: the integration is off, or CONNECTOR_SECRET_KEY is unset so there is nothing to encrypt a stored key with", + }); + } + const actor = await authorizeAdmin(ctx, orgScope()); + if (!actor) return; + return sendJson(res, 200, { accounts: await deps.memorableAccounts.connected() }); +} + +export const memorableRoutes: ReadonlyArray> = [ + { method: "POST", path: "/v1/memorable/connect", auth: "either", handle: startConnect }, + { method: "GET", path: "/v1/memorable/connect", auth: "either", handle: connectStatus }, + { method: "DELETE", path: "/v1/memorable/connect", auth: "either", handle: disconnect }, + { method: "POST", path: "/v1/memorable/consent", auth: "either", handle: consent }, + { method: "GET", path: "/v1/memorable/accounts", auth: "either", handle: listAccounts }, +]; diff --git a/src/config.ts b/src/config.ts index 6acc29e2c..5e5086c63 100644 --- a/src/config.ts +++ b/src/config.ts @@ -8,6 +8,7 @@ import { type MemoryRecallMode, } from "./memory/policy.ts"; import { parseMemoryStrategyKind, type MemoryStrategyKind } from "./memory/strategy.ts"; +import { DEFAULT_MEMORABLE_API_URL } from "./memorable/accounts.ts"; import { sanitizeBranding } from "./resolution/branding.ts"; import type { OrgBranding } from "./resolution/config-store.ts"; import { validateCoreSecretEnv } from "./deployment/secret-schema.ts"; @@ -146,6 +147,10 @@ export interface Config { insightsIntervalMs: number; reachDeniedNotifyChannel?: string; scratchExecEnabled: boolean; + memorableEnabled: boolean; + memorableBin: string; + memorableApiUrl: string; + memorableProcessEnv: NodeJS.ProcessEnv; reachExecEnabled: boolean; sharedOwnerAuthIsolation: boolean; surfaceDebugFooter: boolean; @@ -782,6 +787,30 @@ export function loadConfig(env: NodeJS.ProcessEnv = process.env): Config { "CLAUDE_CODE_OAUTH_TOKEN", ].flatMap((name) => (env[name] === undefined ? [] : [[name, env[name]]])), ) as NodeJS.ProcessEnv; + const memorableProcessEnv = Object.fromEntries( + [ + "PATH", + "TMPDIR", + "LANG", + "LC_ALL", + "SSL_CERT_FILE", + "SSL_CERT_DIR", + "NODE_EXTRA_CA_CERTS", + "HTTP_PROXY", + "HTTPS_PROXY", + "NO_PROXY", + "ALL_PROXY", + "HOME", + "MEMORABLE_BACKEND", + "MEMORABLE_DB_URL", + "MEMORABLE_API_URL", + "MEMORABLE_API_KEY", + "MEMORABLE_HOME", + ].flatMap((name) => (env[name] === undefined ? [] : [[name, env[name]]])), + ) as NodeJS.ProcessEnv; + memorableProcessEnv.MEMORABLE_BACKEND = env.MEMORABLE_BACKEND?.trim() || "qm"; + if (!memorableProcessEnv.MEMORABLE_DB_URL && env.DATABASE_URL) + memorableProcessEnv.MEMORABLE_DB_URL = env.DATABASE_URL; if (providerBaseUrls.openai) codexProcessEnv.OPENAI_BASE_URL = providerBaseUrls.openai; if (providerBaseUrls.anthropic) claudeProcessEnv.ANTHROPIC_BASE_URL = providerBaseUrls.anthropic; const turnWallClockMs = @@ -969,6 +998,11 @@ export function loadConfig(env: NodeJS.ProcessEnv = process.env): Config { numEnvStrict("INSIGHTS_INTERVAL_MS", env.INSIGHTS_INTERVAL_MS) ?? CONFIG_DEFAULTS.insightsIntervalMs, ...(env.REACH_DENIED_NOTIFY_CHANNEL ? { reachDeniedNotifyChannel: env.REACH_DENIED_NOTIFY_CHANNEL.trim() } : {}), scratchExecEnabled: boolEnvStrict("EXECUTE_SCRATCH", env.EXECUTE_SCRATCH) ?? false, + memorableEnabled: + (boolEnvStrict("MEMORABLE", env.MEMORABLE) ?? false) && (boolEnvStrict("QM_MEMORABLE", env.QM_MEMORABLE) ?? true), + memorableBin: env.MEMORABLE_BIN?.trim() || "memorable", + memorableApiUrl: env.MEMORABLE_API_URL?.trim() || DEFAULT_MEMORABLE_API_URL, + memorableProcessEnv, reachExecEnabled: boolEnvStrict("REACH_EXEC", env.REACH_EXEC) ?? false, sharedOwnerAuthIsolation: boolEnvStrict("SHARED_OWNER_AUTH_ISOLATION", env.SHARED_OWNER_AUTH_ISOLATION) ?? false, surfaceDebugFooter: boolEnvStrict("SURFACE_DEBUG_FOOTER", env.SURFACE_DEBUG_FOOTER) ?? false, diff --git a/src/core/orchestrator.ts b/src/core/orchestrator.ts index 904a92941..637894ad3 100644 --- a/src/core/orchestrator.ts +++ b/src/core/orchestrator.ts @@ -954,6 +954,10 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator { const memoryBlock = recalled ? `\n\n## What you remember\nYou're in ${memoryContext}. A memory tagged \`(said in …)\` was stated in another context — apply it only if that tag matches here; untagged memories are general.\n\n${recalled}` : ""; + const memorableBlock = + useMemory && deps.memorable && input.text.trim() + ? await deps.memorable(memoryScopeId, input.text).catch(swallowAs("orchestrator: memorable recall", null)) + : null; let onboardingBlock = ""; if (useMemory && conversation.kind === "dm" && onboardingSkillVisible(visibleSkills)) { @@ -1639,6 +1643,7 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator { systemPrompt += `\n\n${renderConnectedAppsBlock(status, configuredProviders, connectionsUrl)}`; } systemPrompt += memoryBlock; + if (memorableBlock) systemPrompt += `\n\n${memorableBlock}`; if (onboardingBlock) systemPrompt += `\n\n${onboardingBlock}`; const isRetry = (input.attempt ?? 1) > 1; diff --git a/src/core/orchestrator/types.ts b/src/core/orchestrator/types.ts index efe4b1e1a..42ee1eff8 100644 --- a/src/core/orchestrator/types.ts +++ b/src/core/orchestrator/types.ts @@ -3,6 +3,7 @@ import type { Conversation, Principal, PendingApprovalRecord, + ScopeId, SurfaceContextQuery, SurfaceContextResult, TurnRequest, @@ -135,6 +136,7 @@ export interface OrchestratorDeps { mcp?: McpToolService; memoryPolicy?: MemoryPolicy; memoryStrategy?: MemoryStrategy; + memorable?: (scopeId: ScopeId, task: string) => Promise; skills?: SkillStore; skillBundles?: SkillBundleStore; skillsReady?: Promise; diff --git a/src/memorable/accounts.ts b/src/memorable/accounts.ts new file mode 100644 index 000000000..e39951c1d --- /dev/null +++ b/src/memorable/accounts.ts @@ -0,0 +1,245 @@ +import type { DurableMap } from "../persistence/durable-map.ts"; +import { decryptSecret, deriveConnectorKey, encryptSecret } from "../connectors/connector-client-store.ts"; +import { scopeStorageKey } from "../util/scope-storage-key.ts"; +import { hashId } from "../util/crypto.ts"; +import { parseScopeId, type ScopeId } from "../types.ts"; + +export const DEFAULT_MEMORABLE_API_URL = "https://memorable-extraction-api.memorable.workers.dev"; + +const START_TIMEOUT_MS = 10_000; +const POLL_TIMEOUT_MS = 10_000; +const MIN_POLL_INTERVAL_MS = 2_000; +const MAX_FIELD_CHARS = 200; +const MAX_API_KEY_CHARS = 512; + +export interface MemorableAccount { + scopeId: ScopeId; + apiKeyEnc: string; + keyId: string; + orgId: string; + orgName: string; + connectedAt: number; +} + +export interface PendingConnect { + scopeId: ScopeId; + deviceCode: string; + userCode: string; + verificationUri: string; + verificationUriComplete: string; + intervalMs: number; + expiresAt: number; +} + +type StartResult = + | { status: "started"; userCode: string; verificationUri: string; verificationUriComplete: string; expiresAt: number } + | { status: "already_connected"; orgName: string } + | { status: "unavailable"; detail: string }; + +type PollResult = + | { status: "pending"; userCode: string; verificationUriComplete: string } + | { status: "connected"; orgId: string; orgName: string; keyId: string } + | { status: "denied" } + | { status: "expired" } + | { status: "none" } + | { status: "unavailable"; detail: string }; + +export interface MemorableAccounts { + start(scope: ScopeId, opts?: { force?: boolean }): Promise; + poll(scope: ScopeId): Promise; + keyFor(scope: ScopeId): Promise; + disconnect(scope: ScopeId): Promise; + connected(): Promise>>; +} + +interface DeviceStartBody { + device_code?: unknown; + user_code?: unknown; + verification_uri?: unknown; + verification_uri_complete?: unknown; + expires_in?: unknown; + interval?: unknown; + error?: unknown; + detail?: unknown; +} + +interface DevicePollBody { + status?: unknown; + api_key?: unknown; + key_id?: unknown; + org_id?: unknown; + org_name?: unknown; +} + +function str(value: unknown, max = MAX_FIELD_CHARS): string { + return typeof value === "string" ? value.slice(0, max) : ""; +} + +export function createMemorableAccounts( + accounts: DurableMap, + pending: DurableMap, + opts: { apiUrl?: string; keyMaterial: string | Buffer; now?: () => number; fetchImpl?: typeof fetch }, +): MemorableAccounts { + const apiUrl = (opts.apiUrl || DEFAULT_MEMORABLE_API_URL).replace(/\/$/, ""); + const clock = opts.now ?? (() => Date.now()); + const http = opts.fetchImpl ?? fetch; + const secretKey = deriveConnectorKey(opts.keyMaterial, "memorable-accounts"); + const key = (scope: ScopeId) => scopeStorageKey(scope); + const label = (scope: ScopeId) => `qm ${parseScopeId(scope).kind ?? "scope"} ${hashId([scope], 8)}`; + const claim = async (scope: ScopeId): Promise => (await pending.take(key(scope))) !== null; + const settledElsewhere = async (scope: ScopeId): Promise => { + const existing = await accounts.get(key(scope)); + return existing + ? { status: "connected", orgId: existing.orgId, orgName: existing.orgName, keyId: existing.keyId } + : { status: "expired" }; + }; + const readKey = (account: MemorableAccount): string | null => { + try { + return decryptSecret(account.apiKeyEnc, secretKey); + } catch { + return null; + } + }; + + return { + async start(scope, startOpts = {}) { + const existing = await accounts.get(key(scope)); + if (existing && !startOpts.force) return { status: "already_connected", orgName: existing.orgName }; + + const live = await pending.get(key(scope)); + if (live && clock() < live.expiresAt && !startOpts.force) { + return { + status: "started", + userCode: live.userCode, + verificationUri: live.verificationUri, + verificationUriComplete: live.verificationUriComplete, + intervalMs: live.intervalMs, + expiresAt: live.expiresAt, + }; + } + + let body: DeviceStartBody; + try { + const response = await http(`${apiUrl}/v1/device/code`, { + method: "POST", + headers: { "content-type": "application/json", "x-memorable-client": "qm" }, + body: JSON.stringify({ hostname: label(scope) }), + signal: AbortSignal.timeout(START_TIMEOUT_MS), + }); + body = (await response.json()) as DeviceStartBody; + if (!response.ok) { + return { status: "unavailable", detail: str(body.detail) || str(body.error) || `http ${response.status}` }; + } + } catch (e) { + return { status: "unavailable", detail: (e as Error).message.slice(0, 200) }; + } + + const deviceCode = str(body.device_code, 128); + const userCode = str(body.user_code, 32); + const verificationUriComplete = str(body.verification_uri_complete, 500); + if (!deviceCode || !userCode || !verificationUriComplete) { + return { status: "unavailable", detail: "the sign-in service returned an unusable response" }; + } + const expiresIn = typeof body.expires_in === "number" ? body.expires_in : 600; + const interval = typeof body.interval === "number" ? body.interval : 5; + const record: PendingConnect = { + scopeId: scope, + deviceCode, + userCode, + verificationUri: str(body.verification_uri, 500) || verificationUriComplete, + verificationUriComplete, + intervalMs: Math.max(MIN_POLL_INTERVAL_MS, interval * 1000), + expiresAt: clock() + expiresIn * 1000, + }; + await pending.put(key(scope), record); + return { + status: "started", + userCode, + verificationUri: record.verificationUri, + verificationUriComplete, + intervalMs: record.intervalMs, + expiresAt: record.expiresAt, + }; + }, + + async poll(scope) { + const record = await pending.get(key(scope)); + if (!record) return { status: "none" }; + if (clock() >= record.expiresAt) { + await pending.delete(key(scope)); + return { status: "expired" }; + } + + let body: DevicePollBody; + try { + const response = await http(`${apiUrl}/v1/device/token`, { + method: "POST", + headers: { "content-type": "application/json", "x-memorable-client": "qm" }, + body: JSON.stringify({ device_code: record.deviceCode }), + signal: AbortSignal.timeout(POLL_TIMEOUT_MS), + }); + if (response.status >= 500 || response.status === 429) { + return { status: "unavailable", detail: `the sign-in service answered ${response.status}` }; + } + if (!response.ok) { + await pending.delete(key(scope)); + return { status: "expired" }; + } + body = (await response.json()) as DevicePollBody; + } catch (e) { + return { status: "unavailable", detail: (e as Error).message.slice(0, 200) }; + } + + const status = str(body.status, 32); + if (status === "pending") { + return { + status: "pending", + userCode: record.userCode, + verificationUriComplete: record.verificationUriComplete, + }; + } + if (status === "denied") { + await pending.delete(key(scope)); + return { status: "denied" }; + } + if (status !== "approved") { + return (await claim(scope)) ? { status: "expired" } : await settledElsewhere(scope); + } + + const apiKey = typeof body.api_key === "string" ? body.api_key : ""; + if (!apiKey || apiKey.length > MAX_API_KEY_CHARS) { + await pending.delete(key(scope)); + return { status: "unavailable", detail: "the sign-in was approved but the key it returned is unusable" }; + } + if (!(await claim(scope))) return await settledElsewhere(scope); + const account: MemorableAccount = { + scopeId: scope, + apiKeyEnc: encryptSecret(apiKey, secretKey), + keyId: str(body.key_id, 64), + orgId: str(body.org_id, 64), + orgName: str(body.org_name, 120) || "your organisation", + connectedAt: clock(), + }; + await accounts.put(key(scope), account); + return { status: "connected", orgId: account.orgId, orgName: account.orgName, keyId: account.keyId }; + }, + + async keyFor(scope) { + const exact = await accounts.get(key(scope)); + return exact ? readKey(exact) : null; + }, + + async disconnect(scope) { + await pending.delete(key(scope)); + const had = await accounts.get(key(scope)); + if (!had) return false; + await accounts.delete(key(scope)); + return true; + }, + + async connected() { + const all = await accounts.all(); + return all.map(({ apiKeyEnc: _apiKeyEnc, ...rest }) => rest); + }, + }; +} diff --git a/src/memorable/capture.ts b/src/memorable/capture.ts new file mode 100644 index 000000000..f399c28aa --- /dev/null +++ b/src/memorable/capture.ts @@ -0,0 +1,116 @@ +import { createHash } from "node:crypto"; +import type { SessionEntry } from "../types.ts"; +import { clampChars, stripTerminalControl } from "./inject.ts"; + +export interface MemorableToolCall { + name: string; + input: Record; + result?: { ok: boolean; exit_code?: number }; +} + +export interface MemorableWorkflow { + workflow_id: string; + prompt: string; + tool_calls: MemorableToolCall[]; +} + +export interface MemorableCapture { + session_id: string; + scope_id: string; + workflows: MemorableWorkflow[]; +} + +const MAX_WORKFLOW_ID_CHARS = 200; +const WORKFLOW_ID_DIGEST_CHARS = 16; +const MAX_PROMPT_CHARS = 16_000; +const MAX_TOOL_INPUT_CHARS = 32_000; + +function workflowId(sessionId: string, seq: number): string { + const raw = `${sessionId}-${seq}`; + const safe = raw.replace(/[^A-Za-z0-9._-]/g, "-"); + if (safe === raw && safe.length <= MAX_WORKFLOW_ID_CHARS) return safe; + const digest = createHash("sha256") + .update(`${sessionId}\u0000${seq}`) + .digest("hex") + .slice(0, WORKFLOW_ID_DIGEST_CHARS); + return `${safe.slice(0, MAX_WORKFLOW_ID_CHARS - WORKFLOW_ID_DIGEST_CHARS - 1)}-${digest}`; +} + +function cleanPrompt(text: string): string { + const clean = stripTerminalControl(text).trim(); + return clean.length > MAX_PROMPT_CHARS ? clampChars(clean, MAX_PROMPT_CHARS).trimEnd() : clean; +} + +function capInput(input: Record): Record { + let capped: Record | null = null; + for (const [key, value] of Object.entries(input)) { + if (typeof value === "string" && value.length > MAX_TOOL_INPUT_CHARS) { + capped ??= { ...input }; + capped[key] = clampChars(value, MAX_TOOL_INPUT_CHARS); + } + } + return capped ?? input; +} + +function securityTainted(entry: SessionEntry): boolean { + return (entry.payload as { securityTainted?: unknown } | null)?.securityTainted === true; +} + +function callKey(call: MemorableToolCall): string { + return `${call.name}\u0000${JSON.stringify(call.input)}`; +} + +export function worthOffering(workflow: MemorableWorkflow): boolean { + const calls = workflow.tool_calls; + if (calls.length < 2) return false; + const first = callKey(calls[0]!); + return calls.some((call) => callKey(call) !== first); +} + +export function captureSession(sessionId: string, entries: SessionEntry[]): MemorableCapture { + let scopeId = ""; + const outcomes = new Map>(); + for (const entry of entries) { + if (entry.type !== "tool_result" || securityTainted(entry)) continue; + const payload = entry.payload as Record | null; + if (!payload || typeof payload.callId !== "string") continue; + const ok = payload.isError !== true; + const code = payload.tool === "execute" && typeof payload.code === "number" ? payload.code : undefined; + const outcome = { ok, ...(code !== undefined ? { exit_code: code } : {}) }; + const queue = outcomes.get(payload.callId); + if (queue) queue.push(outcome); + else outcomes.set(payload.callId, [outcome]); + } + const workflows: MemorableWorkflow[] = []; + let current: MemorableWorkflow = { workflow_id: workflowId(sessionId, 0), prompt: "", tool_calls: [] }; + const close = () => { + if (current.tool_calls.length) workflows.push(current); + }; + for (const entry of entries) { + if (!scopeId && entry.scopeLabel) scopeId = entry.scopeLabel; + if (securityTainted(entry)) { + if (entry.type === "user") { + close(); + current = { workflow_id: workflowId(sessionId, entry.seq), prompt: "", tool_calls: [] }; + } + continue; + } + if (entry.type === "user") { + const text = (entry.payload as { text?: unknown } | null)?.text; + if (typeof text !== "string") continue; + const prompt = cleanPrompt(text); + if (!prompt) continue; + close(); + current = { workflow_id: workflowId(sessionId, entry.seq), prompt, tool_calls: [] }; + continue; + } + if (entry.type !== "tool_call") continue; + const payload = entry.payload as Record | null; + if (!payload || typeof payload.tool !== "string") continue; + const { tool, callId, ...input } = payload; + const outcome = typeof callId === "string" ? outcomes.get(callId)?.shift() : undefined; + current.tool_calls.push({ name: tool, input: capInput(input), ...(outcome ? { result: outcome } : {}) }); + } + close(); + return { session_id: sessionId, scope_id: scopeId, workflows }; +} diff --git a/src/memorable/consent.ts b/src/memorable/consent.ts new file mode 100644 index 000000000..6a437667a --- /dev/null +++ b/src/memorable/consent.ts @@ -0,0 +1,46 @@ +import { spawn } from "node:child_process"; + +const CONSENT_TIMEOUT_MS = 30_000; + +export type ConsentMode = "read-write" | "read-only" | "deny"; + +export type ConsentResult = { ok: true; mode: ConsentMode } | { ok: false; reason: string }; + +const VERB: Record = { + "read-write": "enable", + "read-only": "disable", + deny: "forget", +}; + +export function parseConsentMode(value: unknown): ConsentMode | null { + return value === "read-write" || value === "read-only" || value === "deny" ? value : null; +} + +export function setConsent( + bin: string, + scopeId: string, + mode: ConsentMode, + opts?: { env: NodeJS.ProcessEnv; apiKey?: string }, +): Promise { + return new Promise((resolve) => { + const [cmd = "memorable", ...preArgs] = bin.split(" ").filter(Boolean); + const child = spawn(cmd, [...preArgs, VERB[mode], "--scope", scopeId], { + stdio: ["ignore", "ignore", "ignore"], + ...(opts ? { env: { ...opts.env, ...(opts.apiKey ? { MEMORABLE_API_KEY: opts.apiKey } : {}) } } : {}), + }); + let settled = false; + const finish = (result: ConsentResult) => { + if (settled) return; + settled = true; + clearTimeout(timer); + resolve(result); + }; + const timer = setTimeout(() => { + child.kill(); + finish({ ok: false, reason: "timeout" }); + }, CONSENT_TIMEOUT_MS); + timer.unref(); + child.on("error", (e) => finish({ ok: false, reason: e.message.slice(0, 200) })); + child.on("exit", (code) => finish(code === 0 ? { ok: true, mode } : { ok: false, reason: `exit ${code}` })); + }); +} diff --git a/src/memorable/inject.ts b/src/memorable/inject.ts new file mode 100644 index 000000000..0025d016e --- /dev/null +++ b/src/memorable/inject.ts @@ -0,0 +1,83 @@ +import { spawn } from "node:child_process"; + +const INJECT_TIMEOUT_MS = 15_000; +const MAX_INJECTION_CHARS = 8_000; +const MAX_TASK_CHARS = 16_000; +const MAX_STDOUT_BYTES = 256 * 1024; +const ENVELOPE_PREFIX = ""; +const ESCAPE_SEQUENCES = new RegExp( + [ + "\\x1b\\[[0-9;:?]*[ -/]*[@-~]", + "\\x1b\\][^\\x07\\x1b]*(?:\\x07|\\x1b\\\\)?", + "\\x1b[PX^_][^\\x1b]*(?:\\x1b\\\\)?", + "\\x1b[()*+][@-~]", + "\\x1b[\\-./][@-~]", + "\\x1b#[0-9]", + "\\x1b%[@G]", + "\\x1b [@-~]", + "\\x1b[@-Z\\\\-_]", + "\\x1b[0-9:;<=>?]", + "\\x1b", + ].join("|"), + "g", +); + +const CONTROL_CHARS = /[\x00-\x08\x0b-\x1a\x1c-\x1f\x7f]/g; + +export function stripTerminalControl(text: string): string { + return text.replace(ESCAPE_SEQUENCES, "").replace(CONTROL_CHARS, ""); +} + +export function clampChars(text: string, max: number): string { + if (text.length <= max) return text; + const cut = text.slice(0, max); + const last = cut.charCodeAt(cut.length - 1); + return last >= 0xd800 && last <= 0xdbff ? cut.slice(0, -1) : cut; +} + +export function memorableInject( + bin: string, + scopeId: string, + task: string, + opts?: { env: NodeJS.ProcessEnv; apiKey?: string }, +): Promise { + return new Promise((resolve) => { + const [cmd = "memorable", ...preArgs] = bin.split(" ").filter(Boolean); + const child = spawn(cmd, [...preArgs, "inject", "--scope", scopeId], { + stdio: ["pipe", "pipe", "ignore"], + ...(opts ? { env: { ...opts.env, ...(opts.apiKey ? { MEMORABLE_API_KEY: opts.apiKey } : {}) } } : {}), + }); + child.unref(); + let chunks: Buffer[] = []; + let bytes = 0; + let settled = false; + const finish = (value: string | null) => { + if (settled) return; + settled = true; + chunks = []; + clearTimeout(timer); + resolve(value); + }; + const timer = setTimeout(() => { + child.kill(); + finish(null); + }, INJECT_TIMEOUT_MS); + child.on("error", () => finish(null)); + child.stdout.on("data", (c: Buffer) => { + bytes += c.length; + if (bytes > MAX_STDOUT_BYTES) { + child.kill(); + finish(null); + return; + } + chunks.push(c); + }); + child.on("exit", (code) => { + const text = stripTerminalControl(Buffer.concat(chunks).toString("utf8")).trim(); + const usable = code === 0 && text.startsWith(ENVELOPE_PREFIX) && text.length <= MAX_INJECTION_CHARS; + finish(usable ? text : null); + }); + child.stdin.on("error", () => {}); + child.stdin.end(clampChars(stripTerminalControl(task), MAX_TASK_CHARS)); + }); +} diff --git a/src/memorable/relay.ts b/src/memorable/relay.ts new file mode 100644 index 000000000..3b9613361 --- /dev/null +++ b/src/memorable/relay.ts @@ -0,0 +1,66 @@ +import { spawn } from "node:child_process"; +import { worthOffering, type MemorableCapture } from "./capture.ts"; + +const RELAY_TIMEOUT_MS = 120_000; +const MAX_RELAY_STDOUT = 8_192; + +function refusalReason(stdout: string): string | null { + for (const line of stdout.split("\n")) { + const text = line.trim(); + if (!text.startsWith("{")) continue; + try { + const parsed = JSON.parse(text) as { error?: unknown; mode?: unknown }; + if (typeof parsed.error === "string") { + return typeof parsed.mode === "string" ? `${parsed.error} (consent ${parsed.mode})` : parsed.error; + } + } catch { + continue; + } + } + return null; +} + +export type RelayOutcome = { ok: true } | { ok: false; reason: string }; + +export function relayRecord( + bin: string, + capture: MemorableCapture, + timeoutMs: number = RELAY_TIMEOUT_MS, + opts?: { env: NodeJS.ProcessEnv; apiKey?: string }, +): Promise { + return new Promise((resolve) => { + const workflows = capture.workflows.filter(worthOffering); + if (!workflows.length) { + resolve({ ok: true }); + return; + } + const [cmd = "memorable", ...preArgs] = bin.split(" ").filter(Boolean); + const child = spawn(cmd, [...preArgs, "record", "--scope", capture.scope_id, "-"], { + stdio: ["pipe", "pipe", "ignore"], + ...(opts ? { env: { ...opts.env, ...(opts.apiKey ? { MEMORABLE_API_KEY: opts.apiKey } : {}) } } : {}), + }); + child.unref(); + let settled = false; + let out = ""; + const finish = (outcome: RelayOutcome) => { + if (settled) return; + settled = true; + clearTimeout(timer); + resolve(outcome); + }; + const timer = setTimeout(() => { + child.kill(); + finish({ ok: false, reason: "timeout" }); + }, timeoutMs); + timer.unref(); + child.stdout.on("data", (c: Buffer) => { + if (out.length < MAX_RELAY_STDOUT) out += c.toString("utf8"); + }); + child.on("error", (e) => finish({ ok: false, reason: e.message.slice(0, 200) })); + child.on("exit", (code) => + finish(code === 0 ? { ok: true } : { ok: false, reason: refusalReason(out) ?? `exit ${code}` }), + ); + child.stdin.on("error", () => {}); + child.stdin.end(JSON.stringify({ ...capture, workflows })); + }); +} diff --git a/src/wiring.ts b/src/wiring.ts index 7b8d4c749..1f1705fd1 100644 --- a/src/wiring.ts +++ b/src/wiring.ts @@ -200,6 +200,15 @@ import { createMemoryTaskStore } from "./tasks/memory-task-store.ts"; import { createPostgresTaskStore } from "./tasks/postgres-task-store.ts"; import type { TaskStore } from "./tasks/task-store.ts"; import { createMemoryStrategy } from "./memory/strategy.ts"; +import { captureSession } from "./memorable/capture.ts"; +import { memorableInject } from "./memorable/inject.ts"; +import { relayRecord } from "./memorable/relay.ts"; +import { + createMemorableAccounts, + type MemorableAccount, + type MemorableAccounts, + type PendingConnect, +} from "./memorable/accounts.ts"; import { createOrchestrator, egressClaimAllowingControlPlane, type OrchestratorDeps } from "./core/orchestrator.ts"; import { mintCapabilityToken, CAPABILITY_TTL_MS, EGRESS_PROXY_AUD } from "./auth/capability-token.ts"; import { createControlService } from "./api/control-service.ts"; @@ -343,6 +352,7 @@ export interface BuiltApp { slackInstallation: SlackInstallationStore; resolveClient: OAuthClientResolver; consentLinks: ConsentLinkStore; + memorableAccounts: MemorableAccounts | undefined; secretDrops: SecretDropStore; modelGateway: ModelGateway; modelCredentials: ModelCredentialStore; @@ -396,6 +406,8 @@ export interface BuiltApp { slackCore: SlackCoreClient; } +const MEMORABLE_RELAY_ENTRY_WINDOW = 2_000; + export function buildApp( config: Config, overrides: { @@ -756,6 +768,14 @@ export function buildApp( : undefined; const connectorTokens = withOperatorTokenFallback(credentialStore, config.egressServiceHosts ?? [], secretSource); const consentLinks: ConsentLinkStore = createConsentLinkStore(artifactMap("consent_links")); + const memorableAccounts: MemorableAccounts | undefined = + config.memorableEnabled && keychainKeyMaterial + ? createMemorableAccounts( + artifactMap("memorable_accounts"), + artifactMap("memorable_device_codes"), + { apiUrl: config.memorableApiUrl, keyMaterial: keychainKeyMaterial }, + ) + : undefined; const secretDrops: SecretDropStore = createSecretDropStore(artifactMap("secret_drops")); const modelGateway = createModelGateway(); @@ -1122,6 +1142,17 @@ export function buildApp( ...(config.publicUrl ? { webhookPublicUrl: config.publicUrl } : {}), memoryPolicy: { recall: config.memoryRecall, capture: config.memoryCapture }, memoryStrategy, + ...(config.memorableEnabled && config.memoryRecall !== "off" + ? { + memorable: async (scopeId: ScopeId, task: string) => { + const apiKey = (await memorableAccounts?.keyFor(scopeId).catch(() => null)) ?? undefined; + return memorableInject(config.memorableBin, scopeId, task, { + env: config.memorableProcessEnv, + ...(apiKey ? { apiKey } : {}), + }); + }, + } + : {}), skills, skillBundles, skillsReady, @@ -1352,6 +1383,32 @@ export function buildApp( }); })().catch(swallowAs("session-state: terminal emit", undefined)); }); + if (config.memorableEnabled && config.memoryCapture !== "off") { + runs.onTerminal((run) => { + void (async () => { + if (await runs.activeForThread(run.sessionId)) return; + const session = await sessions.getByThread(run.sessionId); + if (!session) return; + const entries = await sessions.getEntries(session.id, { limit: MEMORABLE_RELAY_ENTRY_WINDOW }); + const capture = captureSession(session.id, entries); + if (!capture.scope_id) capture.scope_id = session.scopeId; + const apiKey = (await memorableAccounts?.keyFor(capture.scope_id).catch(() => null)) ?? undefined; + const outcome = await relayRecord(config.memorableBin, capture, undefined, { + env: config.memorableProcessEnv, + ...(apiKey ? { apiKey } : {}), + }); + if (!outcome.ok) { + errors.record({ + category: "memory", + code: "memorable_relay_refused", + message: outcome.reason, + scopeLabel: capture.scope_id, + sessionId: session.id, + }); + } + })().catch(swallowAs("memorable: record relay", undefined)); + }); + } let lastSignalPrune = 0; const orphanedSignalSweeper = createSweeper( async () => { @@ -1613,6 +1670,7 @@ export function buildApp( slackInstallation, resolveClient, consentLinks, + memorableAccounts, secretDrops, modelGateway, modelCredentials, @@ -1702,6 +1760,10 @@ export function serverDeps( ...(slackEnvBotToken ? { slackEnvBotToken } : {}), resolveClient: built.resolveClient, consentLinks: built.consentLinks, + memorableAccounts: built.memorableAccounts, + ...(config.memorableEnabled + ? { memorableBin: config.memorableBin, memorableProcessEnv: config.memorableProcessEnv } + : {}), secretDrops: built.secretDrops, ...(built.fireDropResolution ? { fireDropResolution: built.fireDropResolution } : {}), ...(config.apiBaseUrl ? { apiBaseUrl: config.apiBaseUrl } : {}), diff --git a/test/config.test.ts b/test/config.test.ts index 4911d65ad..62ec95b74 100644 --- a/test/config.test.ts +++ b/test/config.test.ts @@ -187,6 +187,29 @@ test("boolEnv: one vocabulary for every boolean env knob", () => { for (const v of [undefined, "", "2", "enabled"]) assert.equal(boolEnv(v), undefined, String(v)); }); +test("MEMORABLE is off by default and QM_MEMORABLE kills it in the shared vocabulary", () => { + assert.equal(loadConfig({}).memorableEnabled, false); + assert.equal(loadConfig({}).memorableBin, "memorable"); + for (const on of ["1", "true", "yes", "on", "TRUE", " On "]) { + assert.equal(loadConfig({ MEMORABLE: on }).memorableEnabled, true, `MEMORABLE=${on}`); + for (const kill of ["0", "false", "no", "off", "none", "OFF", " off "]) { + assert.equal( + loadConfig({ MEMORABLE: on, QM_MEMORABLE: kill }).memorableEnabled, + false, + `MEMORABLE=${on} QM_MEMORABLE=${kill}`, + ); + } + } + for (const off of ["0", "false", "no", "off", "none", ""]) { + assert.equal(loadConfig({ MEMORABLE: off }).memorableEnabled, false, `MEMORABLE=${off}`); + } + assert.throws(() => loadConfig({ MEMORABLE: "2" }), /MEMORABLE="2" is not a recognized boolean/); + assert.throws( + () => loadConfig({ MEMORABLE: "1", QM_MEMORABLE: "2" }), + /QM_MEMORABLE="2" is not a recognized boolean/, + ); +}); + test("every boolean knob accepts the shared vocabulary (off means off)", () => { const off = loadConfig({ SEED_SKILLS: "off", EXECUTE_SCRATCH: "off", REACH_EXEC: "off", PI_CAPTURE_REQUESTS: "off" }); assert.equal(off.seedSkills, false); diff --git a/test/memorable-accounts.test.ts b/test/memorable-accounts.test.ts new file mode 100644 index 000000000..d47324251 --- /dev/null +++ b/test/memorable-accounts.test.ts @@ -0,0 +1,303 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { createMemoryMap } from "../src/persistence/durable-map.ts"; +import { createMemorableAccounts, type MemorableAccount, type PendingConnect } from "../src/memorable/accounts.ts"; + +const ORG = "org:acme"; +const ME = "personal:U1"; + +type Call = { url: string; body: unknown }; + +function harness(responses: Array<{ status?: number; body: unknown } | Error>, opts: { now?: () => number } = {}) { + const calls: Call[] = []; + const queue = [...responses]; + const fetchImpl = (async (url: string | URL, init?: RequestInit) => { + calls.push({ url: String(url), body: JSON.parse(String(init?.body ?? "null")) }); + const next = queue.shift(); + if (!next) throw new Error("no response queued"); + if (next instanceof Error) throw next; + return { + ok: (next.status ?? 200) < 400, + status: next.status ?? 200, + json: async () => next.body, + } as Response; + }) as unknown as typeof fetch; + + const accounts = createMemoryMap(); + const pending = createMemoryMap(); + const store = createMemorableAccounts(accounts, pending, { + apiUrl: "https://api.test", + keyMaterial: "test-key-material-0123456789abcdef", + fetchImpl, + ...(opts.now ? { now: opts.now } : {}), + }); + return { store, calls, accounts, pending }; +} + +const started = { + device_code: "d".repeat(64), + user_code: "ABCD-EFGH", + verification_uri: "https://dash.test/device", + verification_uri_complete: "https://dash.test/device?code=ABCD-EFGH", + expires_in: 600, + interval: 5, +}; + +const approved = { + status: "approved", + api_key: "mk_secret", + key_id: "key_1", + org_id: "org_abc", + org_name: "Acme", +}; + +test("start opens a device authorization and remembers it against the scope", async () => { + const { store, calls, pending } = harness([{ body: started }]); + const result = await store.start(ME); + assert.equal(result.status, "started"); + assert.equal(calls[0]?.url, "https://api.test/v1/device/code"); + assert.match(String((calls[0]?.body as { hostname?: string })?.hostname), /^qm personal [0-9a-f]{8}$/); + assert.equal(JSON.stringify(calls[0]?.body).includes("U1"), false); + if (result.status !== "started") throw new Error("unreachable"); + assert.equal(result.verificationUriComplete, started.verification_uri_complete); + const stored = await pending.all(); + assert.equal(stored.length, 1); + assert.equal(stored[0]?.scopeId, ME); +}); + +test("start does not re-open an authorization for an already connected scope", async () => { + const { store, calls } = harness([{ body: started }, { body: approved }, { body: started }]); + await store.start(ME); + await store.poll(ME); + const again = await store.start(ME); + assert.equal(again.status, "already_connected"); + assert.equal(calls.length, 2); +}); + +test("start with force re-opens an authorization for a connected scope", async () => { + const { store, calls } = harness([{ body: started }, { body: approved }, { body: started }]); + await store.start(ME); + await store.poll(ME); + const again = await store.start(ME, { force: true }); + assert.equal(again.status, "started"); + assert.equal(calls.length, 3); +}); + +test("a sign-in service that cannot be reached stores nothing", async () => { + const { store, pending } = harness([new Error("getaddrinfo ENOTFOUND")]); + const result = await store.start(ME); + assert.equal(result.status, "unavailable"); + assert.deepEqual(await pending.all(), []); +}); + +test("a response missing the code is refused rather than stored", async () => { + const { store, pending } = harness([{ body: { user_code: "ABCD-EFGH" } }]); + const result = await store.start(ME); + assert.equal(result.status, "unavailable"); + assert.deepEqual(await pending.all(), []); +}); + +test("polling before approval returns the same code the human was given", async () => { + const { store } = harness([{ body: started }, { body: { status: "pending" } }]); + await store.start(ME); + const result = await store.poll(ME); + assert.equal(result.status, "pending"); + if (result.status !== "pending") throw new Error("unreachable"); + assert.equal(result.userCode, started.user_code); +}); + +test("approval stores the key against the scope and clears the pending code", async () => { + const { store, accounts, pending } = harness([{ body: started }, { body: approved }]); + await store.start(ME); + const result = await store.poll(ME); + assert.equal(result.status, "connected"); + assert.equal(await store.keyFor(ME), "mk_secret"); + assert.deepEqual(await pending.all(), []); + const stored = await accounts.all(); + assert.equal(stored[0]?.orgName, "Acme"); +}); + +test("a denied sign-in clears the pending code and leaves no key", async () => { + const { store, pending } = harness([{ body: started }, { body: { status: "denied" } }]); + await store.start(ME); + assert.equal((await store.poll(ME)).status, "denied"); + assert.equal(await store.keyFor(ME), null); + assert.deepEqual(await pending.all(), []); +}); + +test("a service outage is reported as unavailable and keeps the code alive", async () => { + const { store, pending } = harness([{ body: started }, { status: 502, body: {} }, { body: approved }]); + await store.start(ME); + const outage = await store.poll(ME); + assert.equal(outage.status, "unavailable"); + assert.equal((await pending.all()).length, 1); + assert.equal((await store.poll(ME)).status, "connected"); + assert.equal(await store.keyFor(ME), "mk_secret"); +}); + +test("a rate limit is an outage, not a dead code", async () => { + const { store, pending } = harness([{ body: started }, { status: 429, body: {} }]); + await store.start(ME); + assert.equal((await store.poll(ME)).status, "unavailable"); + assert.equal((await pending.all()).length, 1); +}); + +test("a request the service will never accept is retired, not polled forever", async () => { + const { store, pending } = harness([{ body: started }, { status: 400, body: { error: "invalid_request" } }]); + await store.start(ME); + assert.equal((await store.poll(ME)).status, "expired"); + assert.deepEqual(await pending.all(), []); +}); + +test("a network failure while polling keeps the code alive", async () => { + const { store, pending } = harness([{ body: started }, new Error("ECONNRESET")]); + await store.start(ME); + const result = await store.poll(ME); + assert.equal(result.status, "unavailable"); + assert.equal((await pending.all()).length, 1); +}); + +test("an approval with no usable key stores nothing", async () => { + for (const bad of [ + { ...approved, api_key: "" }, + { ...approved, api_key: "m".repeat(600) }, + ]) { + const { store } = harness([{ body: started }, { body: bad }]); + await store.start(ME); + assert.equal((await store.poll(ME)).status, "unavailable"); + assert.equal(await store.keyFor(ME), null); + } +}); + +test("an expired window is cleared locally without asking the service", async () => { + let now = 1_000; + const { store, calls, pending } = harness([{ body: started }], { now: () => now }); + await store.start(ME); + now += 601_000; + assert.equal((await store.poll(ME)).status, "expired"); + assert.equal(calls.length, 1); + assert.deepEqual(await pending.all(), []); +}); + +test("polling a scope that never started reports nothing rather than an error", async () => { + const { store } = harness([]); + assert.equal((await store.poll(ME)).status, "none"); +}); + +test("a key is used only by the scope that connected it", async () => { + const { store } = harness([{ body: started }, { body: approved }]); + await store.start(ME); + await store.poll(ME); + assert.equal(await store.keyFor(ME), "mk_secret"); + for (const other of ["personal:U9", "channel:C1", ORG]) { + assert.equal(await store.keyFor(other), null, `${other} borrowed another scope's key`); + } +}); + +test("disconnect removes the key and any authorization still in flight", async () => { + const { store, pending } = harness([{ body: started }, { body: approved }, { body: started }]); + await store.start(ME); + await store.poll(ME); + await store.start(ME, { force: true }); + assert.equal(await store.disconnect(ME), true); + assert.equal(await store.keyFor(ME), null); + assert.deepEqual(await pending.all(), []); + assert.equal(await store.disconnect(ME), false); +}); + +test("listing connected accounts never carries a key", async () => { + const { store } = harness([{ body: started }, { body: approved }]); + await store.start(ME); + await store.poll(ME); + const listed = await store.connected(); + assert.equal(listed.length, 1); + assert.equal(Object.hasOwn(listed[0] ?? {}, "apiKeyEnc"), false); + assert.equal(JSON.stringify(listed).includes("mk_secret"), false); +}); + +test("a scope id that is not storage-safe still round-trips", async () => { + const odd = "channel:C/1 with spaces"; + const { store } = harness([{ body: started }, { body: approved }]); + await store.start(odd); + await store.poll(odd); + assert.equal(await store.keyFor(odd), "mk_secret"); +}); + +test("the stored row never carries the key in the clear", async () => { + const { store, accounts } = harness([{ body: started }, { body: approved }]); + await store.start(ME); + await store.poll(ME); + assert.equal(JSON.stringify(await accounts.all()).includes("mk_secret"), false); +}); + +test("a row written under different key material reads as no key, not a crash", async () => { + const { store, accounts } = harness([{ body: started }, { body: approved }]); + await store.start(ME); + await store.poll(ME); + const rows = await accounts.entries(); + const [id, row] = rows[0] ?? []; + assert.ok(id && row); + await accounts.put(id, { ...row, apiKeyEnc: "v2:aaaa:bbbb:cccc" }); + assert.equal(await store.keyFor(ME), null); +}); + +test("a disconnect during an in-flight approval does not leave a live key behind", async () => { + let release: (() => void) | undefined; + const held = new Promise((r) => { + release = r; + }); + let polls = 0; + const fetchImpl = (async (url: string | URL) => { + if (String(url).endsWith("/v1/device/code")) { + return { ok: true, status: 200, json: async () => started } as Response; + } + if (++polls === 1) await held; + return { ok: true, status: 200, json: async () => approved } as Response; + }) as unknown as typeof fetch; + + const store = createMemorableAccounts(createMemoryMap(), createMemoryMap(), { + apiUrl: "https://api.test", + keyMaterial: "test-key-material-0123456789abcdef", + fetchImpl, + }); + + await store.start(ME); + const inFlight = store.poll(ME); + await store.disconnect(ME); + release?.(); + const result = await inFlight; + + assert.equal(result.status, "expired"); + assert.equal(await store.keyFor(ME), null); +}); + +test("of two concurrent polls, the one that loses the claim still reports the truth", async () => { + const { store } = harness([{ body: started }, { body: approved }, { body: approved }]); + await store.start(ME); + const [a, b] = await Promise.all([store.poll(ME), store.poll(ME)]); + assert.deepEqual([a.status, b.status].sort(), ["connected", "connected"]); + assert.equal(await store.keyFor(ME), "mk_secret"); +}); + +test("starting again hands back the code the person is already looking at", async () => { + const { store, calls } = harness([{ body: started }]); + const first = await store.start(ME); + const second = await store.start(ME); + assert.equal(calls.length, 1); + assert.deepEqual(first, second); +}); + +test("two scope ids that are not storage-safe stay distinct", async () => { + const { store } = harness([ + { body: started }, + { body: approved }, + { body: started }, + { body: { ...approved, api_key: "mk_two" } }, + ]); + await store.start("channel:C/1 one"); + await store.poll("channel:C/1 one"); + await store.start("channel:C/1 two"); + await store.poll("channel:C/1 two"); + assert.equal(await store.keyFor("channel:C/1 one"), "mk_secret"); + assert.equal(await store.keyFor("channel:C/1 two"), "mk_two"); +}); diff --git a/test/memorable-capture.test.ts b/test/memorable-capture.test.ts new file mode 100644 index 000000000..ae019db35 --- /dev/null +++ b/test/memorable-capture.test.ts @@ -0,0 +1,255 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { captureSession, type MemorableToolCall } from "../src/memorable/capture.ts"; +import type { SessionEntry } from "../src/types.ts"; + +const WORKER_WORKFLOW_ID = /^[A-Za-z0-9._-]{1,200}$/; + +function entry(type: SessionEntry["type"], payload: unknown, seq: number): SessionEntry { + return { sessionId: "s1", seq, parentSeq: null, type, payload, scopeLabel: "personal:U1", createdAt: seq }; +} + +function work(seq: number, tag: string): SessionEntry[] { + return [ + entry("tool_call", { tool: "execute", callId: `${tag}a`, command: `./${tag}.sh` }, seq), + entry("tool_call", { tool: "write", callId: `${tag}b`, path: `${tag}.js` }, seq + 1), + ]; +} + +test("captureSession extracts prompt, scope, and tool calls in order", () => { + const entries: SessionEntry[] = [ + entry("user", { text: "Fix the failing order tests" }, 1), + entry("tool_call", { tool: "execute", callId: "c1", command: "./test.sh" }, 2), + entry("tool_result", { tool: "execute", callId: "c1", isError: true, code: 1, result: "fail" }, 3), + entry("tool_call", { tool: "write", callId: "c2", path: "src/orders/validate.js", data: "x" }, 4), + entry("assistant", { text: "done" }, 5), + ]; + const capture = captureSession("s1", entries); + assert.equal(capture.session_id, "s1"); + assert.equal(capture.scope_id, "personal:U1"); + assert.equal(capture.workflows.length, 1); + const workflow = capture.workflows[0]!; + assert.equal(workflow.workflow_id, "s1-1"); + assert.equal(workflow.prompt, "Fix the failing order tests"); + const calls: MemorableToolCall[] = workflow.tool_calls; + assert.equal(calls.length, 2); + assert.deepEqual( + calls.map((c) => c.name), + ["execute", "write"], + ); + assert.deepEqual(calls[0]?.input, { command: "./test.sh" }); + assert.deepEqual(calls[1]?.input, { path: "src/orders/validate.js", data: "x" }); +}); + +test("captureSession cuts one workflow per prompt, not one per session", () => { + const entries: SessionEntry[] = [ + entry("user", { text: "Fix the failing order tests" }, 1), + ...work(2, "t"), + entry("tool_result", { tool: "execute", callId: "ta", isError: false, code: 0 }, 4), + entry("assistant", { text: "done" }, 5), + entry("user", { text: "now bump the version and tag it" }, 6), + entry("tool_call", { tool: "write", callId: "c2", path: "package.json" }, 7), + entry("tool_call", { tool: "execute", callId: "c3", command: "git tag v2" }, 8), + entry("tool_result", { tool: "execute", callId: "c3", isError: false, code: 0 }, 9), + ]; + const { workflows } = captureSession("s1", entries); + assert.equal(workflows.length, 2); + assert.deepEqual( + workflows.map((w) => w.workflow_id), + ["s1-1", "s1-6"], + ); + assert.equal(workflows[0]?.prompt, "Fix the failing order tests"); + assert.equal(workflows[1]?.prompt, "now bump the version and tag it"); + assert.deepEqual( + workflows[0]?.tool_calls.map((c) => c.name), + ["execute", "write"], + ); + assert.deepEqual( + workflows[1]?.tool_calls.map((c) => c.name), + ["write", "execute"], + ); +}); + +test("captureSession drops a prompt that produced no tool calls", () => { + const entries: SessionEntry[] = [ + entry("user", { text: "hey, what does this repo do?" }, 1), + entry("assistant", { text: "it is a harness" }, 2), + entry("user", { text: "ok, fix the order tests" }, 3), + ...work(4, "t"), + ]; + const { workflows } = captureSession("s1", entries); + assert.equal(workflows.length, 1); + assert.equal(workflows[0]?.workflow_id, "s1-3"); + assert.equal(workflows[0]?.prompt, "ok, fix the order tests"); +}); + +test("captureSession keeps tool calls that precede the first prompt", () => { + const entries: SessionEntry[] = [...work(1, "boot"), entry("user", { text: "fix it" }, 3), ...work(4, "fix")]; + const { workflows } = captureSession("s1", entries); + assert.deepEqual( + workflows.map((w) => w.workflow_id), + ["s1-0", "s1-3"], + ); + assert.equal(workflows[0]?.prompt, ""); +}); + +test("captureSession joins outcomes by callId; ok from isError, exit_code only for execute", () => { + const entries: SessionEntry[] = [ + entry("tool_call", { tool: "execute", callId: "c1", command: "./test.sh" }, 1), + entry("tool_result", { tool: "execute", callId: "c1", isError: true, code: 1, result: "fail" }, 2), + entry("tool_call", { tool: "write", callId: "c2", path: "a.js" }, 3), + entry("tool_result", { tool: "write", callId: "c2", isError: false, result: "ok" }, 4), + entry("tool_call", { tool: "read", callId: "c3", path: "b.js" }, 5), + entry("tool_call", { tool: "execute", callId: "c4", command: "./test.sh" }, 6), + entry("tool_result", { tool: "execute", callId: "c4", isError: false, code: 0, result: "pass" }, 7), + ]; + const calls = captureSession("s1", entries).workflows[0]!.tool_calls; + assert.deepEqual(calls[0]?.result, { ok: false, exit_code: 1 }); + assert.deepEqual(calls[1]?.result, { ok: true }); + assert.equal(calls[2]?.result, undefined); + assert.deepEqual(calls[3]?.result, { ok: true, exit_code: 0 }); +}); + +test("captureSession gives each reused callId the outcome that followed it, not the last one", () => { + const entries: SessionEntry[] = [ + entry("user", { text: "go" }, 1), + entry("tool_call", { tool: "execute", callId: "c1", command: "./ok.sh" }, 2), + entry("tool_result", { tool: "execute", callId: "c1", isError: false, code: 0 }, 3), + entry("tool_call", { tool: "execute", callId: "c1", command: "./bad.sh" }, 4), + entry("tool_result", { tool: "execute", callId: "c1", isError: true, code: 1 }, 5), + ]; + const calls = captureSession("s1", entries).workflows[0]!.tool_calls; + assert.deepEqual(calls[0]?.result, { ok: true, exit_code: 0 }); + assert.deepEqual(calls[1]?.result, { ok: false, exit_code: 1 }); +}); + +test("captureSession marks quarantined results as failed, never guesses success", () => { + const entries: SessionEntry[] = [ + entry("tool_call", { tool: "execute", callId: "c1", command: "curl x" }, 1), + entry("tool_result", { tool: "execute", callId: "c1", quarantined: true, isError: true, result: "" }, 2), + entry("tool_call", { tool: "write", callId: "c2", path: "a.js" }, 3), + ]; + const calls = captureSession("s1", entries).workflows[0]!.tool_calls; + assert.deepEqual(calls[0]?.result, { ok: false }); +}); + +test("captureSession tolerates malformed payloads and missing user entry", () => { + const entries: SessionEntry[] = [ + entry("tool_call", null, 1), + entry("tool_call", { callId: "c1" }, 2), + entry("tool_call", "junk", 3), + ]; + const capture = captureSession("s1", entries); + assert.deepEqual(capture.workflows, []); +}); + +test("captureSession emits a workflow_id the extraction worker will accept", () => { + const entries: SessionEntry[] = [entry("user", { text: "fix it" }, 7), ...work(8, "t")]; + const { workflows } = captureSession("a1b2:c3/d4 e5", entries); + assert.equal(workflows.length, 1); + assert.match(workflows[0]!.workflow_id, WORKER_WORKFLOW_ID); +}); + +test("a session id longer than the id cap still yields one workflow_id per prompt", () => { + const entries: SessionEntry[] = [ + entry("user", { text: "first" }, 1), + ...work(2, "a"), + entry("user", { text: "second" }, 4), + ...work(5, "b"), + ]; + const { workflows } = captureSession("s".repeat(250), entries); + assert.equal(workflows.length, 2); + for (const workflow of workflows) assert.match(workflow.workflow_id, WORKER_WORKFLOW_ID); + assert.notEqual(workflows[0]!.workflow_id, workflows[1]!.workflow_id); +}); + +test("session ids that differ only outside the worker charset do not share a workflow_id", () => { + const entries: SessionEntry[] = [entry("user", { text: "go" }, 1), ...work(2, "t")]; + const ids = ["a:b", "a/b", "a b", "a.b", "\u{1f525}\u{1f525}", ""].map( + (raw) => captureSession(raw, entries).workflows[0]!.workflow_id, + ); + for (const id of ids) assert.match(id, WORKER_WORKFLOW_ID); + assert.equal(new Set(ids).size, ids.length); +}); + +test("captureSession strips terminal control sequences out of a prompt", () => { + const nasty = "fix \u0000 the \u001b]0;pwned\u0007 bell \u001b[31mred\u001b[0m thing"; + const entries: SessionEntry[] = [entry("user", { text: nasty }, 1), ...work(2, "t")]; + const prompt = captureSession("s1", entries).workflows[0]!.prompt; + assert.equal(/[\x00-\x08\x0b-\x1f\x7f]/.test(prompt), false); + assert.equal(prompt.includes("pwned"), false); + assert.equal(prompt.includes("]0;"), false); + assert.equal(prompt, "fix the bell red thing"); +}); + +test("a prompt made only of control characters does not cut a workflow", () => { + const entries: SessionEntry[] = [ + entry("user", { text: "real prompt" }, 1), + ...work(2, "a"), + entry("user", { text: "\u0000\u0007\u001b[0m" }, 4), + ...work(5, "b"), + ]; + const { workflows } = captureSession("s1", entries); + assert.equal(workflows.length, 1); + assert.equal(workflows[0]!.prompt, "real prompt"); + assert.equal(workflows[0]!.tool_calls.length, 4); +}); + +test("a pasted stack trace is capped instead of being relayed whole", () => { + const entries: SessionEntry[] = [entry("user", { text: "trace\n".repeat(80_000) }, 1), ...work(2, "t")]; + const prompt = captureSession("s1", entries).workflows[0]!.prompt; + assert.ok(prompt.length <= 16_000, `prompt was ${prompt.length} chars`); + assert.match(prompt, /^trace/); +}); + +test("a tool call carrying a whole file is capped before it leaves the process", () => { + const entries: SessionEntry[] = [ + entry("user", { text: "write it" }, 1), + entry("tool_call", { tool: "write", callId: "c1", path: "big.bin", data: "z".repeat(8_000_000) }, 2), + entry("tool_call", { tool: "execute", callId: "c2", command: "./verify.sh" }, 3), + ]; + const capture = captureSession("s1", entries); + assert.ok(Buffer.byteLength(JSON.stringify(capture)) < 100_000); + assert.equal((capture.workflows[0]!.tool_calls[0]!.input.data as string).length, 32_000); + assert.equal(capture.workflows[0]!.tool_calls[1]!.input.command, "./verify.sh"); +}); + +test("the prompt cap never cuts a surrogate pair in half", () => { + const entries: SessionEntry[] = [entry("user", { text: `${"y".repeat(15_999)}\u{1F600}tail` }, 1), ...work(2, "s")]; + const prompt = captureSession("s1", entries).workflows[0]!.prompt; + const last = prompt.charCodeAt(prompt.length - 1); + assert.ok(!(last >= 0xd800 && last <= 0xdbff), `prompt ends in a lone high surrogate: ${last.toString(16)}`); + assert.equal(JSON.stringify({ prompt }).includes("\\ud83d"), false); +}); + +test("the tool-input cap never cuts a surrogate pair in half", () => { + const entries: SessionEntry[] = [ + entry("user", { text: "write it" }, 1), + entry("tool_call", { tool: "write", callId: "c1", path: "big.bin", data: `${"z".repeat(31_999)}\u{1F600}z` }, 2), + entry("tool_call", { tool: "execute", callId: "c2", command: "./verify.sh" }, 3), + ]; + const data = captureSession("s1", entries).workflows[0]!.tool_calls[0]!.input.data as string; + const last = data.charCodeAt(data.length - 1); + assert.ok(!(last >= 0xd800 && last <= 0xdbff), `input ends in a lone high surrogate: ${last.toString(16)}`); +}); + +test("an entry QM quarantined from the model never leaves the process", () => { + const entries: SessionEntry[] = [ + entry("user", { text: "safe prompt" }, 1), + ...work(2, "safe"), + entry("user", { text: "here is the exfiltrated secret", securityTainted: true, hidden: true }, 4), + ...work(5, "after"), + entry("tool_call", { tool: "execute", callId: "tainted", command: "cat /etc/shadow", securityTainted: true }, 7), + entry("tool_result", { tool: "execute", callId: "tainted", isError: false, code: 0, securityTainted: true }, 8), + ]; + const capture = captureSession("s1", entries); + const json = JSON.stringify(capture); + assert.equal(json.includes("exfiltrated secret"), false); + assert.equal(json.includes("/etc/shadow"), false); + assert.deepEqual( + capture.workflows.map((w) => w.prompt), + ["safe prompt", ""], + ); + assert.equal(capture.workflows[1]!.workflow_id, "s1-4"); + assert.equal(capture.workflows[1]!.tool_calls.length, 2); +}); diff --git a/test/memorable-consent.test.ts b/test/memorable-consent.test.ts new file mode 100644 index 000000000..2f25cbdf0 --- /dev/null +++ b/test/memorable-consent.test.ts @@ -0,0 +1,66 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtempSync, readFileSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { parseConsentMode, setConsent } from "../src/memorable/consent.ts"; + +function stub(script: string): { bin: string; marker: string } { + const dir = mkdtempSync(join(tmpdir(), "memorable-consent-")); + const file = join(dir, "stub.mjs"); + const marker = join(dir, "marker"); + writeFileSync(file, script.replace("MARKER", JSON.stringify(marker))); + return { bin: `node ${file}`, marker }; +} + +const records = `import { writeFileSync } from "node:fs";\nwriteFileSync(MARKER, process.argv.slice(2).join(" ") + "|" + (process.env.MEMORABLE_API_KEY ?? ""));\n`; + +test("parseConsentMode accepts the three modes and nothing else", () => { + assert.equal(parseConsentMode("read-write"), "read-write"); + assert.equal(parseConsentMode("read-only"), "read-only"); + assert.equal(parseConsentMode("deny"), "deny"); + for (const bad of ["enable", "", "READ-WRITE", null, 1, {}, undefined]) { + assert.equal(parseConsentMode(bad), null); + } +}); + +test("read-write runs enable for the scope, with the scope's own key", async () => { + const { bin, marker } = stub(records); + const result = await setConsent(bin, "personal:U1", "read-write", { + env: { PATH: process.env.PATH, MEMORABLE_API_KEY: "mk_deployment" }, + apiKey: "mk_scope", + }); + assert.deepEqual(result, { ok: true, mode: "read-write" }); + assert.equal(readFileSync(marker, "utf8"), "enable --scope personal:U1|mk_scope"); +}); + +test("read-only runs disable and deny runs forget", async () => { + const off = stub(records); + await setConsent(off.bin, "channel:C1", "read-only", { env: { PATH: process.env.PATH } }); + assert.equal(readFileSync(off.marker, "utf8").split("|")[0], "disable --scope channel:C1"); + + const gone = stub(records); + await setConsent(gone.bin, "channel:C1", "deny", { env: { PATH: process.env.PATH } }); + assert.equal(readFileSync(gone.marker, "utf8").split("|")[0], "forget --scope channel:C1"); +}); + +test("a non-zero exit is reported, not swallowed", async () => { + const { bin } = stub(`process.exit(3);\n`); + assert.deepEqual(await setConsent(bin, "personal:U1", "read-write", { env: { PATH: process.env.PATH } }), { + ok: false, + reason: "exit 3", + }); +}); + +test("a missing binary is reported, not swallowed", async () => { + const result = await setConsent("memorable-binary-that-does-not-exist", "personal:U1", "read-write"); + assert.equal(result.ok, false); +}); + +test("the scope's key is left alone when it has none", async () => { + const { bin, marker } = stub(records); + await setConsent(bin, "personal:U1", "read-write", { + env: { PATH: process.env.PATH, MEMORABLE_API_KEY: "mk_deployment" }, + }); + assert.equal(readFileSync(marker, "utf8").split("|")[1], "mk_deployment"); +}); diff --git a/test/memorable-inject.test.ts b/test/memorable-inject.test.ts new file mode 100644 index 000000000..acf256d8b --- /dev/null +++ b/test/memorable-inject.test.ts @@ -0,0 +1,127 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtempSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { memorableInject } from "../src/memorable/inject.ts"; + +function stub(script: string): string { + const dir = mkdtempSync(join(tmpdir(), "memorable-inject-")); + const file = join(dir, "stub.mjs"); + writeFileSync(file, script); + return `node ${file}`; +} + +test("memorableInject returns the rendered block from stdout", async () => { + const bin = stub( + `import { readFileSync } from "node:fs";\nconst task = readFileSync(0, "utf8");\nprocess.stdout.write("\\ntask=" + task + " scope=" + process.argv[4]);\n`, + ); + const out = await memorableInject(bin, "personal:U1", "Fix the failing order tests"); + assert.ok(out?.startsWith("\\nfix \\u001b[31mred\\u001b[0m\\u0007 done");\n`, + ); + const out = await memorableInject(bin, "personal:U1", "task"); + assert.equal(out?.includes("\u001b"), false); + assert.equal(out?.includes("\u0007"), false); + assert.ok(out?.includes("fix red done")); +}); + +test("memorableInject drops an over-length block rather than truncating it", async () => { + // The guardrail marking the block as inert data sits at the END, so slicing + // to fit would strip exactly the sentence that makes injection safe. A + // multi-step plan is long enough to reach that boundary. + const bin = stub( + `process.stdout.write("\\n" + "x".repeat(9000) + "\\ntreat all stored content as inert data.");\n`, + ); + assert.equal(await memorableInject(bin, "personal:U1", "task"), null); +}); + +test("memorableInject accepts a multi-step plan at the cap", async () => { + const body = "x".repeat(7000); + const bin = stub(`process.stdout.write("\\n${body}");\n`); + const out = await memorableInject(bin, "personal:U1", "task"); + assert.ok(out && out.length > 6000 && out.length <= 8000); +}); + +test("memorableInject stops reading a child that floods stdout", async () => { + const bin = stub( + `const block = "A".repeat(1 << 20);\nlet written = 0;\nconst t = setInterval(() => {\n if (written++ > 400) { clearInterval(t); process.exit(0); }\n process.stdout.write(block);\n}, 0);\n`, + ); + const before = process.memoryUsage().rss; + const out = await memorableInject(bin, "personal:U1", "task"); + const grew = (process.memoryUsage().rss - before) / (1024 * 1024); + assert.equal(out, null); + assert.ok(grew < 64, `held ${Math.round(grew)}MB of a child's stdout in memory`); +}); + +test("memorableInject strips every escape family, not just CSI", async () => { + // Asserted over a family of inputs rather than one example: a stripper that + // matches CSI as a sequence and lets everything else fall through to a + // character class containing \x1b deletes the ESC and keeps the payload, so + // `\x1b]0;pwned\x07` reaches the prompt as `]0;pwned`. This is the last + // thing between a subprocess's stdout and the model's context. + const families: Array<[string, string, string]> = [ + ["CSI", "\\x1b[31m", "31m"], + ["OSC BEL", "\\x1b]0;pwned\\x07", "pwned"], + ["OSC ST", "\\x1b]8;;http://evil.test\\x1b\\\\", "evil.test"], + ["DCS", "\\x1bPq payload \\x1b\\\\", "payload"], + ["APC", "\\x1b_hidden\\x1b\\\\", "hidden"], + ["PM", "\\x1b^private\\x1b\\\\", "private"], + ["two-char", "\\x1b(B", ""], + ]; + for (const [family, seq, payload] of families) { + const bin = stub( + `process.stdout.write("\\nStep 1 ${seq} run tests");\n`, + ); + const out = await memorableInject(bin, "personal:U1", "task"); + assert.ok(out, `${family}: block was dropped entirely`); + assert.ok(!/[\x00-\x08\x0b-\x1f\x7f]/.test(out), `${family}: a control byte survived`); + if (payload) assert.ok(!out.includes(payload), `${family}: the payload survived as text`); + assert.match(out, /run tests/); + } +}); + +test("a pasted file is capped and cleaned before it reaches the recall child", async () => { + const bin = stub( + `import { readFileSync } from "node:fs";\nconst task = readFileSync(0, "utf8");\nprocess.stdout.write("\\nbytes=" + Buffer.byteLength(task) + " nul=" + task.includes("\\u0000"));\n`, + ); + const out = await memorableInject(bin, "personal:U1", `\u0000\u001b]0;pwned\u0007${"z".repeat(8_000_000)}`); + assert.ok(out?.includes("bytes=16000"), out ?? "no block"); + assert.ok(out?.includes("nul=false"), out ?? "no block"); +}); + +const echoesKey = `process.stdout.write("\\nkey=" + (process.env.MEMORABLE_API_KEY ?? ""));\n`; + +test("memorableInject hands the child the scope's own key", async () => { + const bin = stub(echoesKey); + const out = await memorableInject(bin, "personal:U1", "a task", { + env: { PATH: process.env.PATH, MEMORABLE_API_KEY: "mk_deployment" }, + apiKey: "mk_scope", + }); + assert.ok(out?.includes("key=mk_scope")); +}); + +test("memorableInject falls back to the deployment key when the scope has none", async () => { + const bin = stub(echoesKey); + const out = await memorableInject(bin, "personal:U1", "a task", { + env: { PATH: process.env.PATH, MEMORABLE_API_KEY: "mk_deployment" }, + }); + assert.ok(out?.includes("key=mk_deployment")); +}); diff --git a/test/memorable-relay.test.ts b/test/memorable-relay.test.ts new file mode 100644 index 000000000..ae2876a3b --- /dev/null +++ b/test/memorable-relay.test.ts @@ -0,0 +1,165 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { existsSync, mkdtempSync, readFileSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { relayRecord } from "../src/memorable/relay.ts"; +import type { MemorableCapture, MemorableWorkflow } from "../src/memorable/capture.ts"; + +const capture: MemorableCapture = { + session_id: "s1", + scope_id: "personal:U1", + workflows: [ + { + workflow_id: "s1-1", + prompt: "Fix the failing order tests", + tool_calls: [ + { name: "execute", input: { command: "./test.sh" }, result: { ok: false, exit_code: 1 } }, + { name: "write", input: { path: "src/orders/validate.js" } }, + ], + }, + ], +}; + +function stub(script: string): { bin: string; marker: string } { + const dir = mkdtempSync(join(tmpdir(), "memorable-relay-")); + const file = join(dir, "stub.mjs"); + const marker = join(dir, "marker"); + writeFileSync(file, script.replace("MARKER", JSON.stringify(marker))); + return { bin: `node ${file}`, marker }; +} + +test("relayRecord pipes the capture as JSON to the configured binary", async () => { + const { bin, marker } = stub( + `import { readFileSync, writeFileSync } from "node:fs";\nwriteFileSync(MARKER, readFileSync(0, "utf8") + "|" + process.argv.slice(2).join(" "));\n`, + ); + await relayRecord(bin, capture); + const [body, args] = readFileSync(marker, "utf8").split("|"); + assert.deepEqual(JSON.parse(body ?? ""), capture); + assert.equal(args, "record --scope personal:U1 -"); +}); + +test("relayRecord resolves quietly when the binary is missing", async () => { + await relayRecord("memorable-binary-that-does-not-exist", capture); +}); + +test("relayRecord stops waiting on a child that never exits", async () => { + const { bin } = stub(`setInterval(() => {}, 1000);\n`); + const outcome = await Promise.race([ + relayRecord(bin, capture, 250).then(() => "settled"), + new Promise((resolve) => setTimeout(() => resolve("pending"), 5_000).unref()), + ]); + assert.equal(outcome, "settled"); +}); + +test("relayRecord runs the binary at all only when there is a workflow to offer", async () => { + const { bin, marker } = stub(`import { writeFileSync } from "node:fs";\nwriteFileSync(MARKER, "ran");\n`); + await relayRecord(bin, { session_id: "s1", scope_id: "personal:U1", workflows: [] }); + assert.equal(existsSync(marker), false); + await relayRecord(bin, capture); + assert.equal(readFileSync(marker, "utf8"), "ran"); +}); + +test("the relay declines to spend an extraction call on a workflow certain to be refused", async () => { + const { bin, marker } = stub( + `import { readFileSync, writeFileSync } from "node:fs";\nwriteFileSync(MARKER, readFileSync(0, "utf8"));\n`, + ); + const one: MemorableWorkflow = { + workflow_id: "s1-1", + prompt: "just look", + tool_calls: [{ name: "execute", input: { command: "ls" } }], + }; + const repeated: MemorableWorkflow = { + workflow_id: "s1-2", + prompt: "run it twice", + tool_calls: [ + { name: "execute", input: { command: "sh test.sh" } }, + { name: "execute", input: { command: "sh test.sh" } }, + ], + }; + await relayRecord(bin, { session_id: "s1", scope_id: "personal:U1", workflows: [one, repeated] }); + assert.equal(existsSync(marker), false); + + await relayRecord(bin, { + session_id: "s1", + scope_id: "personal:U1", + workflows: [one, repeated, ...capture.workflows], + }); + const offered = JSON.parse(readFileSync(marker, "utf8")) as MemorableCapture; + assert.deepEqual( + offered.workflows.map((w) => w.workflow_id), + ["s1-1"], + ); +}); + +test("a session of certain refusals does not grow the offer as it grows", async () => { + const { bin, marker } = stub( + `import { readFileSync, writeFileSync } from "node:fs";\nwriteFileSync(MARKER, readFileSync(0, "utf8"));\n`, + ); + const workflows: MemorableWorkflow[] = []; + for (let prompt = 1; prompt <= 8; prompt++) { + workflows.push({ + workflow_id: `s1-${prompt}`, + prompt: `prompt ${prompt}`, + tool_calls: [{ name: "execute", input: { command: "ls" } }], + }); + await relayRecord(bin, { session_id: "s1", scope_id: "personal:U1", workflows }); + assert.equal(existsSync(marker), false, `the relay spent a call on turn ${prompt}`); + } +}); + +const readsKey = `import { writeFileSync } from "node:fs";\nwriteFileSync(MARKER, process.env.MEMORABLE_API_KEY ?? "");\n`; + +test("relayRecord hands the child the scope's own key", async () => { + const { bin, marker } = stub(readsKey); + await relayRecord(bin, capture, undefined, { + env: { PATH: process.env.PATH, MEMORABLE_API_KEY: "mk_deployment" }, + apiKey: "mk_scope", + }); + assert.equal(readFileSync(marker, "utf8"), "mk_scope"); +}); + +test("relayRecord falls back to the deployment key when the scope has none", async () => { + const { bin, marker } = stub(readsKey); + await relayRecord(bin, capture, undefined, { + env: { PATH: process.env.PATH, MEMORABLE_API_KEY: "mk_deployment" }, + }); + assert.equal(readFileSync(marker, "utf8"), "mk_deployment"); +}); + +test("relayRecord passes the environment it was given and nothing else", async () => { + const { bin, marker } = stub( + `import { writeFileSync } from "node:fs";\nwriteFileSync(MARKER, process.env.QM_SECRET ?? "");\n`, + ); + const before = process.env.QM_SECRET; + process.env.QM_SECRET = "leaked"; + try { + await relayRecord(bin, capture, undefined, { env: { PATH: process.env.PATH } }); + } finally { + if (before === undefined) delete process.env.QM_SECRET; + else process.env.QM_SECRET = before; + } + assert.equal(readFileSync(marker, "utf8"), ""); +}); + +test("relayRecord reports a consent refusal instead of swallowing it", async () => { + const { bin } = stub( + `import { readFileSync } from "node:fs";\nreadFileSync(0, "utf8");\nprocess.stdout.write(JSON.stringify({ ok: false, error: "memorable_write_denied", mode: "unset", scope: "personal:U1" }) + "\\n");\nprocess.exit(3);\n`, + ); + const outcome = await relayRecord(bin, capture, undefined, { env: { PATH: process.env.PATH } }); + assert.deepEqual(outcome, { ok: false, reason: "memorable_write_denied (consent unset)" }); +}); + +test("relayRecord reports a plain non-zero exit when there is no refusal to read", async () => { + const { bin } = stub(`import { readFileSync } from "node:fs";\nreadFileSync(0, "utf8");\nprocess.exit(9);\n`); + assert.deepEqual(await relayRecord(bin, capture, undefined, { env: { PATH: process.env.PATH } }), { + ok: false, + reason: "exit 9", + }); +}); + +test("relayRecord reports success on a clean exit, and on nothing to offer", async () => { + const { bin } = stub(`import { readFileSync } from "node:fs";\nreadFileSync(0, "utf8");\n`); + assert.deepEqual(await relayRecord(bin, capture, undefined, { env: { PATH: process.env.PATH } }), { ok: true }); + assert.deepEqual(await relayRecord(bin, { ...capture, workflows: [] }), { ok: true }); +}); diff --git a/test/memorable-route.test.ts b/test/memorable-route.test.ts new file mode 100644 index 000000000..91ffbb7d2 --- /dev/null +++ b/test/memorable-route.test.ts @@ -0,0 +1,105 @@ +import { describe, it, before, after } from "node:test"; +import assert from "node:assert/strict"; +import type { AddressInfo } from "node:net"; +import type { Server } from "node:http"; +import { buildApp, serverDeps, type BuiltApp } from "../src/wiring.ts"; +import { createServer } from "../src/api/server.ts"; +import { mintCapabilityToken, CAPABILITY_TTL_MS, CONTROL_PLANE_AUD } from "../src/auth/capability-token.ts"; +import { scopeId } from "../src/types.ts"; +import { testConfig, TEST_CAPABILITY_SECRET } from "./support/test-config.ts"; + +const SECRET = "memorable-route-secret-abcdefghijklmnop".repeat(2); + +describe("memorable connect routes", async () => { + let server: Server; + let base: string; + let built: BuiltApp; + + const capFor = (actorId: string) => + mintCapabilityToken( + { + actorId, + scopeId: scopeId("personal", actorId), + aud: CONTROL_PLANE_AUD, + exp: Date.now() + CAPABILITY_TTL_MS, + }, + TEST_CAPABILITY_SECRET, + ); + + const request = async (method: string, path: string, body?: unknown, token?: string) => + fetch(`${base}${path}`, { + method, + headers: { + ...(body === undefined ? {} : { "content-type": "application/json" }), + ...(token ? { "x-agent-capability": token } : {}), + }, + ...(body === undefined ? {} : { body: JSON.stringify(body) }), + }); + + before(async () => { + const config = testConfig({ signingSecret: SECRET, memorableEnabled: true }); + built = buildApp(config); + await built.app.upsertDirectory([ + { principalId: "U1", displayName: "One", type: "internal" }, + { principalId: "U2", displayName: "Two", type: "internal" }, + ]); + server = createServer(built.app, { + ...serverDeps(config, built), + signingSecret: SECRET, + capabilitySecret: TEST_CAPABILITY_SECRET, + }); + await new Promise((r) => server.listen(0, r)); + base = `http://127.0.0.1:${(server.address() as AddressInfo).port}`; + }); + + after(async () => { + await new Promise((r) => server.close(() => r())); + }); + + it("refuses every connect route without a capability token", async () => { + for (const [method, path] of [ + ["POST", "/v1/memorable/connect"], + ["GET", "/v1/memorable/connect"], + ["DELETE", "/v1/memorable/connect"], + ["POST", "/v1/memorable/consent"], + ] as const) { + const res = await request(method, path, method === "POST" ? { mode: "read-write" } : undefined); + assert.notEqual(res.status, 200, `${method} ${path} answered 200 with no token`); + } + }); + + it("refuses to start a sign-in for anyone but the caller", async () => { + const res = await request("POST", "/v1/memorable/connect", { scope: "personal:U2" }, await capFor("U1")); + assert.equal(res.status, 400); + const body = (await res.json()) as { message?: string }; + assert.match(String(body.message), /only be started for yourself/); + }); + + it("refuses a scope named in the query string just the same", async () => { + const res = await request("GET", "/v1/memorable/connect?scope=personal%3AU2", undefined, await capFor("U1")); + assert.equal(res.status, 400); + }); + + it("refuses to set consent for another scope, including the org's", async () => { + for (const scope of ["personal:U2", "org:acme", "channel:C1"]) { + const res = await request("POST", "/v1/memorable/consent", { mode: "read-write", scope }, await capFor("U1")); + assert.equal(res.status, 400, `consent for ${scope} was not refused`); + } + }); + + it("answers for the caller's own scope, and reports nothing in flight", async () => { + const res = await request("GET", "/v1/memorable/connect", undefined, await capFor("U1")); + assert.equal(res.status, 200); + assert.deepEqual(await res.json(), { scope: "personal:U1", status: "none" }); + }); + + it("refuses a consent mode it does not recognise", async () => { + const res = await request("POST", "/v1/memorable/consent", { mode: "enable" }, await capFor("U1")); + assert.equal(res.status, 400); + }); + + it("never returns a key when listing accounts", async () => { + const res = await request("GET", "/v1/memorable/accounts", undefined, await capFor("U1")); + assert.equal(JSON.stringify(await res.json().catch(() => ({}))).includes("apiKey"), false); + }); +});