Comprehensive feature spec and e2e test coverage: full surface audit, new specs for gaps, and test-fix plan for the three permanently-failing e2e lanes

[Al629176]

Summary

A full audit of OpenCompany's feature surface against the existing 99 Playwright specs reveals: three spec lanes failing on every PR since 2026-09-04 (issue #2110), six shipped features with no regression coverage at all, and four open bugs whose reproduction path exists only in the issue description and not in the test suite. This issue delivers the complete feature specification, the new spec files, and the acceptance bar that marks the suite healthy.

---

Problem

The test suite is broken as a merge gate. Console E2E and Console E2E (live brain) have been red on main and on every open PR since approximately 2026-09-04 (issue #2110). Three specs fail deterministically:

test/e2e/connections-authority.spec.ts:145  →  #company-credential not found
test/e2e/desktop-connections.spec.ts:420   →  [data-testid="no-connection-run-here"] not found
test/e2e/workflow-selection-persists.spec.ts:176  →  click timeout (60 s)

The live-brain lane additionally fails composio-catalog-deadline.spec.ts:223 after the catalog-deadline fix in PR #2035. Because CI builds the merge commit, every author reads these as "my branch broke E2E" — a false signal that erodes trust in the gate.

Six shipped features have no spec coverage. The following surfaces exist in the codebase and in closed PRs but have no corresponding Playwright spec:

Feature	Shipped in	Missing test
"Just chatting" sends empty tool list to model	open bug #2094	Agent markup leaks into chat undetected
Composio BYOK — per-company key stored server-side	PR #2060	No scope or persistence proof
Ledger rejects rows missing a required field	PR #2048	Partial entries silently accepted until observed in prod
Blocker → DM → reply → resume (Epic #1860)	PRs #2004 #2006 #2017 #2038	End-to-end park/resume never driven by a test
Attachment upload+resolve scope parity	Bug #2029	Attachment rejection only noticed in the field
Inference test mislabels 404 as bad URL	Bug #1993	Label correctness unverified across all HTTP shapes

Four open bugs have no reproduction path in the suite — they were filed from field reports or code reads, and a re-regression would surface only from users.

---

Feature specification (complete surface)

The following is the full feature list derived from the source tree (src/server/ops/, src/server/graphql/, frontend views/), closed PRs, and open epics. Each entry maps to a test-coverage state.

Auth / Onboarding

• Magic-link login → session cookie → console (spec: wiring.spec.ts)
• Password login toggle (spec: login-centred.spec.ts)
• Resend throttle (spec: login-resend.spec.ts)
• First-run setup wizard → named company + first integration (spec: onboarding-gate.spec.ts, company-setup.spec.ts)
• First-run product tour, dismissal, persistence (spec: tour-completion-persists.spec.ts, tour-popover-in-viewport.spec.ts)
• OAuth onboarding resume after redirect (spec: oauth-onboarding-resume.spec.ts)
• Member vs admin role enforcement on Connections/Settings (spec: connections-authority.spec.ts — FAILING)
• Multi-host sign-in and host-switcher (spec: sidebar-host-switcher.spec.ts, manage-hosts.spec.ts)

Chat

• Channel list renders in sidebar, unread badges update (spec: chat-channel-membership.spec.ts, chat-mention-badge.spec.ts)
• Sending a message and receiving a reply (spec: wiring.spec.ts, chat-live-events.spec.ts)
• Live receipt across sync→async detach (spec: chat-detached-post-failure.spec.ts)
• Mentions: @-complete, highlight, DM route (spec: chat-mentions.spec.ts)
• Thread replies and avatar rendering (spec: chat-thread-avatar.spec.ts)
• Blocker disambig prompt threaded to origin (bug chat: a blocker disambiguation prompt is posted unthreaded, though its doc says it threads #2039, no spec)
• Attachment upload+resolve scope parity (bug chat: attachments are rejected as "not a file in this company's workspace" — upload/list and resolve_attachments disagree on scope #2029, new spec: attachment-scope.spec.ts)
• "Just chatting" sends empty tool list (bug A "Just chatting" turn still advertises its tools, so the model's tool markup lands in the operator's chat #2094, new spec: just-chatting-no-tools.spec.ts)
• Composer mode selection persists between navigations (partial: workflow-selection-persists.spec.ts — FAILING)
• Dispatch marker rendered in transcript (spec: chat-dispatch-marker.spec.ts)
• Inline approval rendered and actionable in transcript (spec: chat-inline-approval.spec.ts)
• Presence indicators (spec: chat-presence.spec.ts)

Work Board / Tasks

• Kanban columns render with correct cards (spec: board-columns.spec.ts)
• Drag to reorder / move between columns (spec: board-drag.spec.ts)
• Chat → card dispatch: a Do-it-once message produces a card (spec: chat-to-card.spec.ts)
• Task detail view: steps, attempts, artifacts tabs (spec: agent-runs.spec.ts)
• Assignee picker (spec: assignee-picker.spec.ts)
• Budget display on team card (spec: team-budget.spec.ts, team-budget-edit.spec.ts)
• Task export (ops: src/server/ops/task_export/, no spec)

Approvals

• Inline approval line in chat (spec: chat-approval-line.spec.ts)
• Approval timeout honesty — expired approval settles the run (spec: approval-timeout-honesty.spec.ts)
• Blocker parked → DM delivered → reply resumes run (Epic Epic: blockers are conversations — park → DM → reply → resume from the failed step #1860, new spec: blocker-dm-flow.spec.ts)
• All four verdict buttons accessible: approve / skip / amend / reject (PR feat(server): let an operator answer a blocker with all four verdicts, not two (#2028) #2038, covered in new blocker spec)

Workflows

• Canvas renders and is keyboard-accessible (workflow-canvas-escape.spec.ts, workflow-canvas-fit.spec.ts)
• Node config inspector (workflow-node-config.spec.ts, workflow-node-inspector-reveal.spec.ts)
• Create, edit, delete a workflow (workflow-create-affordance.spec.ts, workflow-edit-delete.spec.ts)
• Live run with inference (workflow-run-inference.spec.ts, workflow-canvas-live.spec.ts)
• Run history rail and result panel (workflow-run-history.spec.ts, workflow-run-result.spec.ts)
• Inline workflow approval (workflow-inline-approval.spec.ts)
• Week-1 nudge to save a workflow (workflow-week1-nudge.spec.ts)
• Selection persists across company switch (spec: workflow-selection-persists.spec.ts — FAILING)
• Delete/pause sweep does not miss a concurrent registration (bug workflow delete/pause sweep can miss a run that registers just after the sweep runs #2059, no spec)

Workspace

• File tree renders seeded notes; no localStorage fallback (spec: workspace.spec.ts)
• Create, edit, autosave a note (workspace.spec.ts)
• Out-of-band write visible after reload (workspace.spec.ts)
• Note download (ops: src/server/ops/workspace.rs, bug Workspace note download reads the whole note into memory before sending it #2071 — reads whole file into memory, no spec)

Ledgers

• Board view renders rows by status (spec: ledger-list-row.spec.ts)
• Ledger refuses a row missing a required field (PR fix(ledger): refuse a row missing a field the ledger declares required #2048, new spec: ledger-required-fields.spec.ts)
• "Rows that could not be read" section absent when all entries are valid (new spec)

Connections

• Provider catalog loads and category filter works (tool-grant-from-connect-page.spec.ts)
• Catalog read has a deadline — page does not hang (PR fix(console): give the Composio catalog read a deadline that outlasts the host's own (#2007) #2035, covered in new spec: composio-byok.spec.ts)
• Composio BYOK: per-company key stored server-side (PR feat(composio): bring your own Composio account (BYOK) #2060, new spec: composio-byok.spec.ts)
• Member cannot see or submit credential forms (spec: connections-authority.spec.ts — FAILING)
• Desktop host not-started state (spec: desktop-connections.spec.ts — FAILING)
• MCP server load timeout handled gracefully (spec: mcp-load-timeout.spec.ts)
• MCP agent integration (spec: mcp-agent.spec.ts)
• Add-host connector flow (spec: add-host-connectors.spec.ts)

Inference / Settings

• Inference key saved and test-connection fires (inference.spec.ts)
• Test-connection labels 404 as model-unavailable, not bad URL (bug setup: inference test mislabels a 404 "model/provider not permitted" as "no chat endpoint at that URL" #1993, new spec: inference-setup-errors.spec.ts)
• 401 labelled as invalid API key (new spec)
• Network failure labelled as unreachable (new spec)
• Managed card stores key against correct provider, not OpenRouter (bug The Managed card is labelled TinyHumans but stores keys against OpenRouter — #1737's trap still works #1962, no spec)

Memory / Brain

• Memory list virtualized — large sets do not freeze (spec: brain-virtualization.spec.ts)
• Memory engine selector visible (spec: brain-memory.spec.ts)
• CortexDB engine selectable and connects (PR Make cortex a selectable memory engine (#1936) #2065, no spec)

Observatory / Runs

• Runs list renders attempts (spec: agent-runs.spec.ts)
• Deep trace view (ops: src/server/ops/deep_trace.rs, spec: workflow-run-traces.spec.ts)

Company / Team

• Agent roster cards render (spec: company-cards.spec.ts)
• Agent detail panel and profile sheet (spec: agent-detail.spec.ts, agent-profile-panel.spec.ts, agent-profile-sheet.spec.ts)
• Desks view (spec: team-desks.spec.ts)
• People invite (spec: people-invite.spec.ts)

Finance / Usage

• Finances page renders 4 tiles (pages-render.spec.ts covers route render)
• Usage dashboard — credit/token breakdown (enhancement Feature: Credit/token usage dashboard — model-wise breakdown, credit-to-token mapping, usage timeline #1938, no spec)

Skills / Pages

• Skills registry (skills-registry.spec.ts)
• Pages render with SDK assets (pages-render.spec.ts)

Accessibility / Shell

• Sidebar navigation keyboard accessibility (sidebar-navigation-accessibility.spec.ts)
• Skip-to-content link present (page-headings.spec.ts)
• Shell two-layer layout (shell-two-layer.spec.ts)
• Theme toggle visible (theme-toggle-visible.spec.ts)
• Toast dismissal (toast-dismissal.spec.ts)
• Mobile responsiveness (spec: overview-responsive-chrome.spec.ts)

---

New spec files (this PR / sub-issues)

Six new specs written against the gaps above:

File	Covers
just-chatting-no-tools.spec.ts	Issue #2094 — empty tool list in Just chatting mode
composio-byok.spec.ts	PR #2060 — BYOK key persistence, catalog deadline, member gate
ledger-required-fields.spec.ts	PR #2048 — 422 on missing required field, no "could not be read"
blocker-dm-flow.spec.ts	Epic #1860 — park → DM → approve → resume end-to-end
attachment-scope.spec.ts	Issue #2029 — upload and resolve use the same scope
inference-setup-errors.spec.ts	Issue #1993 — correct label for 404 / 401 / network failure

---

Solution

1 — Fix the three permanently-failing specs (blocker for all PRs)

connections-authority.spec.ts:145 — #company-credential not found. The element id or data-testid changed in a recent UI refactor (likely the Connections redesign tracked in Epic #2104). Fix: re-locate the element by its current selector, or add a stable data-testid to the credential input in ConnectionConsole.tsx.

desktop-connections.spec.ts:420 — [data-testid="no-connection-run-here"] not found. The "offer to start here" affordance was renamed or restructured in the desktop connections redesign. Fix: align the data-testid in the component with what the spec expects, or update the spec to use the current structure.

workflow-selection-persists.spec.ts:176 — click timeout. The company-switch sequence navigates away from the workflow before the click target appears. Fix: add a waitFor on the navigation settling before clicking, or use page.route to intercept the company-switch redirect.

2 — Land the six new specs

The files listed above are committed to frontend/test/e2e/. Each spec includes:

• A clear issue/PR reference in its doc comment
• test.skip(!LIVE_BRAIN, ...) guards where the echo brain is not enough
• An explicit assertion that the pre-fix behavior does not reappear

3 — Add missing coverage for remaining gaps

Follow-up sub-issues for:

• workflow-delete-pause-sweep.spec.ts (bug workflow delete/pause sweep can miss a run that registers just after the sweep runs #2059)
• managed-card-provider-key.spec.ts (bug The Managed card is labelled TinyHumans but stores keys against OpenRouter — #1737's trap still works #1962)
• cortex-memory-engine.spec.ts (PR Make cortex a selectable memory engine (#1936) #2065)
• task-export.spec.ts (ops surface, no coverage)
• workspace-note-download-stream.spec.ts (bug Workspace note download reads the whole note into memory before sending it #2071 — reads whole file into memory before sending)

---

Acceptance criteria

• Failing lanes green — connections-authority.spec.ts, desktop-connections.spec.ts, and workflow-selection-persists.spec.ts pass on main and on every subsequent PR. The Console E2E and Console E2E (live brain) lanes are no longer permanently red.
• Six new specs in the suite — just-chatting-no-tools, composio-byok, ledger-required-fields, blocker-dm-flow, attachment-scope, inference-setup-errors are in frontend/test/e2e/ and each passes on the appropriate lane (default or live-brain).
• Regression safety — Each new spec has at least one assertion that would fail if the bug it covers reappeared (i.e., it is not vacuous — composio-catalog-deadline was vacuous before PR fix(console): give the Composio catalog read a deadline that outlasts the host's own (#2007) #2035 fixed the deadline; this bar prevents a repeat).
• Diff coverage ≥ 80% — Any Rust changes required to fix the failing specs (e.g. adding a data-testid via a rendered attribute) meet the changed-lines gate.
• No skip-all — No new spec uses test.skip(true, ...) unconditionally. Each skip is guarded by a runtime condition (missing host feature, absent fixture user) and documented.
• Observable — The Console E2E CI step reports a passing count ≥ 99 (all existing specs) and the live-brain step reports a passing count ≥ 103 (99 + 4 new live-brain specs).
• Spec count documented — CLAUDE.md or a comment in playwright.config.ts notes the target spec count so a future PR that accidentally drops a spec is caught.

---

Related

• Console E2E is red on main and on every PR: three specs failing since ~2026-09-04 #2110 — the three permanently-failing specs (immediate blocker)
• A "Just chatting" turn still advertises its tools, so the model's tool markup lands in the operator's chat #2094 — Just chatting tool leak
• feat(composio): bring your own Composio account (BYOK) #2060 — Composio BYOK
• fix(ledger): refuse a row missing a field the ledger declares required #2048 — Ledger required-field rejection
• Epic: blockers are conversations — park → DM → reply → resume from the failed step #1860 — Epic: blockers are conversations
• chat: attachments are rejected as "not a file in this company's workspace" — upload/list and resolve_attachments disagree on scope #2029 — Attachment scope mismatch
• setup: inference test mislabels a 404 "model/provider not permitted" as "no chat endpoint at that URL" #1993 — Inference test mislabels 404
• workflow delete/pause sweep can miss a run that registers just after the sweep runs #2059 — Workflow delete/pause sweep race
• The Managed card is labelled TinyHumans but stores keys against OpenRouter — #1737's trap still works #1962 — Managed card stores against wrong provider
• Epic: redesign Managed connection card labeling (Connections & Credentials overhaul) #2104 — Connections redesign (likely cause of connections-authority failure)
• Workspace note download reads the whole note into memory before sending it #2071 — Workspace note download reads whole file into memory