diff --git a/skills/test-driven-development/SKILL.md b/skills/test-driven-development/SKILL.md index 60d2609ca5..3eccc65859 100644 --- a/skills/test-driven-development/SKILL.md +++ b/skills/test-driven-development/SKILL.md @@ -203,6 +203,12 @@ Next failing test for next feature. | **Clear** | Name describes behavior | `test('test1')` | | **Shows intent** | Demonstrates desired API | Obscures what code should do | +When writing or changing any test, read [writing-good-tests.md](writing-good-tests.md) for the rules that keep tests honest: +- Name the production change that would make the test fail — before writing it +- Assert on real behavior, never on mock behavior +- Keep test-only code in test utilities, out of production classes +- Understand a dependency's side effects before mocking it + ## Why Order Matters **"I'll write tests after to verify it works"** @@ -354,13 +360,6 @@ Bug found? Write failing test reproducing it. Follow TDD cycle. Test proves fix Never fix bugs without a test. -## Testing Anti-Patterns - -When adding mocks or test utilities, read [testing-anti-patterns.md](testing-anti-patterns.md) to avoid common pitfalls: -- Testing mock behavior instead of real behavior -- Adding test-only methods to production classes -- Mocking without understanding dependencies - ## Final Rule ``` diff --git a/skills/test-driven-development/testing-anti-patterns.md b/skills/test-driven-development/testing-anti-patterns.md deleted file mode 100644 index e77ab6b6d6..0000000000 --- a/skills/test-driven-development/testing-anti-patterns.md +++ /dev/null @@ -1,299 +0,0 @@ -# Testing Anti-Patterns - -**Load this reference when:** writing or changing tests, adding mocks, or tempted to add test-only methods to production code. - -## Overview - -Tests must verify real behavior, not mock behavior. Mocks are a means to isolate, not the thing being tested. - -**Core principle:** Test what the code does, not what the mocks do. - -**Following strict TDD prevents these anti-patterns.** - -## The Iron Laws - -``` -1. NEVER test mock behavior -2. NEVER add test-only methods to production classes -3. NEVER mock without understanding dependencies -``` - -## Anti-Pattern 1: Testing Mock Behavior - -**The violation:** -```typescript -// ❌ BAD: Testing that the mock exists -test('renders sidebar', () => { - render(); - expect(screen.getByTestId('sidebar-mock')).toBeInTheDocument(); -}); -``` - -**Why this is wrong:** -- You're verifying the mock works, not that the component works -- Test passes when mock is present, fails when it's not -- Tells you nothing about real behavior - -**your human partner's correction:** "Are we testing the behavior of a mock?" - -**The fix:** -```typescript -// ✅ GOOD: Test real component or don't mock it -test('renders sidebar', () => { - render(); // Don't mock sidebar - expect(screen.getByRole('navigation')).toBeInTheDocument(); -}); - -// OR if sidebar must be mocked for isolation: -// Don't assert on the mock - test Page's behavior with sidebar present -``` - -### Gate Function - -``` -BEFORE asserting on any mock element: - Ask: "Am I testing real component behavior or just mock existence?" - - IF testing mock existence: - STOP - Delete the assertion or unmock the component - - Test real behavior instead -``` - -## Anti-Pattern 2: Test-Only Methods in Production - -**The violation:** -```typescript -// ❌ BAD: destroy() only used in tests -class Session { - async destroy() { // Looks like production API! - await this._workspaceManager?.destroyWorkspace(this.id); - // ... cleanup - } -} - -// In tests -afterEach(() => session.destroy()); -``` - -**Why this is wrong:** -- Production class polluted with test-only code -- Dangerous if accidentally called in production -- Violates YAGNI and separation of concerns -- Confuses object lifecycle with entity lifecycle - -**The fix:** -```typescript -// ✅ GOOD: Test utilities handle test cleanup -// Session has no destroy() - it's stateless in production - -// In test-utils/ -export async function cleanupSession(session: Session) { - const workspace = session.getWorkspaceInfo(); - if (workspace) { - await workspaceManager.destroyWorkspace(workspace.id); - } -} - -// In tests -afterEach(() => cleanupSession(session)); -``` - -### Gate Function - -``` -BEFORE adding any method to production class: - Ask: "Is this only used by tests?" - - IF yes: - STOP - Don't add it - Put it in test utilities instead - - Ask: "Does this class own this resource's lifecycle?" - - IF no: - STOP - Wrong class for this method -``` - -## Anti-Pattern 3: Mocking Without Understanding - -**The violation:** -```typescript -// ❌ BAD: Mock breaks test logic -test('detects duplicate server', () => { - // Mock prevents config write that test depends on! - vi.mock('ToolCatalog', () => ({ - discoverAndCacheTools: vi.fn().mockResolvedValue(undefined) - })); - - await addServer(config); - await addServer(config); // Should throw - but won't! -}); -``` - -**Why this is wrong:** -- Mocked method had side effect test depended on (writing config) -- Over-mocking to "be safe" breaks actual behavior -- Test passes for wrong reason or fails mysteriously - -**The fix:** -```typescript -// ✅ GOOD: Mock at correct level -test('detects duplicate server', () => { - // Mock the slow part, preserve behavior test needs - vi.mock('MCPServerManager'); // Just mock slow server startup - - await addServer(config); // Config written - await addServer(config); // Duplicate detected ✓ -}); -``` - -### Gate Function - -``` -BEFORE mocking any method: - STOP - Don't mock yet - - 1. Ask: "What side effects does the real method have?" - 2. Ask: "Does this test depend on any of those side effects?" - 3. Ask: "Do I fully understand what this test needs?" - - IF depends on side effects: - Mock at lower level (the actual slow/external operation) - OR use test doubles that preserve necessary behavior - NOT the high-level method the test depends on - - IF unsure what test depends on: - Run test with real implementation FIRST - Observe what actually needs to happen - THEN add minimal mocking at the right level - - Red flags: - - "I'll mock this to be safe" - - "This might be slow, better mock it" - - Mocking without understanding the dependency chain -``` - -## Anti-Pattern 4: Incomplete Mocks - -**The violation:** -```typescript -// ❌ BAD: Partial mock - only fields you think you need -const mockResponse = { - status: 'success', - data: { userId: '123', name: 'Alice' } - // Missing: metadata that downstream code uses -}; - -// Later: breaks when code accesses response.metadata.requestId -``` - -**Why this is wrong:** -- **Partial mocks hide structural assumptions** - You only mocked fields you know about -- **Downstream code may depend on fields you didn't include** - Silent failures -- **Tests pass but integration fails** - Mock incomplete, real API complete -- **False confidence** - Test proves nothing about real behavior - -**The Iron Rule:** Mock the COMPLETE data structure as it exists in reality, not just fields your immediate test uses. - -**The fix:** -```typescript -// ✅ GOOD: Mirror real API completeness -const mockResponse = { - status: 'success', - data: { userId: '123', name: 'Alice' }, - metadata: { requestId: 'req-789', timestamp: 1234567890 } - // All fields real API returns -}; -``` - -### Gate Function - -``` -BEFORE creating mock responses: - Check: "What fields does the real API response contain?" - - Actions: - 1. Examine actual API response from docs/examples - 2. Include ALL fields system might consume downstream - 3. Verify mock matches real response schema completely - - Critical: - If you're creating a mock, you must understand the ENTIRE structure - Partial mocks fail silently when code depends on omitted fields - - If uncertain: Include all documented fields -``` - -## Anti-Pattern 5: Integration Tests as Afterthought - -**The violation:** -``` -✅ Implementation complete -❌ No tests written -"Ready for testing" -``` - -**Why this is wrong:** -- Testing is part of implementation, not optional follow-up -- TDD would have caught this -- Can't claim complete without tests - -**The fix:** -``` -TDD cycle: -1. Write failing test -2. Implement to pass -3. Refactor -4. THEN claim complete -``` - -## When Mocks Become Too Complex - -**Warning signs:** -- Mock setup longer than test logic -- Mocking everything to make test pass -- Mocks missing methods real components have -- Test breaks when mock changes - -**your human partner's question:** "Do we need to be using a mock here?" - -**Consider:** Integration tests with real components often simpler than complex mocks - -## TDD Prevents These Anti-Patterns - -**Why TDD helps:** -1. **Write test first** → Forces you to think about what you're actually testing -2. **Watch it fail** → Confirms test tests real behavior, not mocks -3. **Minimal implementation** → No test-only methods creep in -4. **Real dependencies** → You see what the test actually needs before mocking - -**If you're testing mock behavior, you violated TDD** - you added mocks without watching test fail against real code first. - -## Quick Reference - -| Anti-Pattern | Fix | -|--------------|-----| -| Assert on mock elements | Test real component or unmock it | -| Test-only methods in production | Move to test utilities | -| Mock without understanding | Understand dependencies first, mock minimally | -| Incomplete mocks | Mirror real API completely | -| Tests as afterthought | TDD - tests first | -| Over-complex mocks | Consider integration tests | - -## Red Flags - -- Assertion checks for `*-mock` test IDs -- Methods only called in test files -- Mock setup is >50% of test -- Test fails when you remove mock -- Can't explain why mock is needed -- Mocking "just to be safe" - -## The Bottom Line - -**Mocks are tools to isolate, not things to test.** - -If TDD reveals you're testing mock behavior, you've gone wrong. - -Fix: Test real behavior or question why you're mocking at all. diff --git a/skills/test-driven-development/writing-good-tests.md b/skills/test-driven-development/writing-good-tests.md new file mode 100644 index 0000000000..d3c4482fd3 --- /dev/null +++ b/skills/test-driven-development/writing-good-tests.md @@ -0,0 +1,198 @@ +# Writing Good Tests + +**Load this reference when:** writing or changing tests, adding mocks, or +adding cleanup/helper methods for tests. + +## Overview + +A test exists to catch a specific break. Two principles govern everything +here: + +``` +1. Every test names the break it catches +2. Every test exercises the real thing +``` + +Strict TDD produces both naturally: a test written first and watched +failing against real code has already proven it can fail, and only earns +a mock when the real dependency proves slow or external. + +## Principle 1: Name the Break + +Before writing the test body, answer: **what production change should +make this test fail — and is that change a bug or a decision?** A test +earns its place by catching a wrong branch, missing side effect, wrong +argument, boundary case, or broken contract. + +**Derive expectations independently.** Use literals and hand-checked +fixtures; table-driven tests with literal `want` values are the preferred +shape. An expectation computed by the code under test — or its helpers — +passes no matter what that code does: + +```typescript +// ❌ Mirror assertion: the same builder computes both sides — always true +const expected = buildSearchQuery({ tag: 'urgent' }); +expect(buildSearchQuery({ tag: 'urgent' })).toBe(expected); + +// ✅ Hand-derived literal +expect(buildSearchQuery({ tag: 'urgent' })).toBe('tag:"urgent"'); +``` + +**No change detectors.** If only intentional decisions can fail a test — +a constant's value, exact message wording, private structure — it fires +on redesign and sleeps through bugs. Test the behavior that depends on +the decision: not `expect(MAX_RETRIES).toBe(5)` but "a failing call is +retried 5 times and the 6th attempt never happens." + +**Behavior, not text.** Asserting that a script, skill, or config +contains an exact line proves only that the source is the source. Run +scripts against controlled inputs and assert outputs, side effects, or +exit codes. Documents that instruct agents are tested by the consuming +agent's behavior (superpowers:writing-skills); prose for humans earns no +test at all. + +**Your code, not the framework.** Test the contract your code makes at +its boundaries — the route you register, the query you emit, the payload +you produce. Upstream mechanics are their maintainers' tests to write +(the classic: asserting your router invokes a registered handler — that +is the framework's test, not yours). When upstream behavior genuinely +surprised you, write one narrow characterization test naming the +assumption. The same boundary applies inside your code: constructors, +getters, constants, and trivial forwarding earn tests only when they +validate, normalize, default, derive, enforce, or cause side effects — +otherwise assert the first consumer-visible result that depends on them. + +### Gate Function + +``` +BEFORE writing the test body: + Name the production change that would make this test fail. + + Cannot name one → redesign around an observable behavior + "The source text changed" → run the artifact and assert its effects + Only intentional decisions → change detector; test the behavior + that depends on the decision + + Confirm the expected value is derived without the code under test. + IF it reuses the code's logic or helpers: + Replace it with a literal or hand-checked fixture +``` + +## Principle 2: Exercise the Real Thing + +**The mock earns no assertions.** A mock assertion passes when the mock +is present and fails when it is absent — it says nothing about the +component. Assert the real component's behavior; if the mock is what you +are checking, unmock it or delete the assertion. + +```typescript +// ✅ Real behavior +expect(screen.getByRole('navigation')).toBeInTheDocument(); + +// ❌ Mock existence +expect(screen.getByTestId('sidebar-mock')).toBeInTheDocument(); +``` + +**your human partner's correction:** "Are we testing the behavior of a +mock?" + +**Mock at the right level.** Learn every side effect of the real method +before replacing it; mock the slow or external operation and keep what +the test depends on real. When unsure, run the test against the real +implementation first and observe what actually needs to happen. + +```typescript +// ❌ The mock swallows the config write that duplicate detection reads +vi.mock('ToolCatalog', () => ({ + discoverAndCacheTools: vi.fn().mockResolvedValue(undefined) +})); + +// ✅ Mock only the slow server startup; the config write stays real +vi.mock('MCPServerManager'); +``` + +**Make doubles specific.** When arguments, call counts, or ordering are +part of the contract, assert them — a fake that accepts anything verifies +nothing. Give each branch (success, error, malformed) its own fixture or +spy, so the wrong branch cannot satisfy the expectation. + +**Mirror real data completely.** Mock the complete structure as it exists +in reality — all documented fields — not just the ones your test reads. +Partial mocks fail silently when downstream code reads an omitted field: +the test passes while integration breaks. + +**Production classes carry production methods only.** Cleanup that only +tests need lives in test utilities, never as a `destroy()` on the +production class. Ask: is this method called only from tests? Does this +class own this resource's lifecycle? Wrong answers → test utility. + +**Prefer real components over complex mocks.** When mock setup outgrows +the test logic, mocks miss methods the real components have, or tests +break when the mock changes, switch to an integration test with real +components. **your human partner's question:** "Do we need to be using a +mock here?" + +### Gate Function + +``` +BEFORE adding a mock or test helper: + List the real method's side effects; keep the ones the test + depends on real — mock the slow/external level below them. + + Mock responses mirror the complete real structure. + + A method only tests call lives in test utilities, not production. + + About to assert on the mock itself? + Unmock it or delete the assertion. +``` + +## Tests Ship With the Implementation + +The TDD cycle — failing test, minimal implementation, refactor — is what +"complete" means. Ship the tests the behavior needs and only those: +trivial code and human prose earn none, and a test written to satisfy +process costs maintenance forever. + +## The Mutation Check + +Before finishing, mentally mutate the production code; at least one test +should fail for each realistic mutation: + +- Wrong constant or argument +- Wrong branch handler +- Missing state change or side effect +- Empty or default return +- Missing validation for zero, empty, nil, unauthorized, or malformed input + +A mutation nothing catches marks the behavior as unprotected — or the +test as tautological. + +## Quick Reference + +| When you... | Do | +|-------------|-----| +| Write any test | Name the break it catches — a bug, not a decision | +| Build an expected value | Derive it by hand; never with the code under test | +| Test a script or document | Run it / pressure-test its consumer; never grep its text | +| Reach for a dependency test | Test your boundary contract, not their documented mechanics | +| Want to assert on a mocked element | Test the real component, or unmock it | +| Are about to mock a method | Learn its side effects; mock the slow/external level | +| Build a mock response | Mirror the real structure completely | +| Need cleanup only tests use | Put it in test utilities | +| Watch mock setup balloon | Switch to an integration test with real components | +| Finish a test file | Run the mutation check | + +## Warning Signs + +- Setup and assertion share the same object, guaranteeing equality +- The test can fail only through a panic, crash, or missing selector +- The test fails on every intentional change, never on accidental breakage +- Expected values are hidden behind loops, builders, or helpers +- The test greps source text, or asserts a removed symbol stays removed +- The test would still matter if only the framework remained +- The test exists for coverage, checking no side effect or outcome +- An assertion checks a `*-mock` test ID, or fails if you remove the mock +- A method is called only from test files +- Mock setup is more than half the test, or you can't explain why the mock is needed +- Mocking "just to be safe"