Skip to content

feat: add experiment-designer skill#494

Open
xjrmh wants to merge 1 commit intoanthropics:mainfrom
xjrmh:add-experiment-designer-skill
Open

feat: add experiment-designer skill#494
xjrmh wants to merge 1 commit intoanthropics:mainfrom
xjrmh:add-experiment-designer-skill

Conversation

@xjrmh
Copy link

@xjrmh xjrmh commented Mar 2, 2026

Summary

  • Adds an experiment-designer skill that guides users through designing statistically rigorous experiments via a structured 8-step workflow
  • Supports 6 experiment types: A/B test, cluster randomized, switchback, causal inference, factorial, and multi-armed bandit
  • Includes reference files for statistical formulas, experiment type selection, and a common metrics library with baselines

Files

File Purpose
skills/experiment-designer/SKILL.md Main skill — 8-step workflow, inference rules, output template
skills/experiment-designer/experiment-types.md Detailed reference for 6 experiment types
skills/experiment-designer/statistics.md Sample size formulas, type-specific adjustments, duration estimation
skills/experiment-designer/metrics-library.md 14 common metrics with baselines and variance values

How it works

Invoke /experiment-designer and describe what you want to test. The skill walks through:

  1. Experiment type selection (inferred from context)
  2. Metrics definition (PRIMARY + GUARDRAIL required)
  3. Statistical parameters & sample size calculation
  4. Randomization strategy
  5. Variance reduction (CUPED, stratification, etc.)
  6. Risk assessment & pre-launch checklist
  7. Monitoring & stopping rules
  8. Summary — produces a structured experiment design document

Test plan

  • Install skill via cp -r skills/experiment-designer ~/.claude/skills/
  • Invoke /experiment-designer I want to test a new checkout flow and verify it walks through all 8 steps
  • Verify statistical formulas produce correct sample size calculations
  • Confirm the final output is a well-structured markdown design document

🤖 Generated with Claude Code

A skill that guides users through designing statistically rigorous
experiments via an 8-step workflow. Supports A/B tests, cluster
randomized, switchback, causal inference, factorial, and multi-armed
bandit designs.

Includes sample size formulas, a common metrics library with baselines,
and generates a structured experiment design document.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants