diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..60045be --- /dev/null +++ b/.gitattributes @@ -0,0 +1,5 @@ +# Shell scripts must check out LF everywhere: bash (incl. Git Bash on Windows) chokes on +# CRLF (`set -uo pipefail\r` is an invalid option), so an autocrlf checkout would install +# broken hooks and rails scripts — and broke the Windows CI leg running the hook tests. +# pwsh is CRLF-tolerant, so .ps1 files are left to the platform default. +*.sh text eol=lf diff --git a/README.md b/README.md index 08a7cc0..feee213 100644 --- a/README.md +++ b/README.md @@ -153,6 +153,12 @@ Full enterprise stack with compliance: - **Visual verification:** Screenshot capture with manual baseline comparison - **Evaluation criteria:** Phase-scoped rubrics for requirements (testability, traceability), design (ADR completeness, interface specificity), foundation (section plan verifiability), and Build-loop code (Result pattern, immutable state, FluentValidation, API docs) +### ado-enterprise +microsoft-enterprise's stack, hosted on Azure DevOps: +- **Stack:** C#/.NET 8, Angular 17, SQL Server, Azure (identical to microsoft-enterprise) +- **Repos & CI/CD:** Azure Repos + Azure Pipelines (`.azuredevops/pipelines/`, branch policies via `az repos policy`) +- **Quality / Compliance / Conventions:** same as microsoft-enterprise (80% coverage, TDD, SOC 2 gates) + ### starter Minimal profile for quick start: - **Stack:** Configurable (defaults to TypeScript/Node) diff --git a/SKILL.md b/SKILL.md index 30e531c..c3fc022 100644 --- a/SKILL.md +++ b/SKILL.md @@ -64,6 +64,7 @@ For long-running phases (especially the Build loop), session continuity is maint Profiles configure the plugin for your company/team: - `microsoft-enterprise` — C#/.NET 8 + Angular 17 + Azure + SOC 2 compliance +- `ado-enterprise` — microsoft-enterprise's stack on Azure Repos + Azure Pipelines - `starter` — Minimal profile, no compliance, quick start Profiles define: technology stack, quality thresholds (coverage, file size limits), compliance frameworks, coding conventions (commit format, naming, immutability). diff --git a/commands/sdlc-doctor.md b/commands/sdlc-doctor.md index 06c8b69..1a4b9ae 100644 --- a/commands/sdlc-doctor.md +++ b/commands/sdlc-doctor.md @@ -12,25 +12,25 @@ Run it after `/sdlc-setup`, when onboarding a second developer, and any time a g ```bash uv run --project ${CLAUDE_PLUGIN_ROOT}/scripts ${CLAUDE_PLUGIN_ROOT}/scripts/doctor.py ``` - Add `--offline` to skip the checks that need `gh` (repo secrets and branch protection). Add `--repo ` to check a repo other than the current directory. + Add `--offline` to skip the checks that need the platform CLI — `gh` on GitHub installs (repo secrets, branch protection), `az` on Azure DevOps installs (variable groups, branch policies). Add `--repo ` to check a repo other than the current directory. 2. **Report the results verbatim.** Each line is already `PASS` / `FAIL` / `WARN` with the fix attached. Do not summarise away the fix lines — they are the actionable part. 3. **Explain what each failure means, in plain terms**, before offering to act. What matters is the consequence, not the check name: - *missing `pwsh`* — the hooks are registered but can never run, so an agent can finish a turn on a red build and nothing objects - *script not executable* — the gate shells out and gets `Permission denied`; because the gates fail closed, this reads as a blocked merge with a confusing reason - - *missing secret* — the named workflows fail closed on every PR - - *no active ruleset* — the gates run and report, but a red PR can still merge + - *missing secret (GitHub) / missing variable group (ADO)* — the named workflows/pipelines fail closed on every PR + - *no active ruleset (GitHub) / no branch policies (ADO)* — the gates run and report, but a red PR can still merge -4. **Offer to fix what is safely fixable.** `chmod +x` on the installed scripts is safe and mechanical — offer it. Setting a repository secret is not: it needs a real credential value from the human, so tell them the exact `gh secret set` command and let them run it. Never invent a secret value. +4. **Offer to fix what is safely fixable.** `chmod +x` on the installed scripts is safe and mechanical — offer it. Providing a credential is not: it needs a real value from the human, so give them the platform-appropriate fix and let them run it — on GitHub the exact `gh secret set` command; on Azure DevOps the variable group to create and link (Key-Vault-backed, via `az pipelines variable-group`). Never invent a secret value. -5. **Treat `WARN` as information, not a problem.** A warning marks something that could not be determined (Windows cannot see POSIX permission bits; `gh` may not be authenticated) or setup a later phase is meant to finish (an unfilled `<>` carries the phase that fills it). Do not push the user to "clear" warnings. +5. **Treat `WARN` as information, not a problem.** A warning marks something that could not be determined (Windows cannot see POSIX permission bits; `gh`/`az` may not be authenticated) or setup a later phase is meant to finish (an unfilled `<>` carries the phase that fills it). Do not push the user to "clear" warnings. 6. **If the harness is not installed at all**, stop and point at `/sdlc-setup`. The rest of the report is meaningless without it. ## Arguments -- `--offline` — skip the checks that need `gh` (secrets, branch protection) +- `--offline` — skip the checks that need `gh`/`az` (secrets / variable groups, branch protection / branch policies) - `--repo ` — check a different repo (default: current directory) ## Exit codes @@ -39,6 +39,8 @@ Run it after `/sdlc-setup`, when onboarding a second developer, and any time a g ## Notes -Required secrets are read from **this repo's installed workflows**, not from a fixed list. A repo that has adapted `ANTHROPIC_API_KEY` to `CLAUDE_CODE_OAUTH_TOKEN`, or deleted the eval workflows, is checked against what it actually runs — the doctor must never tell someone to "fix" a working setup. +The doctor is **pack-aware**: it reads the installed CI/CD pack from `.claude/harness-manifest.json` and the platform-facing checks follow it — an Azure DevOps install is checked with `az` (variable groups, branch policies) and is never told to install `gh` for a platform it does not use. + +Required secrets (GitHub) / variable-group references (ADO) are read from **this repo's installed workflows/pipelines**, not from a fixed list. A repo that has adapted `ANTHROPIC_API_KEY` to `CLAUDE_CODE_OAUTH_TOKEN`, or deleted the eval workflows, is checked against what it actually runs — the doctor must never tell someone to "fix" a working setup. See `ONBOARDING.md` in the repo root for the day-1 checklist this command automates. diff --git a/commands/sdlc-harness.md b/commands/sdlc-harness.md index 12694ea..d61ba02 100644 --- a/commands/sdlc-harness.md +++ b/commands/sdlc-harness.md @@ -18,15 +18,23 @@ uv run --project ${CLAUDE_PLUGIN_ROOT}/scripts ${CLAUDE_PLUGIN_ROOT}/scripts/ins ### Step 2: Report Summarize what was written vs skipped, then remind the user: -- Fill any remaining `{{PLACEHOLDER}}` tokens in `CLAUDE.md` and the workflows. -- **Prove the rails** before trusting them — the shakedown drills in `.github/RAILS.md`. +- Fill any remaining `{{PLACEHOLDER}}` tokens in `CLAUDE.md` and the workflows/pipelines. +- **Prove the rails** before trusting them — the shakedown drills in `.github/RAILS.md` + (the same path holds the ADO guide on azure-devops installs — the pack deliberately overlays it). - Apply branch protection if on GitHub: `bash scripts/rails/apply-branch-protection.sh`. + On Azure DevOps, configure branch policies instead — needs `az` + the `azure-devops` extension: + `bash scripts/rails/configure-branch-policies.sh --dry-run` to preview, then re-run without the + flag to apply. ## What it installs `CLAUDE.md` (governance), `specs/spec-template.md`, `.claude/{settings.json,hooks,agents,skills}`, `.github/workflows/` (ci, grader, correctness, security, deploy-dev, eval-regression, eval-suite) + `RAILS.md`, `.github/{profile/rubrics,rulesets,CODEOWNERS,eval-bypasses.md}`, `scripts/rails/`, -`eval-datasets/`, `prompts/`, and `infra/`. See `.claude/agents/README.md` for the agent/skill +`eval-datasets/`, `prompts/`, and `infra/`. On `platform: azure-devops` profiles the platform +surface is realized as `.azuredevops/pipelines/` (the same rails as Azure Pipelines) and +`.azuredevops/rails/` (rubrics + `branch-policies.json` — build-validation and required-reviewer +policies standing in for rulesets and CODEOWNERS); `.github/RAILS.md` still carries the operator +guide. See `.claude/agents/README.md` for the agent/skill catalog and the on-demand menu. ## Error Handling diff --git a/commands/sdlc-setup.md b/commands/sdlc-setup.md index b7dc2d7..f3cf522 100644 --- a/commands/sdlc-setup.md +++ b/commands/sdlc-setup.md @@ -15,6 +15,7 @@ List available profiles from the plugin's `profiles/` directory (exclude `_schem Present every profile found there to the user. The current built-ins: - **microsoft-enterprise** — C#/.NET 8 + Angular 17 + Azure, SOC 2 compliance, 80% coverage minimum, TDD required +- **ado-enterprise** — microsoft-enterprise's stack on Azure Repos + Azure Pipelines (`platform: azure-devops`); same SOC 2 gates, coverage, and TDD bar - **starter** — Minimal profile, no compliance gates, quick start for any stack - **creative-tooling** — Python/uv-scripts + pytest for creative pipelines (ComfyUI registry/inventory tooling); 80% coverage, TDD + code/security review required, schema- and cross-reference-integrity evaluation criteria, no compliance frameworks @@ -52,7 +53,10 @@ Install the standard-aligned harness (the kit) from the plugin's bundled `harnes the repo. This lays down the governance `CLAUDE.md`, `.claude/{settings,hooks,agents,skills}`, the CI workflows in `.github/workflows/` (7 files: 5 rail gates — ci, grader, correctness, security, deploy-dev — plus 2 eval workflows), the `profile/` rubrics + branch-protection ruleset, and the -`infra/` starters. Idempotent — existing files are left in place and reported as SKIPPED (pass +`infra/` starters. On `platform: azure-devops` profiles the platform surface lands as Azure +Pipelines under `.azuredevops/pipelines/` with branch policies as code +(`.azuredevops/rails/branch-policies.json`) instead of `.github/workflows/` + the ruleset — same +rails, different realization. Idempotent — existing files are left in place and reported as SKIPPED (pass `--force` only when you intend to overwrite). One exception: the JSON merge targets (`.mcp.json` and `.claude/settings.json`) are re-merged on every run by design — the installer deep-merges the payload's entries into them rather than skipping, so pack additions always land. The install @@ -68,7 +72,8 @@ pipeline workflows, which overlay the core placeholders). A profile may also lis small static surface and the installer PRINTS their manual setup steps (self-installing tools are never run by the installer). The install also writes `.mcp.json` at the repo root — the team's shared MCP servers (core: context7, sequential-thinking, playwright; the dotnet pack adds microsoft-learn; the -azure-devops pack adds the Azure DevOps server with an `<>` token to fill). Each +azure-devops pack adds the Azure DevOps server with an `ADO_ORGANIZATION_NOT_SET` sentinel to fill +— a plain sentinel, not a `<>`, because the value lands in an argv). Each developer approves the set once when they first open the repo; auth-requiring servers authenticate per developer (e.g. `az login`) — no credentials ever go in the file. If the profile declares a frontend (`stack.frontend`), the frontend axis also composes: the generic `ux-reviewer` agent, plus @@ -111,15 +116,22 @@ prove the system can be operated and handed over. Quality thresholds and convent `.sdlc/profile.yaml`. ``` -### Step 7: Apply branch protection (optional — needs GitHub + `gh`) -The harness ships a branch-protection ruleset (`.github/rulesets/branch-protection.json`) and an -applier. If the repo is on GitHub and `gh` is authenticated, offer to apply it: +### Step 7: Apply branch protection / branch policies (optional — needs the platform CLI) +On GitHub, the harness ships a branch-protection ruleset (`.github/rulesets/branch-protection.json`) +and an applier. If the repo is on GitHub and `gh` is authenticated, offer to apply it: ```bash bash scripts/rails/apply-branch-protection.sh ``` -This makes the five blocking checks (build-and-test, spec-gate, grader, correctness-review, security-review) +On Azure DevOps, the analogue is branch policies as code (`.azuredevops/rails/branch-policies.json`) +and its applier — needs `az` with the `azure-devops` extension, logged in (`az login`). Offer the +dry run first, then apply: +```bash +bash scripts/rails/configure-branch-policies.sh --dry-run # show the plan, no writes +bash scripts/rails/configure-branch-policies.sh +``` +Either way, this makes the five blocking checks (build-and-test, spec-gate, grader, correctness-review, security-review) + a non-author approval mandatory at merge; deploy-dev runs post-merge and is not a merge check. -Skip if the repo isn't on GitHub yet; the ruleset stays in the repo to apply later. +Skip if the repo isn't on its platform yet; the ruleset / policy file stays in the repo to apply later. ### Step 8: Confirmation Display: @@ -132,13 +144,18 @@ Harness: CLAUDE.md, .claude/, .github/workflows (7 workflows: 5 rails + 2 eval), Next steps: 1. Fill the {{PLACEHOLDER}} tokens in CLAUDE.md (stack, glossary, gated paths); - on Azure DevOps, also replace <> in .mcp.json + on Azure DevOps, also replace ADO_ORGANIZATION_NOT_SET in .mcp.json (docs/harness.md explains every installed piece — point the team there) 2. PROVE THE RAILS before trusting them — run the shakedown drills in .github/RAILS.md 3. Run /sdlc to start the Phase 0 discovery interview 4. Run /sdlc-gate when ready to check exit criteria; /sdlc-next to advance ``` +On `platform: azure-devops` profiles, render the `Harness:` line with the ADO layout instead: +`.azuredevops/pipelines/` (Azure Pipelines rails + eval pipelines) + branch policies in +`.azuredevops/rails/`. Leave the `.github/RAILS.md` drills path as-is — it is correct on both +platforms (the azure-devops pack deliberately overlays that same path with the ADO guide). + ### Step 9: Validate Run the profile validator to confirm the setup is healthy: ```bash diff --git a/commands/sdlc-upgrade.md b/commands/sdlc-upgrade.md index 56799fe..e48b658 100644 --- a/commands/sdlc-upgrade.md +++ b/commands/sdlc-upgrade.md @@ -57,6 +57,31 @@ version; never replace the whole file. - Stage the upgraded files + `.claude/harness-manifest.json` and commit, e.g. `chore: upgrade delivery harness to `. +## Switching CI platform + +Moving a repo between CI platforms (e.g. GitHub Actions → Azure DevOps) is just an upgrade with +a changed profile — the classification table above already describes everything that happens. +The runbook: + +1. Edit `stack.ci_cd.platform` in the frozen `.sdlc/profile.yaml` (`github-actions` ↔ + `azure-devops`). This is the one deliberate use of the Step 1 warning: here the old pack + classifying as RETIRED is the point, not a misreport. +2. Dry-run the upgrade (Step 1) with that profile. Expect the old platform's pack files to + classify `RETIRED` and the new pack's files as `NEW` — that is the swap report, not an error. + The core payload has been platform-aware since Fold A, so on an ADO target the retired GitHub + files no longer churn as `UPDATE`/`CONFLICT` noise. +3. Apply (Step 3). The new pack lands; retired files stay on disk with their manifest entries + dropped. +4. Delete the retired files yourself — upgrade never deletes (see the `RETIRED` row). On a + GitHub → ADO switch that means the `.github/workflows/` rail YAMLs, + `rulesets/branch-protection.json` + `scripts/rails/apply-branch-protection.sh`, and + `CODEOWNERS`; their ADO analogues arrive as `NEW` (`.azuredevops/pipelines/`, branch policies + via `scripts/rails/configure-branch-policies.sh`, a required-reviewer policy in place of + CODEOWNERS). +5. Re-run `/sdlc-doctor` — it follows the installed pack (`az` checks on ADO, `gh` on GitHub) — + and re-prove the rails with the shakedown drills in `.github/RAILS.md` (the rails guide keeps + that path on both platforms; the ADO pack deliberately overlays it). + ## Error Handling - If uv is not installed: `pip install uv` or `brew install uv`. - Exit 2 with `ERROR:` means a bad payload/profile or corrupt manifest — fix and re-run; nothing diff --git a/docs/architecture.md b/docs/architecture.md index 1fae756..2b4f268 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -18,61 +18,47 @@ Comprehensive architectural documentation for `claude-code-sdlc` -- a Claude Cod ## 1. Plugin Anatomy -### 1.1 Plugin Manifest (`plugin.json`) +### 1.1 Plugin Manifest (`.claude-plugin/plugin.json`) -The plugin manifest is the discovery entry point for Claude Code. It declares every component the plugin provides: +The plugin manifest is **metadata only** — name, version, description, author, homepage, +license, keywords. It does not register components; there is no component or profile +registry inside it. The marketplace entry (`.claude-plugin/marketplace.json`) declares +`"source": "."`, so the whole repository directory ships as the plugin payload. ```json { "name": "claude-code-sdlc", - "version": "0.1.0", - "description": "SDLC orchestration plugin for Claude Code -- configurable, company-profile-driven lifecycle management from discovery through monitoring.", - "author": { - "name": "Matt Kruczek", - "url": "https://github.com/MCKRUZ" - }, - "category": "sdlc", - "skills": "SKILL.md", - "commands": "commands/", - "agents": "agents/", - "profiles": { - "microsoft-enterprise": { - "description": "C#/.NET 8 + Angular 17 + Azure + SOC 2 compliance", - "source": "profiles/microsoft-enterprise", - "skills": ["azure-entra-auth", "azure-app-service", "azure-sql", - "azure-key-vault", "azure-app-insights"] - }, - "starter": { - "description": "Minimal profile, no compliance, quick start for any stack", - "source": "profiles/starter" - } - }, - "hooks": [ - "hooks/sdlc-session-start.ps1", - "hooks/sdlc-phase-inject.ps1" - ], - "scripts": { - "runtime": "uv", - "root": "scripts/" - } + "version": "1.3.0", + "description": "SDLC orchestration for Claude Code + one-command install of the full delivery harness (...)", + "author": { "name": "Matt Kruczek", "url": "https://github.com/MCKRUZ" }, + "homepage": "https://github.com/MCKRUZ/claude-code-sdlc", + "repository": "https://github.com/MCKRUZ/claude-code-sdlc", + "license": "MIT", + "keywords": ["sdlc", "lifecycle", "compliance", "quality", "orchestration", "..."] } ``` -### 1.2 Field-by-Field Breakdown - -| Field | Type | Purpose | -|-------|------|---------| -| `name` | string | Unique plugin identifier. Used in Claude Code's plugin registry. | -| `version` | semver | Plugin version. Follows semantic versioning. | -| `description` | string | Human-readable summary shown in plugin listings. | -| `author` | object | Author metadata with `name` and `url` fields. | -| `category` | string | Plugin category for discovery (`"sdlc"`). | -| `skills` | path | Path to `SKILL.md` -- the primary entry point loaded when the plugin activates. | -| `commands` | path | Directory containing slash command definitions (`.md` files). | -| `agents` | path | Directory containing custom agent definitions (`.md` files). | -| `profiles` | object | Map of company/stack profiles. Each entry has a `description`, `source` directory, and optional `skills` array for domain-specific Claude Code skills. | -| `hooks` | array | PowerShell scripts triggered on session lifecycle events. | -| `scripts` | object | Automation scripts config: `runtime` specifies the runner (`uv` for Python), `root` is the scripts directory. | +(Abridged — see `.claude-plugin/plugin.json` for the current contents; the `version` there is +the source of truth.) + +### 1.2 Component Discovery (by convention, not registration) + +Every component is discovered from its conventional location in the plugin directory: + +| Component | Location | How it is discovered | +|-----------|----------|----------------------| +| Main skill | `SKILL.md` | Loaded when the plugin activates. | +| Slash commands | `commands/*.md` | One file per command, picked up by filename. | +| Agents | `agents/*.md` | One file per agent definition. | +| Hooks | `hooks/` | Session/phase context-injection scripts, wired via hook configuration. | +| **Profiles** | `profiles/*/profile.yaml` | **Runtime directory listing** — `/sdlc-setup` lists `profiles/` (excluding `_schema.yaml`) and presents every profile found, including ones it has never seen. Adding a profile is creating the directory; no registration step exists. | +| Scripts | `scripts/*.py` | Invoked explicitly by commands/hooks via `uv run`. | + +Two consequences of the profiles row: a new profile ships automatically (the payload is the +whole directory), and its `company.profile_id` must equal its directory name — compliance +gates resolve at `profiles//compliance/`, and +`scripts/tests/test_validate_profile.py::TestOnDiskProfiles` pins both properties for every +profile on disk. ### 1.3 Component Loading Sequence @@ -366,6 +352,8 @@ claude-code-sdlc/ Plugin root (installed or symlinked) | | |-- profile.yaml C#/.NET 8 + Angular 17 + Azure + SOC 2 | | |-- claude-md-template.md CLAUDE.md template for target projects | | +-- switchboard-rules.json Agent routing rules for this profile +| |-- ado-enterprise/ +| | +-- profile.yaml microsoft-enterprise's stack on Azure Repos + Azure Pipelines | |-- starter/ | | +-- profile.yaml Minimal profile, no compliance | +-- creative-tooling/ diff --git a/docs/commands.md b/docs/commands.md index 908233f..6e09c77 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -68,6 +68,7 @@ The command looks for `.sdlc/state.yaml` in the current directory. If it already Available profiles are listed from the plugin's `profiles/` directory (excluding `_schema.yaml`). Current built-in profiles: - **microsoft-enterprise** -- C#/.NET 8 + Angular 17 + Azure, SOC 2 compliance, 80% coverage minimum, TDD required. +- **ado-enterprise** -- microsoft-enterprise's stack on Azure Repos + Azure Pipelines; SOC 2 compliance, 80% coverage minimum, TDD required. - **starter** -- Minimal profile with no compliance gates; a quick start suitable for any stack. **Step 3: Project Configuration** diff --git a/docs/profiles.md b/docs/profiles.md index 485f117..e444f1c 100644 --- a/docs/profiles.md +++ b/docs/profiles.md @@ -46,6 +46,10 @@ profiles/ profile.yaml # Full enterprise profile compliance/ soc2-gates.yaml # SOC 2 gate definitions per phase + ado-enterprise/ + profile.yaml # microsoft-enterprise's stack on Azure Repos + Azure Pipelines + compliance/ + soc2-gates.yaml # SOC 2 gate definitions per phase creative-tooling/ profile.yaml # Claude Code plugin development profile ``` @@ -483,22 +487,21 @@ Supported `check_type` values: | `metric` | Compares a metric against a profile threshold | `metric`, `threshold_key` | | `manual` | Flags for human verification | `description` | -### Step 5: Register in plugin.json +### Step 5: Done — There Is No Registration Step -Add your profile to the `profiles` section of `plugin.json`: +Profiles are **auto-discovered**: `/sdlc-setup` lists the `profiles/` directory at runtime +(excluding `_schema.yaml`) and presents every profile it finds — including ones not named in +any documentation. The plugin manifest (`.claude-plugin/plugin.json`) is metadata only and has +no profile registry; the plugin ships the whole repository directory +(`.claude-plugin/marketplace.json` declares `"source": "."`), so your new directory ships with +it automatically. -```json -{ - "profiles": { - "my-company": { - "description": "Python/FastAPI + React 18 + AWS + GDPR compliance", - "source": "profiles/my-company" - } - } -} -``` +Two things must be true for the profile to work fully, both enforced by +`scripts/tests/test_validate_profile.py::TestOnDiskProfiles` the moment the directory exists: -The `source` field points to the profile directory. Optionally, include a `skills` array to associate stack-specific skills with the profile. +1. `profile.yaml` passes the schema (Step 3). +2. `company.profile_id` equals the directory name — compliance gates are resolved at + `profiles//compliance/`, so a mismatched id silently loses its gates (Step 4). --- @@ -695,5 +698,5 @@ When writing evaluation criteria for your custom profile: | Phase registry | `phases/phase-registry.yaml` | Phase metadata and gate definitions | | Schema file | `profiles/_schema.yaml` | The validation schema itself | | SOC 2 gates | `profiles/microsoft-enterprise/compliance/soc2-gates.yaml` | SOC 2 gate definitions | -| Plugin manifest | `plugin.json` | Profile registration and metadata | +| Plugin manifest | `.claude-plugin/plugin.json` | Plugin metadata only — profiles are auto-discovered from `profiles/`, not registered | | Validation rules | `references/validation-rules.md` | Additional validation rule documentation | diff --git a/docs/proposals/ado-repo-support-matrix.md b/docs/proposals/ado-repo-support-matrix.md new file mode 100644 index 0000000..eb29e4e --- /dev/null +++ b/docs/proposals/ado-repo-support-matrix.md @@ -0,0 +1,112 @@ +# ADO Repo Support — Fix Matrix (Folded) + +**Goal:** make the plugin fully usable on projects whose repos live in Azure DevOps, not GitHub. + +**Where we are:** the Azure Pipelines side already exists — `harness/packs/cicd/azure-devops/` is a +complete CI/CD pack, and setting `stack.ci_cd.platform: azure-devops` in a profile selects it. But +several parts of the core still assume GitHub regardless of that setting (the installer copies GitHub +files unconditionally, the doctor requires the `gh` CLI, the PR hooks/skills call `gh`, and much of the +command prose names only GitHub paths). + +**This document:** the remaining work, folded into five packages (A–E). Planned as one spec = one +branch = one PR each; see Sequencing for how they actually landed. Effort is human-scale (**S** = +under half a day, **M** = 1–2 days); Claude-assisted, figure ≈ 1 session per fold plus your review. + +| # | Fix | Effort | Priority | Fix description | GitHub functionality today (unchanged — kept as-is) | What it adds for ADO | +|---|-----|--------|----------|-----------------|----------------------------------|--------------------------| +| A | Installer platform-awareness (`scripts/install_harness.py`) | M | **P0** | Make `DIR_MAP`/`EXTRA_FILES` respect `stack.ci_cd.platform` — skip the GitHub payload when platform ≠ github-actions; install rubrics to the platform's expected dir (`.azuredevops/rails/rubrics/` for ADO); add an ADO golden install tree mirroring `enterprise-tree.txt` to prove both | Unconditionally copies `.github/workflows/` (10 Actions YAMLs), `rulesets/branch-protection.json`, `CODEOWNERS`, `apply-branch-protection.sh` into every target repo; rubrics hard-wired to `.github/profile/rubrics/` | A clean install — only `.azuredevops/pipelines/` lands, no dead GitHub workflows; the grader/security/correctness pipelines actually find their rubrics (today a live bug — the ADO rubric path points at nothing); regression-proofed by the golden tree | +| B | Pack-aware doctor (`scripts/doctor.py` + `test_doctor.py`) | M | **P1** | Branch on the installed pack (from `.claude/harness-manifest.json`): require `az` not `gh`, scan `.azuredevops/pipelines/` for variables, check `az repos policy list` instead of `gh api rulesets`; add test cases for the az branch | `gh` is a FAIL-level required tool; secrets via `gh secret list`; branch protection via `gh api repos/.../rulesets`; only scans `.github/workflows/` | `/sdlc-doctor` validates the real setup (az login, variable groups, branch policies) instead of failing you for not having GitHub tooling | +| C | PR-flow rails on ADO (`harness/settings.json`, `harness/hooks/review-gate.{sh,ps1}`, `harness/skills/pr-writer/SKILL.md`) | S | **P1** | Ship a pack-level settings fragment (merge machinery already exists): allow `az repos pr view/list` + `az pipelines runs`, gate `Edit/Write(./.azuredevops/**)`; add `az repos pr create` to both review-gate trigger regexes; parameterize pr-writer's open step (`az repos pr create`, risk as ADO tag / linked work item) | Allowlists `gh pr view/list/diff`, `gh run list/view`; gates edits to `./.github/**` only; review gate fires on `git push` or `gh pr create` only; pr-writer hard-codes `gh pr create` + `risk:high` label | Claude reads PRs/builds without permission prompts; pipeline definitions get the same tamper-protection as the GitHub rails; the human-review-before-PR rail holds when PRs open via az CLI (today silently bypassed); spec→PR traceability and risk flagging survive on ADO PRs. Verify with one live drill: open a real ADO PR, watch the gate fire | +| D | Prose sweep (commands, harness docs, phases, profiles) | S–M | **P2** | One editorial pass, rule = "name both platforms": `commands/sdlc-setup.md` Step 7 + confirmation, `sdlc-harness.md`, `sdlc-doctor.md`, `templates/phases/close/harness-audit.md`; harness `HARNESS.md`/`README.md`/`CLAUDE.md.template` tour; phases 03/08/09 wording (variable groups / Key Vault, Azure Artifacts release, ADO Boards triage); fix `microsoft-enterprise` profile's `platform: github-actions` claim (the `ado-enterprise` example profile was pulled forward — see Fold 0); document the **platform-switch runbook** in `sdlc-upgrade.md` (edit `stack.ci_cd.platform` in the frozen `.sdlc/profile.yaml` → `/sdlc-upgrade` dry-run + `--apply` → old pack files classify RETIRED, new pack lands as NEW → human deletes retired files → doctor + shakedown drills; clean only once Fold A stops core from emitting `.github/` on every platform) | Setup says branch protection "needs GitHub + `gh`"; deliverables/audit checklists enumerate `.github/...` paths; doctor doc advises `gh secret set`; docs tour `.github/` layout; phase prose says "GitHub secrets / release / Issues"; flagship enterprise profile ironically declares GitHub Actions | Commands are executable prose — the model does what the doc says, so `/sdlc-setup` and the close-phase audit actually walk the ADO path; client-facing `docs/harness.md` describes the repo they actually have; the command-contract lint (`test_command_contracts.py`) auto-validates the script/flag references | +| E | Keyless gate auth via Azure AI Foundry + RBAC (`harness/packs/cicd/azure-devops/`: `run-claude-review.sh`, pipeline YAMLs, `README.md`/`RAILS.md`; `scripts/doctor.py`) | M (incl. spike) | **P1** | Make gate auth a profile choice: `api-key` (today's path, kept as fallback) or `foundry-entra`. Foundry mode: pipelines authenticate via a **service connection with workload identity federation** (`AzureCLI@2` wrapper task); `run-claude-review.sh` accepts the keyless path (`CLAUDE_CODE_USE_FOUNDRY=1`, `ANTHROPIC_FOUNDRY_RESOURCE`, Azure default credential chain or `ANTHROPIC_FOUNDRY_AUTH_TOKEN`) **only when foundry mode is explicitly selected — with no foundry flag the script behaves exactly as today (`ANTHROPIC_API_KEY` required, exit 3 without)**; pin model deployments via `ANTHROPIC_DEFAULT_{SONNET,OPUS}_MODEL` (the rails' `--model sonnet\|opus` aliases map through these); doctor checks the service connection / `az account get-access-token` + `Azure AI User` role instead of the key variable. **Spike first**: Azure Pipelines + Foundry is not an officially documented Claude Code path (GitHub Actions is) — prove the WIF → credential-chain flow with one live drill before speccing the full fold | Gates authenticate with a static `ANTHROPIC_API_KEY` — the GitHub pack via the Claude GitHub App / repo secret, the ADO pack via a Key-Vault-backed variable group (`run-claude-review.sh` exits 3 without it) | No static Claude credential anywhere: the pipeline's managed identity gets the **`Azure AI User`** role scoped to the Foundry resource — revocable, auditable via Entra sign-in logs, rotation-free. Data residency option (Azure-hosted deployment). Strengthens the compliance story the profiles already sell (SOC 2 secret-management posture). Works because the ADO gates already run the real Claude Code CLI, which supports Foundry natively — this is config plumbing, not new machinery | + +## Additive guarantee (design rule for every fold) + +Nothing is replaced. The GitHub path and the API-key auth path keep working exactly as they do +today; all new behavior is opt-in via existing selectors (`stack.ci_cd.platform`, the installed +pack manifest, or an explicit foundry auth mode). + +- **Invariant:** a `github-actions`-profile install and an API-key gate run are byte/behavior-identical + before and after each fold. The only behavior that changes is on the `azure-devops` path — which is + broken today, so there is nothing working there to preserve. +- **Enforcement:** the existing test suite must pass **unmodified** — in particular the golden install + tree (`scripts/tests/golden/enterprise-tree.txt`) pins the GitHub layout, and `test_doctor.py` pins + the `gh` checks. New ADO/Foundry tests are additions beside them, never edits to them. A fold that + needs to change an existing test to pass has broken the guarantee. + +## Fold 0 (pulled forward): `ado-enterprise` profile — DONE + VERIFIED on this branch + +Verified by a 4-lens pass (full pytest suite; fresh-repo e2e drill incl. gate check, artifact +snapshot, and an idempotent upgrade dry-run — 0 CONFLICT / 0 RETIRED; byte-level content audit +vs the sibling profile; repo-wide integration-gap sweep). Fixes applied from the audit: stale +`profile_id` in the copied `switchboard-rules.json` + one prose line in `references/azure-patterns.md`; +the profile added to all five curated profile lists (`commands/sdlc-setup.md`, `README.md`, +`SKILL.md`, `docs/commands.md`, `docs/profiles.md`, `docs/architecture.md` tree). New guard: +`test_validate_profile.py::TestOnDiskProfiles` validates every `profiles/*/profile.yaml` against +the real schema and pins `profile_id == directory name` (the compliance-gate lookup contract) — +future profiles get CI coverage the moment the directory exists. Suite: 818 passed. + +A copy of `microsoft-enterprise` with `platform: azure-devops` (plus profile_id and CLAUDE-md +template wording). Pure addition, auto-discovered by `/sdlc-setup`, zero code touched. Ships +first because (1) it unblocks ADO projects today (with the known Fold-A warts), and (2) Fold A's +ADO golden-tree test installs *from* it. Verified: `validate_profile.py` PASS; a scratch install +composes the ADO pack (`.azuredevops/pipelines/` ×10, `configure-branch-policies.sh`, +`run-claude-review.sh`, `post-pr-thread.sh`, ADO MCP fragment merged) — and reproduces both +Fold-A warts exactly as predicted (10 GitHub workflows + ruleset + CODEOWNERS installed anyway; +rubrics at `.github/profile/rubrics/` while `.azuredevops/rails/rubrics/` is missing). + +## Folds A–D — DONE on this branch (consolidated; see Sequencing) + +- **Fold A — DONE** (landed `7a9b999`): platform-aware core install — `stack.ci_cd.platform` gates the GitHub payload and rubrics land in the pack's expected dir; verified by the new ADO golden install tree, with the GitHub golden tree byte-identical. +- **Fold B — DONE** (landed `9c2d4ed`): pack-aware doctor — `az` login / variable-group / branch-policy checks on ADO installs; GitHub doctor output byte-identical (the `gh` pins in `test_doctor.py` unmodified). +- **Fold C — DONE** (landed `84f4702`): PR-flow rails on ADO — `az repos pr create` added to both review-gate trigger regexes, pack-level settings fragment (az allowlist + `.azuredevops/**` edit gate), pr-writer's open step parameterized. + - **Live drill — PASSED** (real ADO repo `RheemDevOps/Expedition_One/test-repo`, 2026-08-09): harness installed from this branch with the `ado-enterprise` profile; installed hook carried the az trigger and the merged az settings fragment. With a `src/` change committed and **no** review receipts, the installed `review-gate.sh` returned `deny` for both `az repos pr create` and `git push`; after writing `code-review`+`simplify` receipts for HEAD, both flipped to allow. End-to-end confirmed: pushed the branch and `az repos pr create` opened a real PR (#36931), then abandoned it and deleted the branch — client project left clean. +- **Fold D — DONE (this commit)**: the prose sweep — commands, harness docs, phases, and profiles name both platforms; the platform-switch runbook documented in `commands/sdlc-upgrade.md`. + +## Sequencing — as planned vs. as landed + +- **Fold 0 first** (done): the profile is the fixture everything else tests against. +- **Planned:** one spec = one branch = one PR per fold; B, C, D, E independent of A and each other, parallelizable across worktrees, review bandwidth the only serialization. +- **Landed:** Folds 0/A/B/C/D consolidated into **PR #47** — one branch, one commit per fold (`383b606` → this commit), reviewed as a unit. Review bandwidth *was* the serialization, and one consolidated review cost less than five; the additive guarantee was still checked commit-by-commit (suite green, GitHub goldens byte-identical at every fold), so the per-fold audit trail survives inside the single PR. +- **A was still the gate** and landed first after Fold 0: until it, every ADO install was polluted with GitHub artifacts and had broken rubric paths. +- **E stays separate, spike-first** (live drill: one rail pipeline authenticating to Foundry via workload identity federation on a real ADO project + Foundry resource) — its own branch and PR, spec'd only after the drill passes. +- Acceptance for any fold: existing test suite green + new ADO golden tree + one real `--profile ` install into a scratch repo, eyeball the tree. Held for A–D. + +## Fold E spike — PASSED with one administrative prerequisite (2026-08-09) + +Ran against a live sandbox Foundry deployment (`arjunm-claude-anthropic`, eastus2, deployments +named identically to model ids: claude-sonnet-5 / claude-opus-4-8 / claude-haiku-4-5 / claude-fable-5), +using the exact rail invocation (`npx @anthropic-ai/claude-code -p ... --output-format text`): + +- **End-to-end completion through Foundry: PROVEN** (`FOUNDRY-SPIKE-OK`) with + `CLAUDE_CODE_USE_FOUNDRY=1` + `ANTHROPIC_FOUNDRY_RESOURCE` + `ANTHROPIC_DEFAULT_*_MODEL` pins. +- **Entra credential chain: PROVEN to authenticate** — with no key set, the CLI walked the Azure + chain off the `az login` context and got `401 Principal does not have access to API/Operation`, + an authorization (not authentication) failure: the token minted and was presented. +- **The keyless gap is exactly one RBAC grant**: the principal needs a data-plane role on the + resource. **Tenant nuance: the role "Azure AI User" does not exist in this tenant — the real + role is `Cognitive Services User`.** Sandbox principal was Contributor-not-Owner, so could not + self-grant (`roleAssignments/write` denied). In the pipeline scenario the grant goes to the + service connection's identity at provisioning time — document it as a provisioning step. +- **Local-testing gotcha for the docs**: a Claude Code *desktop* session exports + `CLAUDE_CODE_PROVIDER_MANAGED_BY_HOST` (+ `ANTHROPIC_FOUNDRY_*`), and any child CLI inherits it + and defers to the app ("Foundry credentials are managed by the desktop app"). Spike runs must + scrub the env (`env -i`); CI agents are clean by nature and unaffected. +- Pipeline WIF leg deferred to implementation: the deterministic pattern is `AzureCLI@2` → + `az account get-access-token --resource https://cognitiveservices.azure.com` → + `ANTHROPIC_FOUNDRY_AUTH_TOKEN` (pre-issued token accepted per CLI docs, v2.1.203+). + +**Verdict: GO** — Fold E's implementation proceeds on proven facts; the RBAC grant and role-name +nuance land in the provisioning docs. + +## Fold E references + +- Claude Code on Microsoft Foundry: https://code.claude.com/docs/en/microsoft-foundry.md (env vars, Entra ID default credential chain, RBAC roles) +- Feature availability by provider: https://code.claude.com/docs/en/feature-availability.md (nothing the rails use is Foundry-gated; web search unavailable on Azure-hosted deployments) +- Foundry requires explicit model deployments + version pinning — no auto-resolution of `sonnet`/`opus` aliases without `ANTHROPIC_DEFAULT_*_MODEL`. + +## Out of scope + +`scripts/specs_to_ado_csv.py` (work-item export) is already ADO-native but has client-specific +area-path heuristics (EPR/CCaaS/Rheem) hard-coded in `classify()` — generalize to config before +it ships in the plugin. Not a GitHub-dependency fix, tracked separately. diff --git a/harness/CLAUDE.md.template b/harness/CLAUDE.md.template index a0e519d..a291e15 100644 --- a/harness/CLAUDE.md.template +++ b/harness/CLAUDE.md.template @@ -15,7 +15,7 @@ {{ONE_PARAGRAPH: what the system does, who uses it, the business outcome it serves — in the client's words.}} -**Stack:** {{e.g. .NET 8 / Angular / SQL Server / Azure / GitHub Actions}} +**Stack:** {{e.g. .NET 8 / Angular / SQL Server / Azure / GitHub Actions or Azure Pipelines}} **Build / test / verify:** `{{BUILD_CMD}}` · `{{TEST_CMD}}` · `{{LINT_CMD}}` **Done means the hook lets you stop.** A change is finished when it is proven against its spec by something other than its author — not when the code exists. The blocking Stop hook @@ -74,7 +74,8 @@ HIGH = all five rungs, including security-reviewer and a named human sign-off. Any change touching these is held to extra review and may trip the security gate: - `{{AUTH_PATHS}}` — authentication / identity / authorization - `{{DATA_PATHS}}` — schema migrations, data access -- `{{PIPELINE_PATHS}}` — `.github/`, the hooks, this CLAUDE.md, branch protection +- `{{PIPELINE_PATHS}}` — `.github/` or `.azuredevops/`, the hooks, this CLAUDE.md, branch + protection / branch policies - `{{INFRA_PATHS}}` — `infra/` (IaC is HIGH risk every time) - `{{PROMPT_PATHS}}` — prompts, model selection, tool definitions (see "Agentic work" below) @@ -117,8 +118,9 @@ The route is a **spike**: bounded, throwaway work whose deliverable is a written inflate all of them. ## Data boundary & secrets (non-negotiable) -- Secrets live in {{CLIENT}}'s Key Vault and GitHub secrets. **Never in code, never in this file, - never in specs.** A key in a commit is the one unrecoverable mistake. +- Secrets live in {{CLIENT}}'s Key Vault and GitHub secrets / Azure DevOps variable groups + (Key-Vault-backed). **Never in code, never in this file, never in specs.** A key in a commit + is the one unrecoverable mistake. - The {{LLM_PROVIDER}} key is {{CLIENT}}-procured under their own agreement. - Tag genuinely sensitive material out of agent context. If the engagement handles regulated or client-sensitive data, write `docs/data-flow-brief.md` during setup — a one-pager covering what diff --git a/harness/HARNESS.md b/harness/HARNESS.md index 85c28aa..14d205e 100644 --- a/harness/HARNESS.md +++ b/harness/HARNESS.md @@ -14,7 +14,7 @@ AI is doing the work. This page explains each piece in plain terms. | `.claude/agents/` | Six AI specialists it can call — two arrive automatically | | `.claude/skills/` | The team's runbooks for recurring jobs (specs, tests, bugs, PRs) | | `.mcp.json` | Shared plug-ins that give the AI extra abilities — approve once, done | -| `.github/workflows/` | The pull-request checks that apply to *everyone*, human or AI | +| `.github/workflows/` (GitHub) / `.azuredevops/pipelines/` (Azure DevOps) | The pull-request checks that apply to *everyone*, human or AI | | `specs/`, `.sdlc/`, `infra/` | Feature specs, project phase tracking, infrastructure starter | ## Two ideas explain the whole design @@ -134,10 +134,12 @@ Personal plug-ins stay in your own machine's config — this file is only for wh team should share. Tools that install themselves (like GitNexus) come with printed instructions instead: `.claude/tools//SETUP.md`. -## The pull-request checks — .github/workflows/ +## The pull-request checks — .github/workflows/ or .azuredevops/pipelines/ -These run at the server and apply to every change, no matter who or what wrote it. The -principle: **work can be proposed by anyone; only a gate lets it through.** +On GitHub these are Actions workflows in `.github/workflows/`; on Azure DevOps the same +checks are Azure Pipelines in `.azuredevops/pipelines/`. Either way they run at the server +and apply to every change, no matter who or what wrote it. The principle: **work can be +proposed by anyone; only a gate lets it through.** | Check | Power | Why | |---|---|---| @@ -149,7 +151,8 @@ principle: **work can be proposed by anyone; only a gate lets it through.** | Deploy to dev | ships | after a green merge; rolls itself back on failure | | Eval checks | blocks / advises | for AI-powered features: blocks if quality measurably drops | -On top sits **branch protection**: the blocking checks are mandatory, and a person who didn't +On top sits **branch protection** (on Azure DevOps: **branch policies** — same idea, the +platform's own mechanism): the blocking checks are mandatory, and a person who didn't write the change must approve it. Everything above in one sentence: *machines verify the facts; a human makes the call.* @@ -161,8 +164,12 @@ never seen catch anything is decoration. - **`specs/`** — one spec per feature. The starting point of all build work. - **`.sdlc/`** — which project phase we're in, plus the record of decisions and artifacts. -- **`.github/profile/rubrics/`** — the written standards the AI reviewers judge against. -- **`scripts/rails/`** — helper scripts the checks use (branch protection, change scoping). +- **`.github/profile/rubrics/`** (GitHub) / **`.azuredevops/rails/rubrics/`** (Azure DevOps) — + the written standards the AI reviewers judge against. On Azure DevOps the override and + accepted-risk ledgers live beside them in `.azuredevops/rails/`. +- **`scripts/rails/`** — helper scripts the checks use (change scoping, plus the one-time + branch setup: `apply-branch-protection.sh` on GitHub, `configure-branch-policies.sh` on + Azure DevOps). - **`infra/`** — an infrastructure starting point. Adapt it; don't use it as-is. - **`.claude/harness-manifest.json`** — the install receipt: which harness version this repo has and a fingerprint of every file *as installed*. It's how an upgrade can tell "still diff --git a/harness/README.md b/harness/README.md index ef956bc..43f0b90 100644 --- a/harness/README.md +++ b/harness/README.md @@ -45,6 +45,15 @@ Each piece is tagged so you adopt deliberately: Where each kit file goes in the client repo. The workflows and hooks reference these exact paths, so install here unless you also repoint the references. +The `.github/...` destinations are the GitHub-Actions layout. On an `azure-devops` profile the +platform-aware installer skips the pure-GitHub payload (`workflows/`, `profile/rulesets/`, +`profile/CODEOWNERS`, `apply-branch-protection.sh`) and overlays the azure-devops pack instead: +the pipelines land in `.azuredevops/pipelines/`, and the neutral governance content — the rubrics +and the two ledgers — is redirected to `.azuredevops/rails/`, where those pipelines read it. +**Branch policies** (`.azuredevops/rails/branch-policies.json`, applied by +`scripts/rails/configure-branch-policies.sh`) replace the ruleset and CODEOWNERS. `.github/RAILS.md` +keeps its path on both platforms — the ADO pack overlays its own realization there on purpose. + | Kit path | Install to | Notes | |---|---|---| | `CLAUDE.md.template` | `./CLAUDE.md` | Replace every `{{TOKEN}}`; delete guidance comments. | @@ -60,18 +69,18 @@ so install here unless you also repoint the references. | `hooks/*` | `./.claude/hooks/` | `stop-gate`, `review-gate`, `save-review-receipt` (`.ps1` + `.sh`), plus `sensitive-edit-nudge` — an advisory (non-blocking) example, installed but not registered. | | `agents/*` | `./.claude/agents/` | `planner`, `architect`, `grader`, `security-reviewer`, `build-error-resolver`, `debugger` — model-tiered; see `agents/README.md`. | | `skills/*` | `./.claude/skills/` | `spec-writer`, `test-writer`, `api-pattern`, `pr-writer`, `eval-builder`, `diagnose`. | -| `workflows/{ci,grader,correctness,security,deploy-dev,eval-regression,eval-suite}.yml` | `./.github/workflows/` | The five rails + the two eval workflows. | -| `workflows/deploy-promote.yml` | `./.github/workflows/` | The second half of the deploy rail: dev→test→prod. Manual trigger only — the target Environment's **required reviewers** are the human go/no-go, and the workflow refuses to run against an environment that has none. | -| `workflows/RAILS.md` | `./.github/RAILS.md` | Operator's guide + shakedown drills. | -| `profile/rubrics/*` | `./.github/profile/rubrics/` | Workflows read these by this path. | -| `workflows/dependency-scan.yml` | `./.github/workflows/` | Weekly scan of the **standing stock** of third-party advisories. Raises (and self-closes) one issue; never blocks — the blocking, diff-scoped half is `ci.yml`'s `dependency-gate` job. On by default: a security scan you have to remember to switch on is not running. | -| `workflows/rails-telemetry.yml` | `./.github/workflows/` | Weekly gate-outcome report, committed as `.github/rails-telemetry.json`. Records what ran, every override by name, and **which checks branch protection actually requires vs which gate jobs exist** — the comparison that catches a gate someone disarmed. No external calls: it reads this repo's own history and writes into this repo. | -| `profile/rails-telemetry.schema.json` | `./.github/rails-telemetry.schema.json` | The report's shape, fixed at version 1 before the install wave so it is not retrofitted across live repos. The fleet collector refuses a version it does not know rather than misreading it. | -| `profile/eval-bypasses.md` | `./.github/eval-bypasses.md` | Override/bypass ledger. | -| `profile/dependency-exceptions.md` | `./.github/dependency-exceptions.md` | Accepted-risk ledger for `dependency-gate`. The `accepted-risk:dependency` label clears one PR; this records why, who decided, whether the vulnerable path is reachable, and when the acceptance expires. Swept at Setup review. | -| `profile/CODEOWNERS` | `./.github/CODEOWNERS` | | +| `workflows/{ci,grader,correctness,security,deploy-dev,eval-regression,eval-suite}.yml` | `./.github/workflows/` (ADO: `./.azuredevops/pipelines/`) | The five rails + the two eval workflows. | +| `workflows/deploy-promote.yml` | `./.github/workflows/` (ADO: `./.azuredevops/pipelines/`) | The second half of the deploy rail: dev→test→prod. Manual trigger only — the target Environment's **required reviewers** are the human go/no-go, and the workflow refuses to run against an environment that has none. | +| `workflows/RAILS.md` | `./.github/RAILS.md` | Operator's guide + shakedown drills. Same path on ADO installs — the ADO pack overlays its own realization here on purpose. | +| `profile/rubrics/*` | `./.github/profile/rubrics/` (ADO: `./.azuredevops/rails/rubrics/`) | Workflows/pipelines read these by this path. | +| `workflows/dependency-scan.yml` | `./.github/workflows/` (ADO: `./.azuredevops/pipelines/`) | Weekly scan of the **standing stock** of third-party advisories. Raises (and self-closes) one issue; never blocks — the blocking, diff-scoped half is `ci.yml`'s `dependency-gate` job. On by default: a security scan you have to remember to switch on is not running. | +| `workflows/rails-telemetry.yml` | `./.github/workflows/` (ADO: `./.azuredevops/pipelines/`) | Weekly gate-outcome report, committed as `.github/rails-telemetry.json`. Records what ran, every override by name, and **which checks branch protection (ADO: branch policies) actually requires vs which gate jobs exist** — the comparison that catches a gate someone disarmed. No external calls: it reads this repo's own history and writes into this repo. | +| `profile/rails-telemetry.schema.json` | `./.github/rails-telemetry.schema.json` | The report's shape, fixed at version 1 before the install wave so it is not retrofitted across live repos. The fleet collector refuses a version it does not know rather than misreading it. Deliberately `.github/` on **both** platforms: both packs' telemetry pipelines commit the report to `.github/rails-telemetry.json` so one collector reads a mixed GitHub/Azure fleet. | +| `profile/eval-bypasses.md` | `./.github/eval-bypasses.md` (ADO: `./.azuredevops/rails/eval-bypasses.md`) | Override/bypass ledger. | +| `profile/dependency-exceptions.md` | `./.github/dependency-exceptions.md` (ADO: `./.azuredevops/rails/dependency-exceptions.md`) | Accepted-risk ledger for `dependency-gate`. The `accepted-risk:dependency` label clears one PR; this records why, who decided, whether the vulnerable path is reachable, and when the acceptance expires. Swept at Setup review. | +| `profile/CODEOWNERS` | `./.github/CODEOWNERS` | ADO: skipped — the required-reviewer branch policy in `branch-policies.json` is the twin. | | `profile/scripts/*` | `./scripts/rails/` | Workflows call `scripts/rails/diff-anchors.sh`. | -| `profile/rulesets/branch-protection.json` | `./.github/rulesets/` | Copied on install; `scripts/rails/apply-branch-protection.sh` reads it from there and applies it to GitHub. | +| `profile/rulesets/branch-protection.json` | `./.github/rulesets/` | Copied on install; `scripts/rails/apply-branch-protection.sh` reads it from there and applies it to GitHub. ADO: skipped — `.azuredevops/rails/branch-policies.json`, applied by `scripts/rails/configure-branch-policies.sh`, is the twin. | | `eval-datasets/*` | `./eval-datasets/` | Golden-set template + how-to (§11 work only). | | `prompts/*` | `./prompts/` | Versioned judge prompts (§11 work only). | | *(generated)* | `./.claude/harness-manifest.json` | Not a kit file — the installer writes it: plugin version + sha256 of every file as installed. `/sdlc-upgrade` reads it to tell factory-original files (safe to update) from repo-adapted ones (left alone). Commit it. | @@ -103,7 +112,7 @@ have run) **and `/update-docs`** — declared there, not duplicated here. Plus `eval-regression.yml` (per-PR gate when prompts/models/tools/agent-behavior change) and `eval-suite.yml` (periodic full benchmark) for §11 agentic deliverables. -The merge bar (in `branch-protection.json`): **CI green + spec-gate green + grader ran + +The merge bar (in `branch-protection.json`; ADO: `branch-policies.json`): **CI green + spec-gate green + grader ran + correctness passed (or recorded override) + a non-author approval**; HIGH adds security pass + a named sign-off in the PR. @@ -119,6 +128,8 @@ named sign-off in the PR. 4. **Workflows + profile** — set the `<>` build/test/deploy commands and gated-path regex. See `workflows/README.md`. 5. **Branch protection** — run `scripts/rails/apply-branch-protection.sh` once GitHub Actions is on. + On Azure DevOps: **branch policies** — run `scripts/rails/configure-branch-policies.sh` once the + pipelines exist. 6. **§11 only** — wire the eval runner behind the `eval-*` workflows; calibrate thresholds. ## Then prove the rails — do not assume them diff --git a/harness/hooks/review-gate.ps1 b/harness/hooks/review-gate.ps1 index c3d9787..175f182 100644 --- a/harness/hooks/review-gate.ps1 +++ b/harness/hooks/review-gate.ps1 @@ -5,7 +5,8 @@ # which settings.json permission patterns cannot express. # # Fires on the Claude Code `PreToolUse` event for Bash. Before the agent is allowed to -# `git push` or `gh pr create` a change that touches compilable source, this proves that +# `git push`, `gh pr create`, or `az repos pr create` a change that touches compilable +# source, this proves that # `/code-review` AND `/simplify` were actually run against the EXACT commit being pushed — # not the agent's recollection that it did them. A missing receipt blocks the push and # tells the agent to run the reviews. @@ -69,10 +70,10 @@ $payload = $null if ($raw) { try { $payload = $raw | ConvertFrom-Json } catch { } } if (-not $payload) { Allow } -# --- Only gate Bash commands that actually INVOKE git push / gh pr create. Match the -# start of each shell segment (split on ; && || | &) so the words appearing inside a -# quoted string, an echo, or `--help` text don't trip the gate. `git -C push` -# is recognized. +# --- Only gate Bash commands that actually INVOKE git push / gh pr create / +# az repos pr create. Match the start of each shell segment (split on ; && || | &) so +# the words appearing inside a quoted string, an echo, or `--help` text don't trip the +# gate. `git -C push` is recognized. if ($payload.tool_name -ne 'Bash') { Allow } $cmd = [string]$payload.tool_input.command if (-not $cmd) { Allow } @@ -87,9 +88,15 @@ if ($cmd -match $pipeToShellRe) { } $gates = $false -foreach ($seg in ($cmd -split '\|\||&&|[;|&]')) { +# Split on newlines too — a multi-line Bash command with the trigger on a later line must not +# walk past this twin while the .sh form (which splits per input line via awk) denies it. +foreach ($seg in ($cmd -split '\|\||&&|[;|&]|\r?\n')) { $s = $seg.Trim() - if ($s -match '^git\s+(-C\s+\S+\s+)?push(\s|$)' -or $s -match '^gh\s+pr\s+create(\s|$)') { + # Strip leading VAR=value assignments (see the .sh twin): an env-prefixed invocation like + # `AZURE_DEVOPS_EXT_PAT=... az repos pr create` is still an invocation. + while ($s -match '^[A-Za-z_][A-Za-z_0-9]*=\S*\s+') { $s = $s.Substring($Matches[0].Length) } + if ($s -match '^git\s+(-C\s+\S+\s+)?push(\s|$)' -or $s -match '^gh\s+pr\s+create(\s|$)' -or + $s -match '^az\s+repos\s+pr\s+create(\s|$)') { $gates = $true; break } } diff --git a/harness/hooks/review-gate.sh b/harness/hooks/review-gate.sh index 002fae2..306b26a 100755 --- a/harness/hooks/review-gate.sh +++ b/harness/hooks/review-gate.sh @@ -49,17 +49,25 @@ if [[ "$cmd" =~ $pipe_to_shell_re ]]; then deny "Blocked: piping a download straight into a shell (curl|sh and friends) is denied by policy. Download to a file, inspect it, then execute it as a separate, reviewable step." fi -# --- Only gate commands that actually INVOKE git push / gh pr create. Inspect the start -# of each shell segment (split on ; && || | &) so the words inside a quoted string or -# an echo don't trip the gate. `git -C push` is recognized. +# --- Only gate commands that actually INVOKE git push / gh pr create / az repos pr create. +# Inspect the start of each shell segment (split on ; && || | &) so the words inside a +# quoted string or an echo don't trip the gate. `git -C push` is recognized. # awk, not sed: BSD sed renders '\n' in a replacement as a literal 'n', which would # leave compound commands unsplit and let `cd x && git push` walk past the gate. gates=false segment_stream="$(printf '%s' "$cmd" | awk '{gsub(/\|\||&&|[;|&]/, "\n"); print}')" while IFS= read -r seg; do s="$(printf '%s' "$seg" | sed -E 's/^[[:space:]]+//')" + # Strip leading VAR=value assignments: `AZURE_DEVOPS_EXT_PAT=... az repos pr create` is a + # documented az auth pattern and still an invocation — the shell treats the prefix as + # environment, not the command. (A value with quoted whitespace defeats this; rare, and + # under-gating there matches the parser's documented coverage boundary.) + while [[ "$s" =~ ^[A-Za-z_][A-Za-z_0-9]*=[^[:space:]]*[[:space:]]+ ]]; do + s="${s:${#BASH_REMATCH[0]}}" + done if [[ "$s" =~ ^git[[:space:]]+(-C[[:space:]]+[^[:space:]]+[[:space:]]+)?push([[:space:]]|$) ]] \ - || [[ "$s" =~ ^gh[[:space:]]+pr[[:space:]]+create([[:space:]]|$) ]]; then + || [[ "$s" =~ ^gh[[:space:]]+pr[[:space:]]+create([[:space:]]|$) ]] \ + || [[ "$s" =~ ^az[[:space:]]+repos[[:space:]]+pr[[:space:]]+create([[:space:]]|$) ]]; then gates=true break fi diff --git a/harness/packs/cicd/azure-devops/pack.yaml b/harness/packs/cicd/azure-devops/pack.yaml index 9fe5388..65c9571 100644 --- a/harness/packs/cicd/azure-devops/pack.yaml +++ b/harness/packs/cicd/azure-devops/pack.yaml @@ -35,6 +35,8 @@ provides: - scripts/run-claude-review.sh # headless Claude Code CLI invocation (no action exists on ADO) - scripts/post-pr-thread.sh # POST/PATCH a PR thread via the ADO REST API branch_policies: branch-policies/policies.json # desired-state consumed by configure-branch-policies.sh + settings_fragment: settings.fragment.json # az tooling perms + .azuredevops/ gated paths, + # merged into .claude/settings.json rails_guide: RAILS.md # operator's guide + shakedown drills (ADO realization) mcp_fragment: mcp.fragment.json # Azure DevOps MCP server, merged into .mcp.json # (ADO_ORGANIZATION_NOT_SET is the Phase-3 placeholder — @@ -116,6 +118,7 @@ overlays: # Overlays the core's GitHub-flavored guide (last wins) so the documented path # (.github/RAILS.md, referenced by /sdlc-setup Step 8) holds the ADO realization. - { src: RAILS.md, dest: .github/RAILS.md } + - { src: settings.fragment.json, dest: .claude/settings.json, merge: true } - { src: mcp.fragment.json, dest: .mcp.json, merge: true } # Installed by the core/profile layer, referenced here (NOT shipped by this pack — see `consumes`): # scripts/rails/diff-anchors.sh <- kit/profile/scripts/diff-anchors.sh diff --git a/harness/packs/cicd/azure-devops/settings.fragment.json b/harness/packs/cicd/azure-devops/settings.fragment.json new file mode 100644 index 0000000..ebc5a98 --- /dev/null +++ b/harness/packs/cicd/azure-devops/settings.fragment.json @@ -0,0 +1,19 @@ +{ + "//": "Merged (not overwritten) into .claude/settings.json by the installer (pack.yaml: merge: true). Adds the Azure DevOps twins of the core file's GitHub entries: read-only az repos/pipelines queries on the allow-list (az repos pr show/list are the ADO analogue of gh pr view/list — az has no 'pr view'; runs list/show mirror gh run list/view), mutating az calls and the branch-policy script behind ask (the analogue of gh api and apply-branch-protection.sh), and the pipeline definitions as a gated path with the same tamper protection ./.github/** has. 'az repos policy' is grouped whole under ask even though 'policy list' is a read: ask rules take precedence over allow in Claude Code, so a list carve-out would be shadowed anyway — and doctor.py calls az via subprocess, invisible to these rules, so it stays headless either way. The core's gh entries survive the merge — harmless on an ADO repo, where nothing invokes gh.", + "permissions": { + "allow": [ + "Bash(az repos pr show:*)", + "Bash(az repos pr list:*)", + "Bash(az pipelines runs list:*)", + "Bash(az pipelines runs show:*)", + "Bash(az pipelines build list:*)" + ], + "ask": [ + "Bash(az repos policy:*)", + "Bash(az devops invoke:*)", + "Bash(scripts/rails/configure-branch-policies.sh:*)", + "Edit(./.azuredevops/**)", + "Write(./.azuredevops/**)" + ] + } +} diff --git a/harness/skills/pr-writer/SKILL.md b/harness/skills/pr-writer/SKILL.md index 82d458e..ee50855 100644 --- a/harness/skills/pr-writer/SKILL.md +++ b/harness/skills/pr-writer/SKILL.md @@ -30,10 +30,11 @@ produces the PR the merge bar and the grader expect. - **Acceptance checks → evidence** — list each check from the spec and the test/line that satisfies it. - **Test plan** — what was run and the result (`dotnet test` etc.), not "should pass." - - **Risk tier + gated paths touched** — so the right gates and reviewers are triggered; add the - `risk:high` label if HIGH. -5. **Open the PR** (`gh pr create`) with that title and body. Do not approve or merge it — a - non-author owns the verdict. + - **Risk tier + gated paths touched** — so the right gates and reviewers are triggered; if HIGH, + add the `risk:high` label (GitHub) or PR label via `--labels risk:high` (Azure DevOps). +5. **Open the PR** with that title and body — `gh pr create` on GitHub, `az repos pr create` on + Azure DevOps (both are gated by the review-gate hook until the receipts exist). Do not approve + or merge it — a non-author owns the verdict. ## Done when - Branch, conventional-commit title, and co-authorship are correct. diff --git a/phases/03-foundation.md b/phases/03-foundation.md index 2a9c207..9e226f3 100644 --- a/phases/03-foundation.md +++ b/phases/03-foundation.md @@ -55,7 +55,7 @@ The five workflows: - **security** (blocks on HIGH): runs the security-reviewer agent. Fires on the `risk:high` label **or** any PR touching a registered gated path (auth, migrations, the pipeline, infra), independent of the spec's tier. - **deploy-dev** (ships): merge to main deploys the merged artifact to the client's dev environment, and restores the last good version when a deploy fails. -**Branch protection** turns the workflows from suggestions into rails. Every PR, to merge, must clear: CI green, the grader has run, correctness passed (or a named-human override is recorded), and a non-author approval. A `risk:high` change adds the security workflow pass and a named human sign-off recorded in the PR. The agent can push only to branches it creates, can never approve or merge its own work, and every agent commit is co-authored so provenance is in the history. +**Branch protection** (branch policies on Azure DevOps) turns the workflows from suggestions into rails. Every PR, to merge, must clear: CI green, the grader has run, correctness passed (or a named-human override is recorded), and a non-author approval. A `risk:high` change adds the security workflow pass and a named human sign-off recorded in the PR. The agent can push only to branches it creates, can never approve or merge its own work, and every agent commit is co-authored so provenance is in the history. **The Stop hook** is the highest-value automation in the standard: a script that fires when an agent tries to finish its turn and refuses to let it stop on a failing build or red tests. It turns "the tests must pass" from a request the agent might rationalize past into a fact about the world. Wire it in `.claude/` settings. @@ -83,7 +83,7 @@ The Phase 2 security gates MUST appear on the map with their guarded paths. Draft the IaC for the dev environment (Bicep on the .NET/Azure default; the pattern is stack-independent). HIGH risk — human-reviewed on every change; the environment provisions from code, not from clicks. Run it through the agent-safe IaC funnel: schema-validate → policy-as-code gate → dry-run (`bicep what-if` or equivalent) → human approval → scoped least-privilege apply. -Secrets land in the client's vault (Key Vault and GitHub secrets) — never in code, never in `CLAUDE.md`, never in a spec. A secret in the repo is the one unrecoverable mistake of the phase: a rotation event and an audit-log review, not an edit. Draft the data-flow brief for client security: what goes to the API, what doesn't, where keys live, who can see usage. +Secrets land in the client's vault (Key Vault and GitHub secrets on GitHub; Key-Vault-backed variable groups on Azure DevOps) — never in code, never in `CLAUDE.md`, never in a spec. A secret in the repo is the one unrecoverable mistake of the phase: a rotation event and an audit-log review, not an edit. Draft the data-flow brief for client security: what goes to the API, what doesn't, where keys live, who can see usage. **For `library`/`cli`/`skill`:** there is no environment to provision. Record `N/A — {project_type}` for the IaC artifacts and proceed; the rails still apply. @@ -209,7 +209,7 @@ Must contain: - [ ] `foundation-report.md`, `risk-tier-map.md`, `cadence-plan.md`, and `build-handoff.md` exist and are complete - [ ] The harness is installed, adapted to the client, committed, and reviewed by the Setup Owner's deputy (never sole-approved by its author) - [ ] The pipeline runs: CI hard gates, the grader required to run, correctness blocking on a high-confidence defect, security on `risk:high`, deploy-dev on merge — all reviewed by the client's DevOps -- [ ] Branch protection enforces CI green + grader-ran + correctness-passed + non-author approval; `risk:high` adds the security workflow and a named sign-off +- [ ] Branch protection (branch policies on ADO) enforces CI green + grader-ran + correctness-passed + non-author approval; `risk:high` adds the security workflow and a named sign-off - [ ] The dev environment is provisioned from code; secrets live in the client's vault, never in code (N/A for `library`/`cli`/`skill`) - [ ] The build-time security gates from Phase 2 are wired, and each has fired on a real PR - [ ] The walking skeleton is deployed to the dev environment through the real pipeline and verified against the Phase 2 definition diff --git a/phases/08-deployment.md b/phases/08-deployment.md index fcdbc34..fc6c6ca 100644 --- a/phases/08-deployment.md +++ b/phases/08-deployment.md @@ -13,7 +13,7 @@ Deploy the system to production safely, with documented rollback capability, ver | `library` / `cli` | Package registry publish (npm, PyPI, NuGet). Staging = local install test in a fresh environment. | Install the package in an isolated environment and run the public API smoke tests | Yank the package version from the registry; pin consumers to previous version | | `skill` | File distribution (copy to `.claude/commands/`). No server, no process. Staging = fresh install on a clean project. | Install the skill files in a new project; run the minimum smoke test set | Delete the skill files; re-copy the previous version | -**For `skill` / `library` projects:** Skip steps that reference staging servers, database migrations, health checks, and monitoring dashboards — they do not apply. Focus on: (1) install verification, (2) smoke test execution, (3) rollback documentation (file deletion / version revert), and (4) release artifact creation (GitHub release / package publish). +**For `skill` / `library` projects:** Skip steps that reference staging servers, database migrations, health checks, and monitoring dashboards — they do not apply. Focus on: (1) install verification, (2) smoke test execution, (3) rollback documentation (file deletion / version revert), and (4) release artifact creation (GitHub release on GitHub; Azure Artifacts or a pipeline artifact on Azure DevOps / package publish). ## Entry Criteria - Phase 7 exit gate passed and `phase8-handoff.md` reviewed diff --git a/phases/09-monitoring.md b/phases/09-monitoring.md index 49a638e..ce4351e 100644 --- a/phases/09-monitoring.md +++ b/phases/09-monitoring.md @@ -10,8 +10,8 @@ Establish production observability so the team knows about problems before users | project_type | Monitoring Approach | |--------------|-------------------| | `service` / `app` | Full infrastructure monitoring: dashboards (RED metrics), alerting rules, on-call rotation, incident response runbook. | -| `library` / `cli` | Package health monitoring: download counts, open issues, version adoption. Alerts = GitHub issue triage criteria. No dashboards. | -| `skill` | Qualitative monitoring only. No server, no metrics pipeline. Monitoring = GitHub Issues + user feedback. Replace all dashboard / alerting / infrastructure content with: issue triage criteria, user feedback channels, and qualitative quality signals. | +| `library` / `cli` | Package health monitoring: download counts, open issues, version adoption. Alerts = GitHub issue (or ADO Boards work item) triage criteria. No dashboards. | +| `skill` | Qualitative monitoring only. No server, no metrics pipeline. Monitoring = GitHub Issues (or ADO Boards work items) + user feedback. Replace all dashboard / alerting / infrastructure content with: issue triage criteria, user feedback channels, and qualitative quality signals. | **For `skill` / `library` projects:** The `monitoring-config.md`, `alert-definitions.md`, and `incident-response.md` artifacts should be reframed as: (1) feedback collection channels and issue triage criteria, (2) severity classification with response SLAs, and (3) escalation process. Do not spend time configuring Grafana dashboards or Prometheus rules that will never be used. @@ -49,7 +49,7 @@ Set up dashboards and metrics collection: - Business metrics (what matters to stakeholders: active users, transactions, etc.) - Dependency health (database, external APIs, queues) -**For `skill` / `library` projects:** Skip infrastructure metrics. Configure: GitHub issue monitoring, download/install tracking, user feedback channels. +**For `skill` / `library` projects:** Skip infrastructure metrics. Configure: GitHub issue monitoring (ADO Boards work-item monitoring on Azure DevOps), download/install tracking, user feedback channels. ### Step 2: Alert Definitions Define alerts that require human response: @@ -95,8 +95,9 @@ Record the result in `drill-record.md`. ### Step 5: Project Retrospective -Start by gathering whatever user feedback the deployment produced — GitHub issues, support -requests, Slack messages, survey results — and look for patterns rather than incidents: what +Start by gathering whatever user feedback the deployment produced — GitHub issues or ADO Boards +work items, support requests, Slack messages, survey results — and look for patterns rather than +incidents: what confused people, what they liked, what broke. Where there is no feedback yet, record that as the finding; "we shipped and heard nothing" is itself worth knowing at Close. diff --git a/profiles/ado-enterprise/claude-md-template.md b/profiles/ado-enterprise/claude-md-template.md new file mode 100644 index 0000000..6f1ad94 --- /dev/null +++ b/profiles/ado-enterprise/claude-md-template.md @@ -0,0 +1,39 @@ +# SDLC Configuration (auto-injected by /sdlc-setup) + +## SDLC Plugin Active +This project uses the claude-code-sdlc plugin for lifecycle management. +- State: `.sdlc/state.yaml` +- Profile: `ado-enterprise` +- Commands: `/sdlc`, `/sdlc-setup`, `/sdlc-status`, `/sdlc-next`, `/sdlc-gate` + +## Stack: Microsoft Enterprise (Azure DevOps) +- **Backend:** C# / .NET 8 / Entity Framework Core / xUnit +- **Frontend:** TypeScript / Angular 17 / NgRx / Playwright +- **Database:** SQL Server with EF Core migrations +- **Cloud:** Azure (App Service, Azure SQL, Key Vault, App Insights) +- **Repos & CI/CD:** Azure Repos + Azure Pipelines (`.azuredevops/pipelines/`, branch policies via `az repos policy`) + +## Quality Standards +- Code coverage: minimum 80%, critical paths 100% +- Max file size: 800 lines, max function: 50 lines +- TDD required: write tests first +- Code review required before merge +- Security review required for auth/payment/sensitive code + +## Coding Conventions +- Commits: `type: description` (feat, fix, refactor, docs, test, chore) +- Branches: `type/ticket-description` +- Immutability: prefer immutable patterns (records, `with`, spread operators) +- No console.log in production code + +## Compliance: SOC 2 +- Audit trail maintained in `.sdlc/state.yaml` +- Change approval: peer review required +- Security review per change inside the Build loop (the security rail blocks at merge) +- Documentation current at the Phase 7 (Documentation) gate + +## Phase Awareness +Before making changes, check the current SDLC phase with `/sdlc`: +- Opening phases (0 Discovery – 3 Foundation): produce and gate the documents; no feature code before Foundation +- Build loop: one spec at a time — Intent → Delegate → Discern; TDD required, checking happens per change, the author never approves their own work +- Closing phases (7 Documentation – 9 Monitoring, then Close & Transfer): prove docs by cold use, deploy through the existing pipeline, alerts from measured baselines diff --git a/profiles/ado-enterprise/compliance/soc2-gates.yaml b/profiles/ado-enterprise/compliance/soc2-gates.yaml new file mode 100644 index 0000000..919232b --- /dev/null +++ b/profiles/ado-enterprise/compliance/soc2-gates.yaml @@ -0,0 +1,95 @@ +# SOC 2 Gate Definitions for Microsoft Enterprise Profile +# These gates are checked at phase transitions when compliance.frameworks includes "soc2" + +version: "1.0" +framework: soc2 + +gates: + - phase: 1 + id: soc2-cc6.1-requirements + name: "Access Control Requirements" + criteria: "CC6.1 — Logical and physical access controls" + check_type: artifact_content + artifact: "requirements.md" + required_content: + - "authentication" + - "authorization" + severity: MUST + + - phase: 2 + id: soc2-cc6.6-boundaries + name: "System Boundaries" + criteria: "CC6.6 — System boundaries and trust zones" + check_type: artifact_content + artifact: "design-doc.md" + required_content: + - "trust boundary" + - "security" + severity: MUST + + - phase: 2 + id: soc2-cc7.1-change-mgmt + name: "Change Management" + criteria: "CC7.1 — Change management through ADRs" + check_type: artifact_exists + artifact: "adrs/" + severity: MUST + + - phase: build + id: soc2-cc6.1-implementation + name: "Access Controls Implemented" + criteria: "CC6.1 — Access control code present and tested" + check_type: manual + description: "Verify auth/authz code exists with unit tests" + severity: MUST + + - phase: build + id: soc2-cc7.2-vulnerability + name: "Vulnerability Management" + criteria: "CC7.2 — Security review completed (per change, on the rails)" + check_type: manual + description: "Verify the security workflow ran and findings are resolved for risk:high changes" + severity: MUST + + - phase: build + id: soc2-cc8.1-code-review + name: "Code Review" + criteria: "CC8.1 — Non-author review documented (per change, on the rails)" + check_type: manual + description: "Verify every merged spec carried a non-author approval and the grader ran" + severity: MUST + + - phase: build + id: soc2-cc7.1-testing + name: "Testing Requirements" + criteria: "CC7.1 — Test coverage meets thresholds" + check_type: metric + metric: coverage + threshold_key: "quality.coverage_minimum" + severity: MUST + + - phase: 7 + id: soc2-cc2.1-documentation + name: "System Documentation" + criteria: "CC2.1 — Documentation is current" + check_type: manual + description: "Verify README and API docs reflect current system" + severity: MUST + + - phase: 8 + id: soc2-cc7.4-deployment + name: "Change Deployment" + criteria: "CC7.4 — Release notes with rollback plan" + check_type: artifact_content + artifact: "release-notes.md" + required_content: + - "rollback" + severity: MUST + + - phase: 9 + id: soc2-cc7.2-monitoring + name: "Monitoring & Detection" + criteria: "CC7.2 — Security monitoring configured" + check_type: manual + description: "Verify alerts configured for security events" + severity: MUST diff --git a/profiles/ado-enterprise/profile.yaml b/profiles/ado-enterprise/profile.yaml new file mode 100644 index 0000000..386f762 --- /dev/null +++ b/profiles/ado-enterprise/profile.yaml @@ -0,0 +1,113 @@ +version: "1.0" + +company: + name: "Example Corp" + profile_id: "ado-enterprise" + +stack: + backend: + language: csharp + framework: dotnet-8 + orm: ef-core + testing: xunit + frontend: + language: typescript + framework: angular-17 + state: ngrx + e2e: playwright + database: + engine: sql-server + migrations: ef-core + cloud: + provider: azure + services: + - app-service + - azure-sql + - key-vault + - app-insights + - entra-id + auth_pattern: DefaultAzureCredential + secrets: azure-key-vault + monitoring: app-insights + ci_cd: + # Selects the Azure DevOps CI/CD pack: Azure Pipelines rails in .azuredevops/pipelines/, + # branch policies as code (az repos policy), ADO MCP server fragment. The repo host is + # Azure Repos — the sibling `microsoft-enterprise` profile is this same stack on GitHub. + platform: azure-devops + +quality: + coverage_minimum: 80 + coverage_critical: 100 + max_file_lines: 800 + max_function_lines: 50 + require_tdd: true + require_code_review: true + require_security_review: true + visual_verification: + enabled: true + baseline_comparison: manual + session_health_check: + enabled: true + command: "dotnet build --no-restore --verbosity quiet" + timeout_seconds: 120 + min_phase: build + evaluation_criteria: + # Phase 1: Requirements quality + - name: "Requirement testability" + phases: [1] + description: "Every requirement must have a concrete acceptance criterion that can be verified by a test or demonstration. Vague criteria like 'the system should be fast' fail; 'P95 response time < 200ms' passes." + severity: warn + - name: "Requirement traceability" + phases: [1] + description: "Every requirement must have a unique ID (REQ-NNN) and trace to at least one stakeholder need from Phase 0. Orphan requirements with no stakeholder justification fail." + severity: warn + # Phase 2: Design quality + - name: "ADR completeness" + phases: [2] + description: "Each ADR must state the problem, options considered (minimum 2), decision rationale, and consequences. Missing any of these sections is a fail." + severity: fail + - name: "Interface contract specificity" + phases: [2] + description: "API contracts must specify request/response schemas, error codes, and auth requirements. Contracts that say 'returns data' without specifying the shape fail." + severity: warn + # Phase 3: Planning quality + - name: "Section plan verifiability" + phases: [3] + description: "Each section plan must include verification criteria that the section-evaluator can check mechanically. Vague criteria like 'works correctly' fail; 'all endpoints return 200 for valid input and 400 for invalid input' passes." + severity: warn + # Build loop: code quality (checked per change by the section-evaluator) + - name: "Result pattern" + phases: ["build"] + description: "Error-prone operations use Result instead of throwing exceptions. Evaluator checks for try/catch blocks that could use Result." + severity: warn + - name: "Immutable state" + phases: ["build"] + description: "NgRx reducers return new state objects. No Array.push(), Object.assign() mutations, or direct property assignment on state." + severity: fail + - name: "API documentation" + phases: ["build"] + description: "All public API endpoints have XML doc comments on the controller action and a corresponding entry in api-contracts.md." + severity: warn + - name: "FluentValidation" + phases: ["build"] + description: "All DTOs received from external input have a corresponding FluentValidation validator class." + severity: fail + +compliance: + frameworks: + - soc2 + audit_trail: true + change_approval: peer-review + +# documentation: +# intake_path: "docs/intake" +# types: [pdf, markdown, text] +# index_budget_tokens: 5000 +# summary_budget_tokens: 750 +# max_documents: 50 + +conventions: + commit_format: "type: description" + branch_naming: "type/ticket-description" + immutability: true + no_console_log: true diff --git a/profiles/ado-enterprise/references/azure-patterns.md b/profiles/ado-enterprise/references/azure-patterns.md new file mode 100644 index 0000000..72697fa --- /dev/null +++ b/profiles/ado-enterprise/references/azure-patterns.md @@ -0,0 +1,92 @@ +# Azure Enterprise Patterns + +Best practices for the ado-enterprise profile, aligned with [Microsoft Skills](https://github.com/microsoft/skills) conventions. + +## Authentication + +### DefaultAzureCredential (Required) +All Azure service connections MUST use `DefaultAzureCredential`. Never hardcode connection strings, keys, or tokens. + +```csharp +// C# — correct pattern +var credential = new DefaultAzureCredential(); +var client = new SecretClient(new Uri(vaultUri), credential); +``` + +```typescript +// TypeScript — correct pattern +import { DefaultAzureCredential } from "@azure/identity"; +const credential = new DefaultAzureCredential(); +``` + +### Managed Identity +- Production: Use system-assigned Managed Identity on Azure App Service +- Development: Use Azure CLI authentication (`az login`) +- CI/CD: Use service principal with federated credentials (OIDC) + +### Entra ID (Azure AD) +- Multi-tenant apps: validate tenant ID in token claims +- Single-tenant: set `ValidateIssuer = true` with specific issuer +- Always validate audience claim matches your app + +## Secrets Management + +### Azure Key Vault (Required) +- All secrets, connection strings, and certificates MUST be stored in Key Vault +- Use `SecretClient` with `DefaultAzureCredential` +- Never store secrets in appsettings.json, environment variables in production, or source control +- Development: Use .NET User Secrets (`dotnet user-secrets`) + +## Monitoring & Observability + +### Application Insights +- Enable auto-instrumentation on App Service +- Use `ILogger` for structured logging (maps to App Insights traces) +- Track custom metrics with `TelemetryClient` +- Use correlation IDs for distributed tracing + +### Health Checks +```csharp +// Required for Azure App Service deployment slots +builder.Services.AddHealthChecks() + .AddSqlServer(connectionString) + .AddAzureBlobStorage(connectionString); +``` + +## Data Access + +### Entity Framework Core +- Always use parameterized queries (EF Core does this by default) +- For raw SQL: `FromSqlInterpolated` only — never `FromSqlRaw` with string concatenation +- Use migrations for schema changes: `dotnet ef migrations add ` +- Enable retry logic for transient failures: + ```csharp + options.UseSqlServer(connectionString, sqlOptions => + sqlOptions.EnableRetryOnFailure(maxRetryCount: 3)); + ``` + +## Error Handling + +### Global Exception Handler +```csharp +app.UseExceptionHandler("/error"); +// Never expose stack traces or internal details in API responses +``` + +### Result Pattern +Use `Result` for business logic errors — reserve exceptions for truly exceptional cases. + +## Security Headers +These MUST be configured on all deployments: +- `X-Frame-Options: DENY` +- `X-Content-Type-Options: nosniff` +- `Content-Security-Policy: default-src 'self'` +- `Strict-Transport-Security: max-age=31536000; includeSubDomains` + +## Related Microsoft Skills +When building with this profile, consider loading these skills from [microsoft/skills](https://github.com/microsoft/skills): +- `azure-entra-enterprise-dotnet` — Entra ID authentication patterns +- `azure-app-service-dotnet` — App Service deployment and configuration +- `azure-sql-dotnet` — Azure SQL with EF Core patterns +- `azure-key-vault-dotnet` — Secret and certificate management +- `azure-app-insights-dotnet` — Application Insights integration diff --git a/profiles/ado-enterprise/switchboard-rules.json b/profiles/ado-enterprise/switchboard-rules.json new file mode 100644 index 0000000..fa1074a --- /dev/null +++ b/profiles/ado-enterprise/switchboard-rules.json @@ -0,0 +1,112 @@ +{ + "version": "1.0", + "profile_id": "ado-enterprise", + "description": "Skill switchboard rules for Microsoft Enterprise stack. Maps phase contexts to recommended skill invocations.", + "rules": [ + { + "phase": 0, + "context": "discovery", + "when": "User describes a new project or problem", + "suggest": "/plan", + "reason": "Structure the discovery findings into a problem statement" + }, + { + "phase": 1, + "context": "requirements", + "when": "Problem statement exists and is approved", + "suggest": "/deep-project", + "reason": "Decompose into well-scoped planning units" + }, + { + "phase": 2, + "context": "design-spec-synthesis", + "when": "Phase 2 starts and planning/spec.md does not exist", + "suggest": "uv run scripts/synthesize_spec.py", + "reason": "Combine Phase 0-1 artifacts into /deep-plan input" + }, + { + "phase": 2, + "context": "design-deep-plan", + "when": "planning/spec.md exists and planning/claude-plan.md does not", + "suggest": "/deep-plan @planning/spec.md", + "reason": "Run research, interview, plan generation, and external review (steps 1-15)" + }, + { + "phase": 2, + "context": "design-artifact-mapping", + "when": "planning/claude-plan.md exists and design-doc.md does not", + "suggest": "uv run scripts/map_deep_plan_artifacts.py --phase 2", + "reason": "Transform /deep-plan outputs into SDLC Phase 2 artifacts" + }, + { + "phase": 3, + "context": "planning-deep-plan-resume", + "when": "deep-plan-checkpoint.yaml exists and planning/sections/index.md does not", + "suggest": "/deep-plan (resume from checkpoint, steps 16-22)", + "reason": "Generate TDD plan and section files from approved plan" + }, + { + "phase": 3, + "context": "planning-artifact-mapping", + "when": "planning/sections/ has section files and section-plans/ is empty", + "suggest": "uv run scripts/map_deep_plan_artifacts.py --phase 3", + "reason": "Transform /deep-plan sections into SDLC section plan format" + }, + { + "phase": 4, + "context": "implementation", + "when": "Section plans exist", + "suggest": "/deep-implement", + "reason": "Execute section plans with TDD methodology" + }, + { + "phase": 4, + "context": "implementation-tdd", + "when": "Writing new code and quality.require_tdd is true", + "suggest": "/tdd", + "reason": "Profile requires test-driven development" + }, + { + "phase": 5, + "context": "quality-review", + "when": "Implementation complete, entering quality phase", + "suggest": "/code-review", + "reason": "Review code for quality and maintainability" + }, + { + "phase": 5, + "context": "quality-security", + "when": "Code touches auth, payments, or sensitive data", + "suggest": "/security-review", + "reason": "Profile requires security review; SOC 2 CC7.2 compliance" + }, + { + "phase": 6, + "context": "testing-e2e", + "when": "Quality review passed, testing critical user flows", + "suggest": "/e2e", + "reason": "Generate and run end-to-end tests with Playwright" + }, + { + "phase": 6, + "context": "testing-coverage", + "when": "Checking coverage against thresholds", + "suggest": "/test-coverage", + "reason": "Verify coverage meets 80% minimum / 100% critical" + }, + { + "phase": 7, + "context": "documentation", + "when": "Testing passed, syncing documentation", + "suggest": "/update-docs", + "reason": "Keep documentation current with code" + } + ], + "stack_hints": { + "backend_test_command": "dotnet test --collect:\"XPlat Code Coverage\"", + "frontend_test_command": "ng test --code-coverage --watch=false --browsers=ChromeHeadless", + "e2e_command": "npx playwright test", + "build_command": "dotnet build && ng build", + "lint_command": "dotnet format --verify-no-changes && ng lint" + } +} diff --git a/scripts/doctor.py b/scripts/doctor.py index d09e080..5d5462b 100644 --- a/scripts/doctor.py +++ b/scripts/doctor.py @@ -9,9 +9,15 @@ uv run scripts/doctor.py # in the client repo uv run scripts/doctor.py --repo - uv run scripts/doctor.py --offline # skip the checks that need gh + uv run scripts/doctor.py --offline # skip the checks that need gh/az Exit 1 if anything FAILs. WARN never fails the run — it marks what could not be determined. + +Platform-aware: the manifest's composed pack ids say which CI platform this install realized +(cicd/github vs cicd/azure-devops), and the platform-facing checks follow it — an Azure DevOps +install is checked with az (variable groups, branch policies), never told to install gh for a +platform it does not use. Anything that is not recognizably an ADO install gets the GitHub +checks, exactly as before the packs existed. """ from __future__ import annotations @@ -38,6 +44,7 @@ # token is not corruption — it is unfinished setup — so the report says who finishes it. TOKEN_OWNER = { "ADO_ORGANIZATION": "Phase 3 — your Azure DevOps organization name", + "VARIABLE_GROUP": "Phase 3 — the variable group exposing ANTHROPIC_API_KEY (Key-Vault-backed)", "GATED_PATHS": "Phase 3 — the paths that trigger the security review", "EVAL_TEST_PROJECT": "Phase 3 — the project holding your eval fixtures", "SOLUTION_OR_PROJECT": "Phase 3 — the solution or project CI builds", @@ -45,7 +52,7 @@ "DEPLOY_STEP": "Phase 8 — the real deploy command", "CAPTURE_LAST_GOOD": "Phase 8 — how you record the currently-live version", "RESTORE_LAST_GOOD": "Phase 8 — how you roll back", - "DEV_ENVIRONMENT": "Phase 3 — the GitHub Environment name", + "DEV_ENVIRONMENT": "Phase 3 — the deploy Environment name (GitHub / Azure DevOps)", "ARTIFACT_NAME": "Phase 3 — the CI artifact the deploy promotes", "CI_WORKFLOW_NAME": "Phase 3 — the CI workflow name that triggers deploy", } @@ -57,6 +64,15 @@ RESIDUAL_TOKEN = re.compile(r"<<([A-Z][A-Z0-9_]*)>>|\b([A-Z][A-Z0-9_]*)_NOT_SET\b") SCANNED_SUFFIXES = {".json", ".yml", ".yaml", ".md", ".sh", ".ps1"} +# A pipeline's `- group: NAME` variable-group reference. Quoted names are captured whole; a bare +# name runs to end-of-line minus a trailing ` # comment` (in YAML a '#' opens a comment only after +# whitespace, and a name containing '#' must be quoted anyway). Comment-only lines never match — +# they cannot begin with `- group:`. +_GROUP_REF = re.compile( + r"""^\s*-\s*group:\s*(?:'([^']+)'|"([^"]+)"|(\S[^#]*?))\s*(?:\#.*)?$""", + re.MULTILINE, +) + @dataclass class Result: @@ -66,15 +82,44 @@ class Result: fix: str = "" -def _run(cmd: list[str], timeout: int = 30) -> tuple[int, str]: +def _run(cmd: list[str], timeout: int = 30, cwd: str | None = None) -> tuple[int, str]: try: p = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout, - encoding="utf-8", errors="replace") + encoding="utf-8", errors="replace", cwd=cwd) return p.returncode, (p.stdout or "") + (p.stderr or "") except (FileNotFoundError, subprocess.TimeoutExpired) as exc: return 127, str(exc) +def _az() -> str: + """Resolve az through shutil.which, which honors PATHEXT. On Windows the Azure CLI installs + as az.cmd — subprocess.run(["az", ...]) without a shell raises FileNotFoundError there, so + the bare name would turn every ADO network check into a permanent, misdiagnosed WARN on the + platform most of this module's users are on.""" + return shutil.which("az") or "az" + + +# ── platform ────────────────────────────────────────────────────────────────────────────────── + +def installed_platform(repo: Path) -> str: + """Which CI platform this install realized, read from the manifest's composed pack ids. + + The install manifest is the one artifact that records what actually composed (the frozen + profile records what was *asked* for; an axis can degrade). Only a manifest that names the + azure-devops CI/CD pack flips the platform-facing checks to az — a missing, corrupt, or + pack-less manifest gets the GitHub checks, which is exactly the pre-pack behavior, so + nothing that exists today changes. + """ + manifest = repo / ".claude" / "harness-manifest.json" + try: + packs = json.loads(manifest.read_text(encoding="utf-8")).get("packs") or [] + return "azure-devops" if "cicd/azure-devops" in packs else "github" + except (OSError, json.JSONDecodeError, AttributeError, TypeError): + # TypeError covers a type-corrupt manifest ("packs": 5) — a diagnostic tool + # must never be the thing that crashes. + return "github" + + # ── tools ───────────────────────────────────────────────────────────────────────────────────── def check_tools(repo: Path) -> list[Result]: @@ -83,11 +128,21 @@ def check_tools(repo: Path) -> list[Result]: settings.json registers the hooks via `pwsh` (PowerShell 7 is cross-platform). Without it the Stop hook never fires — the agent can end a turn on a red build and nothing objects. The harness looks installed and the highest-value rung of the ladder is simply absent. + + The platform CLI follows the installed pack: demanding gh on an Azure DevOps repo is a FAIL + for a tool nothing there uses — the doctor equivalent of telling you to fix a working setup. """ + if installed_platform(repo) == "azure-devops": + platform_cli = ("az", "branch policies, variable groups, PR gates", + "install the Azure CLI (https://aka.ms/azure-cli), " + "then: az extension add --name azure-devops") + else: + platform_cli = ("gh", "branch protection, secrets, PR gates", + "install the GitHub CLI: https://cli.github.com") out = [] for tool, why, fix in [ ("git", "everything", "install git"), - ("gh", "branch protection, secrets, PR gates", "install the GitHub CLI: https://cli.github.com"), + platform_cli, ("pwsh", "THE HOOKS — settings.json runs them through pwsh", "install PowerShell 7: https://aka.ms/powershell"), ]: path = shutil.which(tool) @@ -229,7 +284,7 @@ def check_residual_tokens(repo: Path) -> list[Result]: later as whatever the tool does with a literal `<>`, which on Windows can be a raw cmd.exe syntax error that names neither the token nor the tool. """ - roots = [repo / ".github", repo / ".claude", repo / "scripts" / "rails"] + roots = [repo / ".github", repo / ".azuredevops", repo / ".claude", repo / "scripts" / "rails"] candidates = [p for root in roots if root.exists() for p in root.rglob("*")] if (repo / ".mcp.json").exists(): candidates.append(repo / ".mcp.json") @@ -273,7 +328,7 @@ def check_mcp(repo: Path) -> list[Result]: return [Result(PASS, f"{len(servers)} MCP server(s) declared", ", ".join(sorted(servers)))] -# ── repo state (needs gh) ───────────────────────────────────────────────────────────────────── +# ── repo state (needs gh / az) ──────────────────────────────────────────────────────────────── def required_secrets(repo: Path) -> dict[str, list[str]]: """Secrets THIS repo's workflows actually reference, mapped to the workflows needing them. @@ -299,7 +354,7 @@ def required_secrets(repo: Path) -> dict[str, list[str]]: return needed -def check_secrets(repo: Path) -> list[Result]: +def _check_secrets_github(repo: Path) -> list[Result]: needed = required_secrets(repo) if not needed: return [Result(WARN, "no workflow secrets to check", @@ -327,7 +382,85 @@ def check_secrets(repo: Path) -> list[Result]: return results -def check_branch_protection(repo: Path) -> list[Result]: +def required_variable_groups(repo: Path) -> dict[str, list[str]]: + """Variable groups THIS repo's pipelines actually reference, mapped to the pipelines needing them. + + The ADO twin of required_secrets, with the same lesson baked in: read from the installed + pipelines rather than a fixed list, so a repo that renamed its group (or deleted the eval + pipelines) is never told to "fix" a working setup. ADO pipelines get their secrets from + variable groups (`- group: NAME`, ideally Key-Vault-backed), not per-repo secrets — so the + group's existence is what the doctor can check from here; which variables it exposes lives + server-side. A still-unfilled `<>` token never matches the name pattern: + that is unfinished setup, owned by the residual-token check, not a missing group. + """ + pipe_dir = repo / ".azuredevops" / "pipelines" + if not pipe_dir.exists(): + return {} + needed: dict[str, list[str]] = {} + for yml in sorted(pipe_dir.glob("*.yml")): + body = yml.read_text(encoding="utf-8", errors="replace") + names = set() + for quoted1, quoted2, bare in _GROUP_REF.findall(body): + # Quoted names are captured whole (spaces, parens, unicode, even '#' — quoting is + # the escape hatch for exotic-but-legal group names). A mangled name would FAIL + # demanding a group that does not exist under that name — telling the user to fix + # a working setup, the exact failure this module's header forbids. + name = (quoted1 or quoted2 or bare).strip() + # Not group references: unfinished setup (the residual-token check owns both + # placeholder forms) and runtime template expressions. + if not name or "<<" in name or "${{" in name or name.endswith("_NOT_SET"): + continue + names.add(name) + for name in names: + needed.setdefault(name, []).append(yml.name) + return needed + + +def _check_secrets_ado(repo: Path) -> list[Result]: + """The variable groups the pipelines reference must exist server-side, or the key-consuming + gates fail closed on their first run — the same silent-absence shape as a missing secret.""" + needed = required_variable_groups(repo) + if not needed: + return [Result(WARN, "no pipeline variable groups to check", + "no installed pipeline references a variable group (an unfilled " + "<> token is reported under Unfinished setup)")] + + # cwd=repo so az's org/project auto-detection reads THIS repo's remote, not wherever the + # doctor happens to be running from; --only-show-errors keeps az's success-path stderr + # chatter (extension notices, preview warnings) out of the JSON we parse. + rc, out = _run([_az(), "pipelines", "variable-group", "list", "--output", "json", + "--only-show-errors"], timeout=30, cwd=str(repo)) + if rc != 0: + return [Result(WARN, "variable groups not checked", + "az could not list variable groups (not logged in, azure-devops extension " + "missing, or org/project not detectable from this repo's remote)", + "az login && az extension add --name azure-devops — run from the repo clone")] + try: + present = {g["name"] for g in json.loads(out)} + except (json.JSONDecodeError, KeyError, TypeError): + return [Result(WARN, "variable groups not checked", "unexpected az output")] + + results = [] + for group, pipelines in sorted(needed.items()): + if group in present: + results.append(Result(PASS, f"variable group '{group}' exists", ", ".join(pipelines))) + else: + results.append(Result( + FAIL, f"variable group '{group}' missing", + f"{', '.join(pipelines)} reference it — those gates fail closed without it", + f"create it (ideally Key-Vault-backed) with the keys the pipelines read, " + f"e.g. az pipelines variable-group create --name '{group}' " + f"--variables ANTHROPIC_API_KEY=")) + return results + + +def check_secrets(repo: Path) -> list[Result]: + if installed_platform(repo) == "azure-devops": + return _check_secrets_ado(repo) + return _check_secrets_github(repo) + + +def _check_branch_protection_github(repo: Path) -> list[Result]: rc, out = _run(["gh", "api", "repos/{owner}/{repo}/rulesets"], timeout=30) if rc != 0: return [Result(WARN, "branch protection not checked", @@ -349,6 +482,48 @@ def check_branch_protection(repo: Path) -> list[Result]: ", ".join(r.get("name", "?") for r in active))] +def _check_branch_policies_ado(repo: Path) -> list[Result]: + """Branch policies are ADO's branch protection: build-validation per gate pipeline plus the + non-author approver count. Without them the gates run and nothing makes them mandatory. + Enforcing means isEnabled AND isBlocking — an enabled-but-optional policy is advice, not a + gate, the same "exists but is not enforcing" condition the GitHub twin checks via + enforcement == active. Project-level listing (az detects org/project from the repo's + remote): sibling repos' policies can inflate the count — and can mask a repo with none of + its own — so a PASS here means the PROJECT enforces policies, not necessarily this repo; + configure-branch-policies.sh is what scopes them per-repo. Zero is still unambiguous.""" + rc, out = _run([_az(), "repos", "policy", "list", "--output", "json", + "--only-show-errors"], timeout=30, cwd=str(repo)) + if rc != 0: + return [Result(WARN, "branch policies not checked", + "az could not list branch policies (not logged in, azure-devops extension " + "missing, or org/project not detectable from this repo's remote)", + "az login && az extension add --name azure-devops — run from the repo clone")] + try: + policies = json.loads(out) + except json.JSONDecodeError: + return [Result(WARN, "branch policies not checked", "unexpected az output")] + if not isinstance(policies, list): + return [Result(WARN, "branch policies not checked", "unexpected az output shape")] + if not policies: + return [Result(FAIL, "no branch policies", + "the gates run but nothing makes them mandatory — a red PR can complete", + "scripts/rails/configure-branch-policies.sh (dry-run first)")] + enforcing = [p for p in policies if p.get("isEnabled") and p.get("isBlocking")] + if not enforcing: + return [Result(FAIL, f"{len(policies)} branch policy(ies), none enabled AND blocking", + "policies exist but none enforces — an optional policy is advice, not a gate", + "enable/make blocking: az repos policy update, or re-run " + "configure-branch-policies.sh")] + kinds = sorted({(p.get("type") or {}).get("displayName") or "?" for p in enforcing}) + return [Result(PASS, f"{len(enforcing)} enforcing branch policy(ies)", ", ".join(kinds))] + + +def check_branch_protection(repo: Path) -> list[Result]: + if installed_platform(repo) == "azure-devops": + return _check_branch_policies_ado(repo) + return _check_branch_protection_github(repo) + + # ── report ──────────────────────────────────────────────────────────────────────────────────── SECTIONS = [ @@ -392,7 +567,7 @@ def run(repo: Path, offline: bool = False) -> int: def main() -> int: ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) ap.add_argument("--repo", default=".", help="repo to check (default: current directory)") - ap.add_argument("--offline", action="store_true", help="skip checks that need gh") + ap.add_argument("--offline", action="store_true", help="skip checks that need gh/az") args = ap.parse_args() return run(Path(args.repo), offline=args.offline) diff --git a/scripts/install_harness.py b/scripts/install_harness.py index 69d104f..e6b180b 100644 --- a/scripts/install_harness.py +++ b/scripts/install_harness.py @@ -113,6 +113,42 @@ ] GITIGNORE_LINES = [".claude/.review-receipts/", ".claude/settings.local.json"] +# The core kit's governance assets are laid out GitHub-flavored by default (workflows under +# .github/workflows/, rubrics under .github/profile/rubrics/, a CODEOWNERS, a ruleset). For a +# non-GitHub CI/CD platform that layout is remapped by source-relative path: +# * drop_dirs / drop_files — pure-GitHub MECHANISMS the platform's own CI/CD pack replaces, so +# dropping them leaves no dead GitHub artifacts in the repo. +# * redirect — neutral, platform-agnostic governance CONTENT that the platform's pipelines read +# from a different home (e.g. the azure-devops pack's pipelines reference .azuredevops/rails/). +# Anything unlisted installs at its default, neutral path. github-actions and core-only installs +# (platform None) are ABSENT here on purpose: they fall through to the identity layout, so those +# installs stay byte-for-byte identical to before this seam existed (the additive guarantee). +_CORE_LAYOUT_BY_PLATFORM: dict[str, dict] = { + "azure-devops": { + # Each GitHub mechanism has an ADO analogue shipped by packs/cicd/azure-devops: + # workflows/ -> .azuredevops/pipelines/ (the pipeline YAMLs) + # profile/rulesets/ -> .azuredevops/rails/branch-policies.json + # profile/CODEOWNERS -> required-reviewer branch policy (policies.json) + # apply-branch-protection.sh -> scripts/rails/configure-branch-policies.sh + "drop_dirs": {"workflows/", "profile/rulesets/"}, + "drop_files": {"profile/CODEOWNERS", "profile/scripts/apply-branch-protection.sh"}, + # Neutral content the ADO pipelines read from the rails home (see the azure-devops pack's + # `governance_rubrics` + the .azuredevops/rails/ path references in its pipeline YAMLs). + # rails-telemetry.schema.json is deliberately NOT redirected: BOTH packs' telemetry + # pipelines commit their report to .github/rails-telemetry.json ON PURPOSE — "one + # collector reads a mixed GitHub/Azure fleet without caring which platform wrote a + # file" (azure-pipelines/rails-telemetry.yml header). The operator-side fleet collector + # is external to this repo, so its one-canonical-path contract cannot be updated from + # here; the schema stays beside the report it describes. Moving either means moving + # both AND the collector. + "redirect": { + "profile/rubrics/": ".azuredevops/rails/rubrics/", + "profile/eval-bypasses.md": ".azuredevops/rails/eval-bypasses.md", + "profile/dependency-exceptions.md": ".azuredevops/rails/dependency-exceptions.md", + }, + }, +} + # Profile -> pack resolution. The profile's own stack block is the single source of truth; these # tables are the ONE place profile vocabulary maps to pack ids. A language/platform with no entry # fails closed (there is no pack for it yet) — never a silent core-only install. @@ -211,28 +247,46 @@ def _substitute_in_place(dest: Path, tokens: dict[str, str]) -> None: def _copy_core(payload: Path, target: Path, force: bool, written: set[Path], log: list[str], - missing: list[str]) -> None: + missing: list[str], platform: str | None = None) -> None: """Copy the core maps. A mapped source absent from the payload is logged AND collected in - `missing` — the scan keeps going so every miss is reported, then the caller fails closed.""" + `missing` — the scan keeps going so every miss is reported, then the caller fails closed. + + `platform` (the profile's stack.ci_cd.platform) selects the on-disk layout via + _CORE_LAYOUT_BY_PLATFORM: github-actions and core-only (None) use the identity layout, so those + installs are byte-for-byte unchanged; a non-GitHub platform drops the pure-GitHub mechanisms and + redirects neutral governance content to that platform's rails home.""" + layout = _CORE_LAYOUT_BY_PLATFORM.get(platform or "", {}) + drop_dirs = layout.get("drop_dirs", frozenset()) + drop_files = layout.get("drop_files", frozenset()) + redirect = layout.get("redirect", {}) + # RAILS.md reaches .github/RAILS.md via EXTRA_FILES, so the workflows/ dir scan must not ALSO + # copy it; fold that long-standing skip in with any platform-dropped in-directory files. + dir_skip = {"workflows/RAILS.md"} | set(drop_files) + for src_rel, dest_rel in FILE_MAP + EXTRA_FILES: + if src_rel in drop_files: + log.append(f"OMIT {src_rel} (not used on {platform})") + continue src = payload / src_rel if not src.is_file(): log.append(f"MISS {src_rel} (not in payload)") missing.append(f"{src_rel} (not in payload)") continue - _copy(src, target / dest_rel, force, written, log) + _copy(src, target / redirect.get(src_rel, dest_rel), force, written, log) - # RAILS.md is remapped by EXTRA_FILES; don't also copy it via the workflows dir. - remapped = {"workflows/RAILS.md"} for src_rel, dest_rel in DIR_MAP: + if src_rel in drop_dirs: + log.append(f"OMIT {src_rel} (not used on {platform})") + continue src_dir = payload / src_rel if not src_dir.is_dir(): log.append(f"MISS {src_rel} (not in payload)") missing.append(f"{src_rel} (not in payload)") continue + dest_rel = redirect.get(src_rel, dest_rel) for src in sorted(p for p in src_dir.rglob("*") if p.is_file()): key = src.relative_to(payload).as_posix() - if key in remapped: + if key in dir_skip: continue dest = target / dest_rel / src.relative_to(src_dir) _copy(src, dest, force, written, log) @@ -540,7 +594,11 @@ def _install(payload: Path, target: Path, force: bool, profile_path: Path | None tools_packs, tool_warnings = _resolve_tools(profile, payload) warnings = warnings + fe_warnings + tool_warnings - _copy_core(payload, target, force, written, log, missing) + # The core layout follows the CI/CD platform: a non-GitHub platform drops the GitHub mechanisms + # and redirects neutral governance content (see _CORE_LAYOUT_BY_PLATFORM). None => identity. + stack = (profile or {}).get("stack", {}) or {} + platform = (stack.get("ci_cd", {}) or {}).get("platform") + _copy_core(payload, target, force, written, log, missing, platform) _raise_if_missing(missing) # after the FULL core scan, so every core miss is reported packs = _compose_packs(stack_pack, cicd_pack, frontend_packs, tools_packs, diff --git a/scripts/tests/golden/ado-enterprise-tree.txt b/scripts/tests/golden/ado-enterprise-tree.txt new file mode 100644 index 0000000..c74ac64 --- /dev/null +++ b/scripts/tests/golden/ado-enterprise-tree.txt @@ -0,0 +1,70 @@ +.azuredevops/pipelines/ci.yml +.azuredevops/pipelines/correctness.yml +.azuredevops/pipelines/dependency-scan.yml +.azuredevops/pipelines/deploy-dev.yml +.azuredevops/pipelines/deploy-promote.yml +.azuredevops/pipelines/eval-regression.yml +.azuredevops/pipelines/eval-suite.yml +.azuredevops/pipelines/grader.yml +.azuredevops/pipelines/rails-telemetry.yml +.azuredevops/pipelines/security.yml +.azuredevops/pipelines/templates/checkout-with-base.yml +.azuredevops/pipelines/templates/claude-review.yml +.azuredevops/pipelines/templates/restore-build-test.yml +.azuredevops/pipelines/templates/setup-toolchain.yml +.azuredevops/rails/branch-policies.json +.azuredevops/rails/dependency-exceptions.md +.azuredevops/rails/eval-bypasses.md +.azuredevops/rails/rubrics/correctness.md +.azuredevops/rails/rubrics/grader.md +.azuredevops/rails/rubrics/security.md +.claude/agents/README.md +.claude/agents/architect.md +.claude/agents/build-error-resolver.md +.claude/agents/debugger.md +.claude/agents/grader.md +.claude/agents/planner.md +.claude/agents/security-reviewer.md +.claude/agents/ux-reviewer.md +.claude/harness-manifest.json +.claude/hooks/README.md +.claude/hooks/review-gate.ps1 +.claude/hooks/review-gate.sh +.claude/hooks/save-review-receipt.ps1 +.claude/hooks/save-review-receipt.sh +.claude/hooks/sensitive-edit-nudge.ps1 +.claude/hooks/sensitive-edit-nudge.sh +.claude/hooks/stop-gate.ps1 +.claude/hooks/stop-gate.sh +.claude/rules/clean-architecture.md +.claude/rules/testing.md +.claude/settings.json +.claude/skills/api-pattern-dotnet/SKILL.md +.claude/skills/api-pattern/SKILL.md +.claude/skills/diagnose/SKILL.md +.claude/skills/eval-builder/SKILL.md +.claude/skills/pr-writer/SKILL.md +.claude/skills/spec-writer/SKILL.md +.claude/skills/test-writer/SKILL.md +.github/RAILS.md +.github/rails-telemetry.schema.json +.gitignore +.mcp.json +ALERTS.md +CLAUDE.md +INCIDENT-PLAYBOOK.md +ONBOARDING.md +ROLLBACK.md +docs/harness.md +eval-datasets/README.md +eval-datasets/golden-set.template.yaml +infra/README.md +infra/main.bicep +infra/policy/README.md +prompts/judge.template.v1.md +scripts/rails/configure-branch-policies.sh +scripts/rails/diff-anchors.sh +scripts/rails/post-pr-thread.sh +scripts/rails/run-claude-review.sh +specs/spec-template.md +spikes/spike-template.md diff --git a/scripts/tests/test_doctor.py b/scripts/tests/test_doctor.py index e40a6e1..cf6ba32 100644 --- a/scripts/tests/test_doctor.py +++ b/scripts/tests/test_doctor.py @@ -24,12 +24,16 @@ FAIL, PASS, WARN, + check_branch_protection, check_executable_bits, check_harness_present, check_hooks, check_residual_tokens, check_secrets, + check_tools, + installed_platform, required_secrets, + required_variable_groups, ) @@ -246,6 +250,249 @@ def test_warnings_alone_do_not_fail_the_run(self, repo, monkeypatch): assert doctor.run(repo, offline=True) == 0 +# ── platform awareness: an ADO repo is never told to fix a GitHub it does not have ──────────── + +def _make_ado(repo): + """Flip the standard fixture to an Azure DevOps install: manifest names the azure-devops + CI/CD pack, pipelines live in .azuredevops/pipelines/.""" + (repo / ".claude" / "harness-manifest.json").write_text(json.dumps({ + "profile_id": "ado-test", "packs": ["stacks/dotnet", "cicd/azure-devops"], + "files": {"CLAUDE.md": "x"}, + }), encoding="utf-8") + (repo / ".azuredevops" / "pipelines").mkdir(parents=True) + return repo + + +class TestInstalledPlatform: + def test_the_ado_pack_flips_the_platform(self, repo): + assert installed_platform(_make_ado(repo)) == "azure-devops" + + def test_the_github_pack_is_github(self, repo): + assert installed_platform(repo) == "github" + + def test_no_manifest_defaults_to_github(self, tmp_path): + """The pre-pack behavior. A missing manifest must not change what the doctor says.""" + assert installed_platform(tmp_path) == "github" + + def test_a_corrupt_manifest_defaults_to_github_rather_than_raising(self, repo): + (repo / ".claude" / "harness-manifest.json").write_text("{not json", encoding="utf-8") + assert installed_platform(repo) == "github" + + def test_a_packless_manifest_defaults_to_github(self, repo): + (repo / ".claude" / "harness-manifest.json").write_text( + json.dumps({"profile_id": "core-only", "files": {}}), encoding="utf-8") + assert installed_platform(repo) == "github" + + def test_a_type_corrupt_packs_value_defaults_to_github_rather_than_raising(self, repo): + """Valid JSON, wrong type. A diagnostic tool must never be the thing that crashes.""" + (repo / ".claude" / "harness-manifest.json").write_text( + json.dumps({"profile_id": "x", "packs": 5, "files": {}}), encoding="utf-8") + assert installed_platform(repo) == "github" + + +class TestToolsFollowThePlatform: + def test_an_ado_repo_requires_az_not_gh(self, repo, monkeypatch): + """Demanding gh on an Azure DevOps repo is telling you to fix a working setup.""" + monkeypatch.setattr(doctor.shutil, "which", lambda name: None) + titles = _titles(check_tools(_make_ado(repo))) + assert "az missing" in titles + assert "gh" not in titles + + def test_a_github_repo_still_requires_gh(self, repo, monkeypatch): + monkeypatch.setattr(doctor.shutil, "which", lambda name: None) + titles = _titles(check_tools(repo)) + assert "gh missing" in titles + assert "az missing" not in titles # no az row at all + + def test_git_and_pwsh_are_required_on_both(self, repo, monkeypatch): + monkeypatch.setattr(doctor.shutil, "which", lambda name: None) + for r in (repo, _make_ado(repo)): + titles = _titles(check_tools(r)) + assert "git missing" in titles and "pwsh missing" in titles + + +class TestRequiredVariableGroups: + def _pipe(self, repo, name, body): + (repo / ".azuredevops" / "pipelines" / name).write_text(body, encoding="utf-8") + + def test_groups_come_from_the_installed_pipelines(self, repo): + _make_ado(repo) + self._pipe(repo, "grader.yml", "variables:\n - group: rails-secrets\n") + assert required_variable_groups(repo) == {"rails-secrets": ["grader.yml"]} + + def test_a_group_named_only_in_a_comment_is_not_required(self, repo): + _make_ado(repo) + self._pipe(repo, "ci.yml", "# reference `- group: rails-secrets` once wired\njobs: []\n") + assert required_variable_groups(repo) == {} + + def test_an_unfilled_variable_group_token_is_not_a_missing_group(self, repo): + """`- group: <>` is unfinished setup — the residual-token check owns it. + Demanding a variable group literally named <> would be a wrong instruction.""" + _make_ado(repo) + self._pipe(repo, "grader.yml", "variables:\n - group: <>\n") + assert required_variable_groups(repo) == {} + + def test_the_argv_safe_sentinel_form_is_not_a_missing_group_either(self, repo): + _make_ado(repo) + self._pipe(repo, "grader.yml", "variables:\n - group: VARIABLE_GROUP_NOT_SET\n") + assert required_variable_groups(repo) == {} + + def test_a_quoted_exotic_name_is_captured_whole_not_mangled(self, repo): + """'My Group (Prod)' truncated to 'My Group' would FAIL demanding a group that does not + exist under that name while the real one works — telling the user to fix a working + setup, the module's cardinal sin. Quoting is the escape hatch for exotic legal names.""" + _make_ado(repo) + self._pipe(repo, "ci.yml", "variables:\n - group: 'My Group (Prod)'\n") + assert required_variable_groups(repo) == {"My Group (Prod)": ["ci.yml"]} + + def test_a_bare_name_with_spaces_parens_and_unicode_survives(self, repo): + _make_ado(repo) + self._pipe(repo, "ci.yml", "variables:\n - group: naïve rails (dev) # exposes KEY\n") + assert required_variable_groups(repo) == {"naïve rails (dev)": ["ci.yml"]} + + def test_a_runtime_template_expression_is_not_a_group_name(self, repo): + _make_ado(repo) + self._pipe(repo, "ci.yml", "variables:\n - group: ${{ parameters.groupName }}\n") + assert required_variable_groups(repo) == {} + + def test_a_github_only_repo_has_no_variable_groups(self, repo): + assert required_variable_groups(repo) == {} + + +class TestAdoSecrets: + def _pipe(self, repo, name, body): + (repo / ".azuredevops" / "pipelines" / name).write_text(body, encoding="utf-8") + + def test_dispatch_uses_az_on_an_ado_repo(self, repo, monkeypatch): + _make_ado(repo) + self._pipe(repo, "grader.yml", "variables:\n - group: rails-secrets\n") + seen = [] + monkeypatch.setattr(doctor.shutil, "which", lambda name: name) # az.cmd resolution aside + monkeypatch.setattr(doctor, "_run", + lambda cmd, **k: (seen.append((cmd, k)), (0, "[]"))[1]) + check_secrets(repo) + cmd, kwargs = seen[0] + assert cmd[0] == "az", "an ADO repo must be checked with az, not gh" + assert "--only-show-errors" in cmd, "az stderr chatter must not corrupt the JSON parse" + assert kwargs.get("cwd") == str(repo), \ + "az must detect org/project from --repo's remote, not the doctor's CWD" + + def test_dispatch_still_uses_gh_on_a_github_repo(self, repo, monkeypatch): + (repo / ".github" / "workflows" / "g.yml").write_text( + " key: ${{ secrets.ANTHROPIC_API_KEY }}\n", encoding="utf-8") + seen = [] + monkeypatch.setattr(doctor, "_run", lambda cmd, **k: (seen.append(cmd), (0, "[]"))[1]) + check_secrets(repo) + assert seen and seen[0][0] == "gh" + + def test_a_present_group_passes(self, repo, monkeypatch): + _make_ado(repo) + self._pipe(repo, "grader.yml", "variables:\n - group: rails-secrets\n") + monkeypatch.setattr(doctor, "_run", + lambda *a, **k: (0, json.dumps([{"name": "rails-secrets"}]))) + results = check_secrets(repo) + assert FAIL not in _statuses(results) + + def test_a_missing_group_fails_and_names_the_pipeline(self, repo, monkeypatch): + _make_ado(repo) + self._pipe(repo, "grader.yml", "variables:\n - group: rails-secrets\n") + monkeypatch.setattr(doctor, "_run", lambda *a, **k: (0, json.dumps([]))) + results = check_secrets(repo) + assert FAIL in _statuses(results) + assert "grader.yml" in results[0].detail, "say which pipeline needs it" + + def test_unreachable_az_warns_rather_than_fails(self, repo, monkeypatch): + """Offline is not a broken harness — same rule as the gh path.""" + _make_ado(repo) + self._pipe(repo, "grader.yml", "variables:\n - group: rails-secrets\n") + monkeypatch.setattr(doctor, "_run", lambda *a, **k: (1, "az login required")) + assert check_secrets(repo)[0].status == WARN + + def test_no_group_references_warns_rather_than_inventing_a_requirement(self, repo): + _make_ado(repo) + assert check_secrets(repo)[0].status == WARN + + +class TestAdoBranchPolicies: + def test_dispatch_uses_az_on_an_ado_repo(self, repo, monkeypatch): + _make_ado(repo) + seen = [] + monkeypatch.setattr(doctor.shutil, "which", lambda name: name) + monkeypatch.setattr(doctor, "_run", lambda cmd, **k: (seen.append(cmd), (0, "[]"))[1]) + check_branch_protection(repo) + assert seen and seen[0][:3] == ["az", "repos", "policy"] + + def test_no_policies_fails_and_points_at_the_configure_script(self, repo, monkeypatch): + _make_ado(repo) + monkeypatch.setattr(doctor, "_run", lambda *a, **k: (0, "[]")) + results = check_branch_protection(repo) + assert results[0].status == FAIL + assert "configure-branch-policies.sh" in results[0].fix + + def test_enforcing_policies_pass_naming_their_kinds(self, repo, monkeypatch): + _make_ado(repo) + monkeypatch.setattr(doctor, "_run", lambda *a, **k: (0, json.dumps([ + {"isEnabled": True, "isBlocking": True, "type": {"displayName": "Build"}}, + {"isEnabled": True, "isBlocking": True, + "type": {"displayName": "Minimum number of reviewers"}}, + ]))) + results = check_branch_protection(repo) + assert results[0].status == PASS + assert "Build" in results[0].detail + + def test_disabled_policies_fail_rather_than_passing_on_existence(self, repo, monkeypatch): + """A policy that exists but does not enforce is the same silent absence as none at all.""" + _make_ado(repo) + monkeypatch.setattr(doctor, "_run", lambda *a, **k: (0, json.dumps([ + {"isEnabled": False, "isBlocking": True, "type": {"displayName": "Build"}}, + ]))) + assert check_branch_protection(repo)[0].status == FAIL + + def test_an_enabled_but_optional_policy_is_not_enforcement(self, repo, monkeypatch): + """isEnabled without isBlocking is advice, not a gate — the ADO twin of a GitHub + ruleset whose enforcement is not 'active'. Passing on it would claim a protection + that does not block a red PR.""" + _make_ado(repo) + monkeypatch.setattr(doctor, "_run", lambda *a, **k: (0, json.dumps([ + {"isEnabled": True, "isBlocking": False, "type": {"displayName": "Build"}}, + ]))) + assert check_branch_protection(repo)[0].status == FAIL + + def test_a_dict_shaped_response_warns_rather_than_raising(self, repo, monkeypatch): + monkeypatch.setattr(doctor, "_run", lambda *a, **k: (0, json.dumps({"value": []}))) + assert check_branch_protection(_make_ado(repo))[0].status == WARN + + def test_unreachable_az_warns_rather_than_fails(self, repo, monkeypatch): + _make_ado(repo) + monkeypatch.setattr(doctor, "_run", lambda *a, **k: (1, "not logged in")) + assert check_branch_protection(repo)[0].status == WARN + + def test_a_github_repo_is_still_checked_via_gh_api(self, repo, monkeypatch): + seen = [] + monkeypatch.setattr(doctor, "_run", lambda cmd, **k: (seen.append(cmd), (0, "[]"))[1]) + check_branch_protection(repo) + assert seen and seen[0][0] == "gh" + + +class TestResidualTokensScanAdoHome: + def test_a_token_in_an_ado_pipeline_is_reported(self, repo): + """.azuredevops/ is a scan root — an unfilled <> in security.yml was + invisible before the platform-aware pass, exactly the silent-absence shape.""" + _make_ado(repo) + (repo / ".azuredevops" / "pipelines" / "security.yml").write_text( + "env:\n GATED: <>\n", encoding="utf-8") + results = check_residual_tokens(repo) + assert WARN in _statuses(results) + assert "GATED_PATHS" in _titles(results) + + def test_the_variable_group_token_names_its_owner(self, repo): + _make_ado(repo) + (repo / ".azuredevops" / "pipelines" / "grader.yml").write_text( + "variables:\n - group: <>\n", encoding="utf-8") + fix = [r.fix for r in check_residual_tokens(repo) if "VARIABLE_GROUP" in r.title][0] + assert "Phase 3" in fix + + class TestArgvSafePlaceholders: """A placeholder that lands in an ARGV cannot use << >> (issue #28). diff --git a/scripts/tests/test_golden_repo.py b/scripts/tests/test_golden_repo.py index 4830f1c..fe36586 100644 --- a/scripts/tests/test_golden_repo.py +++ b/scripts/tests/test_golden_repo.py @@ -29,7 +29,9 @@ PAYLOAD = _REPO_ROOT / "harness" PROFILE = _REPO_ROOT / "profiles" / "microsoft-enterprise" / "profile.yaml" STARTER_PROFILE = _REPO_ROOT / "profiles" / "starter" / "profile.yaml" +ADO_PROFILE = _REPO_ROOT / "profiles" / "ado-enterprise" / "profile.yaml" GOLDEN_TREE = Path(__file__).resolve().parent / "golden" / "enterprise-tree.txt" +ADO_GOLDEN_TREE = Path(__file__).resolve().parent / "golden" / "ado-enterprise-tree.txt" REGEN_HINT = ( "installed tree diverged from tests/golden/enterprise-tree.txt.\n" @@ -144,6 +146,128 @@ def test_emitted_workflows_parse(self, golden_repo): yaml.safe_load(wf.read_text(encoding="utf-8")) +@pytest.fixture(scope="class") +def ado_repo(tmp_path_factory): + target = tmp_path_factory.mktemp("ado-repo") + subprocess.run(["git", "init", "-q"], cwd=target, check=True) + out, err = io.StringIO(), io.StringIO() + with redirect_stdout(out), redirect_stderr(err): + rc = install(PAYLOAD, target, force=False, profile_path=ADO_PROFILE) + assert rc == 0, f"install failed rc={rc}\n{out.getvalue()}\n{err.getvalue()}" + return target, out.getvalue(), err.getvalue() + + +class TestAdoGoldenRepo: + """The Azure DevOps twin of TestGoldenRepo (Fold A): the SAME flagship stack, but + profiles/ado-enterprise selects platform: azure-devops, so the core install must drop the + GitHub mechanisms and redirect neutral governance content to the ADO rails home — while the + github install (TestGoldenRepo above, enterprise-tree.txt) stays byte-for-byte unchanged. + + Regenerate the ADO snapshot the same way as the github one (one run does both): + GOLDEN_REGEN=1 uv run --project . pytest tests/test_golden_repo.py -q + """ + + def test_tree_matches_snapshot(self, ado_repo): + target, _, _ = ado_repo + tree = _installed_tree(target) + if os.environ.get("GOLDEN_REGEN") == "1": + ADO_GOLDEN_TREE.parent.mkdir(parents=True, exist_ok=True) + ADO_GOLDEN_TREE.write_text("\n".join(tree) + "\n", encoding="utf-8") + pytest.skip(f"regenerated {ADO_GOLDEN_TREE.name} ({len(tree)} paths) — review and commit it") + assert ADO_GOLDEN_TREE.is_file(), f"missing snapshot {ADO_GOLDEN_TREE}.\n{REGEN_HINT}" + expected = ADO_GOLDEN_TREE.read_text(encoding="utf-8").splitlines() + assert tree == expected, REGEN_HINT + + def test_no_github_actions_payload(self, ado_repo): + # The core no longer dumps GitHub Actions machinery into an ADO repo. These are the exact + # dead artifacts Fold A removes; the ADO pack ships the analogue of each. + target, _, _ = ado_repo + assert not (target / ".github" / "workflows").exists(), "GitHub workflows on an ADO repo" + assert not (target / ".github" / "rulesets").exists(), "GitHub ruleset on an ADO repo" + assert not (target / ".github" / "CODEOWNERS").exists(), "CODEOWNERS on an ADO repo" + assert not (target / "scripts" / "rails" / "apply-branch-protection.sh").exists(), \ + "GitHub-only branch-protection script on an ADO repo" + + def test_rubrics_live_in_the_ado_rails_home(self, ado_repo): + # The live bug Fold A fixes: the ADO pipelines read rubrics from .azuredevops/rails/rubrics, + # but the core used to install them only to .github/profile/rubrics — a path nothing created. + target, _, _ = ado_repo + rubrics = target / ".azuredevops" / "rails" / "rubrics" + for name in ("grader.md", "correctness.md", "security.md"): + assert (rubrics / name).is_file(), f"rubric {name} not in the ADO rails home" + assert not (target / ".github" / "profile").exists(), "rubrics still under .github/ on ADO" + + def test_ledgers_redirected_to_ado_rails_home(self, ado_repo): + # The eval-bypasses / dependency-exceptions ledgers are read by the ADO pipelines from + # .azuredevops/rails/ — same redirect class as the rubrics. + target, _, _ = ado_repo + rails = target / ".azuredevops" / "rails" + assert (rails / "eval-bypasses.md").is_file() + assert (rails / "dependency-exceptions.md").is_file() + assert not (target / ".github" / "eval-bypasses.md").exists() + assert not (target / ".github" / "dependency-exceptions.md").exists() + + def test_settings_carry_the_az_permission_fragment(self, ado_repo): + """Fold C: the ADO pack merges its settings fragment — read-only az queries allowed, + mutating az + the policy script behind ask, and .azuredevops/ gated like .github/.""" + target, _, _ = ado_repo + settings = json.loads((target / ".claude" / "settings.json").read_text(encoding="utf-8")) + allow = settings["permissions"]["allow"] + ask = settings["permissions"]["ask"] + assert "Bash(az repos pr show:*)" in allow, "az read-only queries missing from allow" + assert "az repos pr view" not in json.dumps(allow), \ + "'az repos pr view' does not exist — the az read command is 'pr show' (gh vocabulary)" + assert "Bash(az repos policy:*)" in ask, "mutating az policy calls not behind ask" + assert "Edit(./.azuredevops/**)" in ask, \ + "pipeline definitions lack the tamper protection .github/** has" + assert "Bash(git push --force:*)" in settings["permissions"]["deny"], \ + "core deny rule lost in the fragment merge" + + def test_neutral_scripts_still_land(self, ado_repo): + # diff-anchors.sh is platform-neutral (both packs' pipelines call it) and must survive the + # per-file drop of its GitHub-only sibling apply-branch-protection.sh. + target, _, _ = ado_repo + assert (target / "scripts" / "rails" / "diff-anchors.sh").is_file() + + def test_ado_pipelines_composed(self, ado_repo): + target, _, _ = ado_repo + pipelines = sorted((target / ".azuredevops" / "pipelines").glob("*.yml")) + assert len(pipelines) == 10, f"expected 10 ADO pipelines, got {len(pipelines)}" + + def test_emitted_pipelines_parse(self, ado_repo): + target, _, _ = ado_repo + pipelines = sorted((target / ".azuredevops" / "pipelines").rglob("*.yml")) + assert pipelines, "no ADO pipelines installed" + for pl in pipelines: + yaml.safe_load(pl.read_text(encoding="utf-8")) + + def test_no_seam_token_survives(self, ado_repo): + # Same fail-closed guard the starter repo carries: a surviving <> SEAM token would be + # a pipeline referencing a value the seam never filled. Phase-3 CI_ blanks are not seam + # tokens and are ignored by residual_tokens (this is exactly how the installer audits). + from ci_tokens import residual_tokens + target, _, _ = ado_repo + for path in target.rglob("*"): + if path.is_file() and path.suffix in (".yml", ".yaml"): + left = residual_tokens(path.read_text(encoding="utf-8")) + assert not left, f"{path.name} kept seam tokens: {left}" + + def test_manifest_records_ado_profile_and_pack(self, ado_repo): + target, _, _ = ado_repo + manifest = json.loads( + (target / ".claude" / "harness-manifest.json").read_text(encoding="utf-8") + ) + assert manifest["profile_id"] == "ado-enterprise" + assert "cicd/azure-devops" in manifest["packs"], manifest["packs"] + + def test_ado_install_is_warning_free(self, ado_repo): + # ado-enterprise declares csharp + azure-devops + angular; every axis has a pack, so a + # warning would mean an axis silently degraded. + _, _, err = ado_repo + warnings = [l for l in err.splitlines() if "WARNING" in l] + assert warnings == [], f"ado-enterprise degraded on an axis: {warnings}" + + @pytest.fixture(scope="class") def starter_repo(tmp_path_factory): target = tmp_path_factory.mktemp("starter-repo") @@ -217,3 +341,14 @@ def test_emitted_workflows_parse(self, starter_repo): target, _, _ = starter_repo for wf in sorted((target / ".github" / "workflows").glob("*.yml")): yaml.safe_load(wf.read_text(encoding="utf-8")) + + +class TestGithubSettingsCarryNoAzEntries: + """Fold C's additive guarantee: the az permission fragment ships only with the ADO pack — + a GitHub install's settings.json must not grow az entries it has no use for.""" + + def test_starter_settings_have_no_az_permissions(self, starter_repo): + target, _, _ = starter_repo + settings = (target / ".claude" / "settings.json").read_text(encoding="utf-8") + assert "az repos" not in settings + assert ".azuredevops" not in settings diff --git a/scripts/tests/test_install_harness.py b/scripts/tests/test_install_harness.py index f526b51..cd68c79 100644 --- a/scripts/tests/test_install_harness.py +++ b/scripts/tests/test_install_harness.py @@ -528,6 +528,64 @@ def test_unfilled_token_in_pack_file_fails_closed_naming_file_and_token( assert "dotnet test" in (target / ".github" / "RAILS.md").read_text(encoding="utf-8") +class TestAdoCoreLayout: + """Fold A: on platform azure-devops the CORE install drops the pure-GitHub mechanisms and + redirects neutral governance content to .azuredevops/rails/. The github/core-only layout (every + OTHER test in this file) is untouched — the identity path. Uses the moved-pack trick from + TestCiSeam so an azure-devops CI/CD pack exists to compose; the assertions here are about the + CORE copy, not the pack overlay (the real-payload golden test covers pack composition).""" + + def _install_ado(self, tmp_path, target, valid_profile) -> Path: + payload = make_payload( + tmp_path, toolchain_map={"dotnet": {"action": "UseDotNet@2", "input": "version"}}) + shutil.move(str(payload / "packs" / "cicd" / "github"), + str(payload / "packs" / "cicd" / "azure-devops")) + profile = {**valid_profile, + "stack": {**valid_profile["stack"], "ci_cd": {"platform": "azure-devops"}}} + assert install(payload, target, force=False, + profile_path=_profile_file(tmp_path, profile)) == 0 + return target + + def test_rubrics_redirected_to_ado_rails_home(self, tmp_path, target, valid_profile): + t = self._install_ado(tmp_path, target, valid_profile) + assert (t / ".azuredevops" / "rails" / "rubrics" / "grader.md").is_file() + assert not (t / ".github" / "profile").exists(), "rubrics still under .github/ on ADO" + + def test_ledgers_redirected_to_ado_rails_home(self, tmp_path, target, valid_profile): + t = self._install_ado(tmp_path, target, valid_profile) + assert (t / ".azuredevops" / "rails" / "eval-bypasses.md").is_file() + assert (t / ".azuredevops" / "rails" / "dependency-exceptions.md").is_file() + assert not (t / ".github" / "eval-bypasses.md").exists() + assert not (t / ".github" / "dependency-exceptions.md").exists() + + def test_github_mechanisms_dropped(self, tmp_path, target, valid_profile): + t = self._install_ado(tmp_path, target, valid_profile) + assert not (t / ".github" / "rulesets").exists(), "GitHub ruleset on an ADO repo" + assert not (t / ".github" / "CODEOWNERS").exists(), "CODEOWNERS on an ADO repo" + + def test_neutral_script_survives_the_per_file_drop(self, tmp_path, target, valid_profile): + # profile/scripts/ is a MIXED dir: diff-anchors.sh is neutral (keep), apply-branch- + # protection.sh is GitHub-only (dropped). Only the sibling is dropped, per-file. + t = self._install_ado(tmp_path, target, valid_profile) + assert (t / "scripts" / "rails" / "diff-anchors.sh").is_file() + + def test_telemetry_schema_stays_beside_its_output(self, tmp_path, target, valid_profile): + # Deliberately NOT redirected: the ADO rails-telemetry pipeline writes its report to + # .github/rails-telemetry.json, so the schema stays beside it. + t = self._install_ado(tmp_path, target, valid_profile) + assert (t / ".github" / "rails-telemetry.schema.json").is_file() + + def test_github_platform_keeps_the_github_layout(self, payload, target, tmp_path, valid_profile): + # The control (additive guarantee): the SAME core files stay at their .github/ homes for a + # github-actions profile. Only azure-devops remaps. + assert install(payload, target, force=False, + profile_path=_profile_file(tmp_path, valid_profile)) == 0 + assert (target / ".github" / "profile" / "rubrics" / "grader.md").is_file() + assert (target / ".github" / "rulesets" / "main.json").is_file() + assert (target / ".github" / "CODEOWNERS").is_file() + assert (target / ".github" / "eval-bypasses.md").is_file() + + class TestCoverageFloorPrecedence: """The customer profile outranks the stack pack (composition order: profile is layer 6, the stack pack layer 2). The stack pack's ci-profile declares a DEFAULT floor; a profile that diff --git a/scripts/tests/test_review_gate_hook.py b/scripts/tests/test_review_gate_hook.py new file mode 100644 index 0000000..57410c3 --- /dev/null +++ b/scripts/tests/test_review_gate_hook.py @@ -0,0 +1,185 @@ +"""Behavior tests for the review-gate hook (Fold C) — the SHIPPED payload scripts, not a model. + +The gate's one job: `git push` / `gh pr create` / `az repos pr create` refuse until per-commit +review receipts exist. Fold C added the az trigger; these tests pin all three triggers plus the +non-trigger cases (a PR command *mentioned* is not a PR command *invoked*), against the real +harness/hooks/review-gate.sh executed the way Claude Code executes it: PreToolUse JSON on stdin. + +The pwsh twin runs the same table where pwsh exists (Windows CI); skipped elsewhere — the .sh +and .ps1 forms ship together and must agree, so the table is shared. +""" + +import json +import os +import shutil +import subprocess +from pathlib import Path + +import pytest + +_REPO_ROOT = Path(__file__).resolve().parent.parent.parent +SH_HOOK = _REPO_ROOT / "harness" / "hooks" / "review-gate.sh" +PS1_HOOK = _REPO_ROOT / "harness" / "hooks" / "review-gate.ps1" + + +def _bash() -> str | None: + """The bash that can actually run the hook. + + On Windows, PATH's `bash` is System32's WSL shim, which on CI runners (no distro + installed) prints "Windows Subsystem for Linux has no installed distributions" and + exits 1 — it never runs the script at all. Git Bash is the bash a real Windows + install executes hooks with, so resolve it explicitly and never fall back to the shim. + """ + if os.name != "nt": + return shutil.which("bash") + for var in ("ProgramFiles", "ProgramFiles(x86)"): + base = os.environ.get(var) + if base and (Path(base) / "Git" / "bin" / "bash.exe").is_file(): + return str(Path(base) / "Git" / "bin" / "bash.exe") + git = shutil.which("git") + if git: # /cmd/git.exe -> /bin/bash.exe + cand = Path(git).parent.parent / "bin" / "bash.exe" + if cand.is_file(): + return str(cand) + found = shutil.which("bash") + return None if found and "system32" in found.lower() else found + + +BASH = _bash() + +needs_sh = pytest.mark.skipif( + BASH is None or shutil.which("jq") is None, + reason="review-gate.sh needs bash (Git Bash on Windows) + jq " + "(the hook itself fails open without jq)") +needs_pwsh = pytest.mark.skipif( + shutil.which("pwsh") is None, reason="review-gate.ps1 needs pwsh") + + +@pytest.fixture +def gated_repo(tmp_path): + """A repo where the gate SHOULD fire: a spec branch whose diff vs main touches src/, + committed clean, with no review receipts.""" + subprocess.run(["git", "init", "-q", "-b", "main"], cwd=tmp_path, check=True) + env = {"GIT_AUTHOR_NAME": "t", "GIT_AUTHOR_EMAIL": "t@t", "GIT_COMMITTER_NAME": "t", + "GIT_COMMITTER_EMAIL": "t@t", "PATH": os.environ["PATH"], "HOME": str(tmp_path)} + (tmp_path / "README.md").write_text("x\n", encoding="utf-8") + subprocess.run(["git", "add", "-A"], cwd=tmp_path, check=True, env=env) + subprocess.run(["git", "commit", "-qm", "init"], cwd=tmp_path, check=True, env=env) + subprocess.run(["git", "checkout", "-qb", "spec/0001-test"], cwd=tmp_path, check=True, env=env) + src = tmp_path / "src" + src.mkdir() + (src / "Thing.cs").write_text("class Thing {}\n", encoding="utf-8") + subprocess.run(["git", "add", "-A"], cwd=tmp_path, check=True, env=env) + subprocess.run(["git", "commit", "-qm", "feat: thing"], cwd=tmp_path, check=True, env=env) + return tmp_path + + +def _payload(command: str) -> str: + return json.dumps({"tool_name": "Bash", "tool_input": {"command": command}}) + + +def _hook_env(repo: Path, extra_env: dict | None) -> dict: + """The hook honors RAILS_* knobs (BASE, SRC_REGEX, KINDS, SKIP) — a developer who + experimented with them in their shell would otherwise flip every deny row to allow. + GIT_DIR/GIT_WORK_TREE would similarly point the hook's git at the wrong repo.""" + env = {k: v for k, v in os.environ.items() + if not k.startswith("RAILS_") and k not in ("GIT_DIR", "GIT_WORK_TREE")} + env["CLAUDE_PROJECT_DIR"] = str(repo) + env.update(extra_env or {}) + return env + + +def _run_sh(repo: Path, command: str, extra_env: dict | None = None): + p = subprocess.run([BASH, str(SH_HOOK)], input=_payload(command), capture_output=True, + text=True, env=_hook_env(repo, extra_env), cwd=repo, timeout=30) + return p.returncode, p.stdout, p.stderr + + +def _run_ps1(repo: Path, command: str, extra_env: dict | None = None): + p = subprocess.run(["pwsh", "-NoProfile", "-File", str(PS1_HOOK)], input=_payload(command), + capture_output=True, text=True, env=_hook_env(repo, extra_env), + cwd=repo, timeout=60) + return p.returncode, p.stdout, p.stderr + + +def _is_deny(stdout: str) -> bool: + if not stdout.strip(): + return False + decision = json.loads(stdout) + return decision["hookSpecificOutput"]["permissionDecision"] == "deny" + + +# The shared trigger table: (command, must_gate). "Must gate" here means DENY, because the +# fixture repo always has unreviewed src/ changes. +TRIGGERS = [ + ("az repos pr create --title 'x' --source-branch spec/0001-test", True), # Fold C + ("gh pr create --title x --body y", True), # regression + ("git push -u origin spec/0001-test", True), # regression + ("cd sub && az repos pr create -t x", True), # compound segment + ("AZURE_DEVOPS_EXT_PAT=xyz az repos pr create -t x", True), # env-prefixed invocation + ("FOO=1 BAR=2 git push", True), # env-prefix, git twin + ("echo hi\naz repos pr create --title x", True), # multi-line command + ("echo az repos pr create", False), # mentioned, not invoked + ("FOO=1 echo az repos pr create", False), # env-prefix on a mention + ("echo 'run gh pr create later'", False), + ("az repos pr list --status active", False), # a query, not a create + ("az repos pr create-thing", False), # suffix word, not the cmd + ("az pipelines runs list", False), + ("git status", False), +] + + +@needs_sh +class TestShTriggers: + @pytest.mark.parametrize("command,must_gate", TRIGGERS, + ids=[t[0][:40] for t in TRIGGERS]) + def test_trigger_table(self, gated_repo, command, must_gate): + rc, out, err = _run_sh(gated_repo, command) + assert rc == 0, f"a hook must exit 0 either way, got {rc}; stderr={err!r}" + assert _is_deny(out) == must_gate, \ + f"{command!r}: expected {'DENY' if must_gate else 'allow'}, stdout={out!r}, stderr={err!r}" + + def test_receipts_open_the_gate_for_az(self, gated_repo): + """The az trigger uses the same receipt evidence as push/gh — receipts for HEAD allow it.""" + sha = subprocess.run(["git", "rev-parse", "--short", "HEAD"], cwd=gated_repo, + capture_output=True, text=True, check=True).stdout.strip() + receipts = gated_repo / ".claude" / ".review-receipts" + receipts.mkdir(parents=True) + for kind in ("code-review", "simplify"): + (receipts / f"{sha}.{kind}").write_text("reviewed\n", encoding="utf-8") + rc, out, err = _run_sh(gated_repo, "az repos pr create --title x") + assert rc == 0 and not _is_deny(out), \ + f"receipts present but still denied: {out!r}, stderr={err!r}" + + def test_the_deny_reason_tells_the_agent_what_to_run(self, gated_repo): + _, out, _ = _run_sh(gated_repo, "az repos pr create --title x") + reason = json.loads(out)["hookSpecificOutput"]["permissionDecisionReason"] + assert "/code-review" in reason or "code-review" in reason + assert "save-review-receipt" in reason + + def test_the_documented_bypass_still_works_for_az(self, gated_repo): + rc, out, err = _run_sh(gated_repo, "az repos pr create --title x", + {"RAILS_SKIP_REVIEW_GATE": "1"}) + assert rc == 0 and not _is_deny(out), f"stdout={out!r}, stderr={err!r}" + + +@needs_pwsh +class TestPs1Triggers: + @pytest.mark.parametrize("command,must_gate", TRIGGERS, + ids=[t[0][:40] for t in TRIGGERS]) + def test_trigger_table(self, gated_repo, command, must_gate): + rc, out, err = _run_ps1(gated_repo, command) + assert rc == 0, f"a hook must exit 0 either way, got {rc}; stderr={err!r}" + assert _is_deny(out) == must_gate, \ + f"{command!r}: expected {'DENY' if must_gate else 'allow'}, stdout={out!r}, stderr={err!r}" + + def test_receipts_open_the_gate_for_az(self, gated_repo): + sha = subprocess.run(["git", "rev-parse", "--short", "HEAD"], cwd=gated_repo, + capture_output=True, text=True, check=True).stdout.strip() + receipts = gated_repo / ".claude" / ".review-receipts" + receipts.mkdir(parents=True) + for kind in ("code-review", "simplify"): + (receipts / f"{sha}.{kind}").write_text("reviewed\n", encoding="utf-8") + rc, out, err = _run_ps1(gated_repo, "az repos pr create --title x") + assert rc == 0 and not _is_deny(out), \ + f"receipts present but still denied: {out!r}, stderr={err!r}" diff --git a/scripts/tests/test_validate_profile.py b/scripts/tests/test_validate_profile.py index dfbbb5c..e94b8e0 100644 --- a/scripts/tests/test_validate_profile.py +++ b/scripts/tests/test_validate_profile.py @@ -1,5 +1,7 @@ """Tests for validate_profile.py.""" +from pathlib import Path + import pytest from validate_profile import ( @@ -281,6 +283,40 @@ def test_coverage_minimum_true_fails(self): assert any("coverage_minimum" in e for e in errors), errors +_PROFILES_DIR = Path(__file__).resolve().parent.parent.parent / "profiles" +_ON_DISK_PROFILES = sorted(p for p in _PROFILES_DIR.glob("*/profile.yaml")) + + +class TestOnDiskProfiles: + """Every profile shipped in profiles/ must pass the real schema — new profile + directories are picked up automatically, so a future profile gets CI coverage + the moment it exists.""" + + @pytest.fixture + def real_schema(self): + from validate_profile import SCHEMA_PATH, load_yaml + return load_yaml(SCHEMA_PATH) + + def test_profiles_discovered(self): + assert _ON_DISK_PROFILES, f"no profiles found under {_PROFILES_DIR}" + + @pytest.mark.parametrize("profile_path", _ON_DISK_PROFILES, + ids=[p.parent.name for p in _ON_DISK_PROFILES]) + def test_on_disk_profile_passes_real_schema(self, profile_path, real_schema): + from validate_profile import load_yaml + errors = validate_profile(load_yaml(profile_path), real_schema) + assert errors == [], f"{profile_path.parent.name}: {errors}" + + @pytest.mark.parametrize("profile_path", _ON_DISK_PROFILES, + ids=[p.parent.name for p in _ON_DISK_PROFILES]) + def test_profile_id_matches_directory(self, profile_path): + # check_gates.py resolves compliance gates at profiles//compliance/, + # so a shipped profile whose id differs from its directory silently loses its gates. + from validate_profile import load_yaml + profile = load_yaml(profile_path) + assert profile["company"]["profile_id"] == profile_path.parent.name + + class TestSchemaStructuralLayer: """The real _schema.yaml is JSON Schema (in YAML); validate_profile runs it through jsonschema as the structural layer. Toy schemas without 'properties' skip that layer.""" diff --git a/templates/phases/close/harness-audit.md b/templates/phases/close/harness-audit.md index cc8a3a3..21a7bcc 100644 --- a/templates/phases/close/harness-audit.md +++ b/templates/phases/close/harness-audit.md @@ -9,7 +9,7 @@ - [ ] `.claude/skills/` — every skill documented and reproducible - [ ] `.claude/agents/` — grader + security-reviewer behavior understood by the client - [ ] `.claude/hooks/` — the Stop hook and any others explained -- [ ] `.github/workflows/` — all five rails; the client's DevOps can operate them +- [ ] `.github/workflows/` (GitHub) / `.azuredevops/pipelines/` (Azure DevOps) — all five rails; the client's DevOps can operate them - [ ] `infra/` — IaC understood; client can provision/destroy environments - [ ] `specs/` — convention clear; a client engineer can write a Ready spec