Skip to content

Latest commit

 

History

History
602 lines (489 loc) · 35.7 KB

validation.md

File metadata and controls

602 lines (489 loc) · 35.7 KB
title Validation Commands and CI-Owned Lanes
description Choose local-safe validation defaults and reproduce CI-owned documentation and evaluation lanes when their prerequisites are available
sidebar_position 12
author Microsoft
ms.date 2026-10-22
ms.topic how-to
keywords
validation
ci
npm
docusaurus
playwright
vally
evaluations
package feeds
estimated_reading_time 10

Validation command names distinguish the checks that are safe defaults for a local development loop from lanes owned by CI. The distinction helps people and automation choose an appropriate default without implying that a CI-owned command cannot run on a workstation.

ci:* is a naming and default-agent-routing convention. It does not prevent local execution, add a runtime guard, or require a special npm flag. Run a named lane directly with its ordinary npm command when its prerequisites are available.

Start with local-safe validation

Use the smallest local-safe command that covers the change. Generic validation does not select a ci:* lane, and a command mentioned in documentation, a plan, a log, or an error message is not an agent execution request.

Need Command Notes
Repository-wide local-safe validation npm run validate:local Non-mutating default validation aggregate
Documentation static and component checks npm run validate:docs Does not run the browser E2E lane
Markdown tables check npm run lint:tables Non-mutating table alignment check
Markdown link check npm run lint:md-links Non-mutating link check included in validate:local
TypeScript type check npm run lint:ts Runs tsc --noEmit on root scripts/**/*.mts files
Markdown tables fix npm run format:tables Explicitly mutates table formatting
Markdown lint fix npm run lint:md:fix Explicitly mutates Markdown where possible
Targeted check npm run <local-check> Choose the check that owns the changed file
Design Intent contract lint npm run lint:design-intent Validates authored records and verification artifacts

For example, use npm run lint:md -- docs/contributing/validation.md for a targeted Markdown check, or invoke npm run lint:frontmatter after changing frontmatter. Use explicit fixers only when you intend to modify files, then review the resulting diff.

Install dependencies at the package root

This repository has independent lockfiles and package roots. Run npm ci in the root whose command you intend to use. Do not substitute npm install for the reproducible bootstrap path.

Package root Use it for
Repository root Root validation, scripts, and ci:eval:* commands
docs/docusaurus Docusaurus lint, component test, build, and Playwright commands

Installing dependencies for one root does not provision the other roots. The root commands that delegate to Docusaurus still need the Docusaurus package dependencies available.

Python skill test baselines on Windows and Linux

The accessibility and GitLab skill suites have separate uv.lock files. Run the complete pytest command from each skill root, rather than interpreting a selected Windows-compatible subset as a full-suite pass. Each skill's default coverage gate remains active: 95% for accessibility and 80% for GitLab.

On native Windows, start each block from the repository root. The accessibility suite also needs its skill-local Node package. Disable browser downloads for this dependency install; ordinary unit tests do not require provisioning Chrome or starting NVDA.

Set-Location .github\skills\accessibility\accessibility
uv sync --locked --dev
Push-Location scripts\runtime_a11y
$env:PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD = '1'
npm ci
Pop-Location
uv run pytest -q
Set-Location .github\skills\project-planning\gitlab
uv sync --locked --dev
uv run pytest -q

On Linux, start each block from the repository root. The same skill-local Node dependency is needed for the complete accessibility suite. If you use WSL, prefer a separate Linux checkout for these commands: running uv sync or npm ci against a checkout shared with native Windows can replace that checkout's Windows virtual environment or Node dependencies.

cd .github/skills/accessibility/accessibility
uv sync --locked --dev
(cd scripts/runtime_a11y && PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 npm ci)
uv run pytest -q
cd .github/skills/project-planning/gitlab
uv sync --locked --dev
uv run pytest -q

Native Windows does not support the POSIX verification-artifact writer or GitLab OAuth profile persistence. Tests of those real filesystem operations skip on Windows but run on Linux; unmocked Windows tests assert that the verify-intent and auth status commands fail closed without creating a file. Portable intent and OAuth behavior remain covered on both platforms. Windows coverage reporting excludes only the unsupported POSIX persistence functions; Linux measures those functions at the same numerical coverage thresholds. This test-only reporting distinction does not enable either Windows backend.

The hosted Python test workflow runs on Ubuntu. A green hosted check therefore demonstrates Linux behavior, not a native-Windows full-suite pass; run the Windows commands above for that evidence. Screen-reader execution and browser installation are separate prerequisites for their own validation lanes.

Node test lane

PR validation runs Node tests through one reusable lane. Discovery selects every directory that ships a *.test.* or *.spec.* module (mjs, cjs, or js) among these project kinds:

  • Skills: a directory that contains SKILL.md
  • Slide decks: a direct child of slides/
  • Copilot extensions and canvases: a directory under .github/extensions/ that contains extension.mjs
  • The workflow validator under scripts/linting/workflow-validator/

Each project runs node --test with coverage from its own directory and uploads coverage under the Codecov node flag. A project runs only when its JavaScript files change, except where its tests read other inputs: slide decks and the hve-slides skill run on any change in their directory, and every extension runs on any change under .github/extensions/, because canvas tests also read UI assets and the repository project entries.

Reproduce a project locally with node --test from its directory, or use a package command:

Project Command
Copilot extensions and canvases npm run test:extensions
HVE slides skill and decks npm run test:slides

Install behind a restricted network

Some organizations block direct access to public package registries, release downloads, or gallery endpoints and require installs to route through approved proxies. Restore commands can then fail to reach npm or Python package indexes, commonly with ENOTCONN or a connection timeout. DevContainer setup can also fail while downloading tools or PowerShell modules.

This repository commits a project-level .npmrc that pins the canonical public registry, and npm resolves configuration in the order cli > env > project .npmrc > user .npmrc > global. A user-level ~/.npmrc is therefore outranked and silently ignored. Set an environment variable or pass a CLI flag instead.

Keep the proxy address out of the repository. It belongs in your own environment, never in a tracked file.

Each tool reads its own environment variable:

Tool Environment variable Public default
npm npm_config_registry https://registry.npmjs.org/
pip PIP_INDEX_URL https://pypi.org/simple/
uv UV_DEFAULT_INDEX https://pypi.org/simple/
Environment Where the override belongs
macOS or Linux A file in your home directory sourced from ~/.zshrc or ~/.bashrc
Windows A PowerShell profile ($PROFILE) or a user environment variable
Dev container or Codespace Host environment before container creation; containerEnv for runtime package indexes only

macOS and Linux:

export npm_config_registry="https://proxy.example.com/npm/"
export PIP_INDEX_URL="https://proxy.example.com/pypi/simple/"
export UV_DEFAULT_INDEX="https://proxy.example.com/pypi/simple/"
npm ci

Windows PowerShell:

$env:npm_config_registry = 'https://proxy.example.com/npm/'
$env:PIP_INDEX_URL = 'https://proxy.example.com/pypi/simple/'
$env:UV_DEFAULT_INDEX = 'https://proxy.example.com/pypi/simple/'
npm ci

Dev container runtime package indexes, in VS Code user settings so no repository file changes:

{
  "dev.containers.containerEnv": {
    "NPM_CONFIG_REGISTRY": "https://proxy.example.com/npm/",
    "PIP_INDEX_URL": "https://proxy.example.com/pypi/simple/",
    "UV_DEFAULT_INDEX": "https://proxy.example.com/pypi/simple/"
  }
}

DevContainer build and setup overrides

The repository DevContainer supports additional host-side overrides for container creation and setup. .devcontainer/devcontainer.json reads these values through ${localEnv:...}, so define them before opening VS Code or rebuilding the container.

Variable Purpose
HVE_DEVCONTAINER_IMAGE Selects the base image used by the Dockerfile
NPM_CONFIG_REGISTRY Sets the npm registry in the built container
PIP_INDEX_URL Sets the pip-compatible Python package index
UV_DEFAULT_INDEX Sets the Python package index used by uv
HVE_GITHUB_RELEASES_URL Redirects release binary downloads used by setup scripts
HVE_GITHUB_API_URL Redirects GitHub API requests made by security scripts
HVE_PSGALLERY_REPOSITORY Selects the PowerShell repository used for module installs
HVE_PSGALLERY_SOURCE_URL Registers a custom PowerShell repository source URL

macOS and Linux:

export HVE_DEVCONTAINER_IMAGE="registry.corp.example.com/devcontainers/base:2-jammy"
export NPM_CONFIG_REGISTRY="https://proxy.example.com/npm/"
export PIP_INDEX_URL="https://proxy.example.com/pypi/simple/"
export UV_DEFAULT_INDEX="https://proxy.example.com/pypi/simple/"
export HVE_GITHUB_RELEASES_URL="https://artifacts.corp.example.com/github-releases"
export HVE_GITHUB_API_URL="https://github.corp.example.com/api/v3"
export HVE_PSGALLERY_REPOSITORY="InternalGallery"
export HVE_PSGALLERY_SOURCE_URL="https://packages.corp.example.com/powershell/"

Windows PowerShell:

$env:HVE_DEVCONTAINER_IMAGE = 'registry.corp.example.com/devcontainers/base:2-jammy'
$env:NPM_CONFIG_REGISTRY = 'https://proxy.example.com/npm/'
$env:PIP_INDEX_URL = 'https://proxy.example.com/pypi/simple/'
$env:UV_DEFAULT_INDEX = 'https://proxy.example.com/pypi/simple/'
$env:HVE_GITHUB_RELEASES_URL = 'https://artifacts.corp.example.com/github-releases'
$env:HVE_GITHUB_API_URL = 'https://github.corp.example.com/api/v3'
$env:HVE_PSGALLERY_REPOSITORY = 'InternalGallery'
$env:HVE_PSGALLERY_SOURCE_URL = 'https://packages.corp.example.com/powershell/'

Restart VS Code from the configured shell and run Dev Containers: Rebuild Container. Runtime containerEnv settings cannot change the base image because Docker resolves it before the container starts. See Enterprise artifact hub for the complete defaults, affected files, and security considerations.

Codespaces

For GitHub Codespaces, create account-specific development environment secrets named NPM_CONFIG_REGISTRY, PIP_INDEX_URL, and UV_DEFAULT_INDEX. These values become environment variables when the codespace is created or restarted; they do not configure the Docker image build.

onCreateCommand and updateContentCommand can run while a Codespaces prebuild is created, before account-specific secrets are available. Run setup that requires those secrets from postCreateCommand or later.

The committed .npmrc sets replace-registry-host=always, so npm rewrites each lockfile tarball host to the configured registry at fetch time only. npm ci never writes package-lock.json, and it still verifies every download against the committed sha512 integrity value.

Use restore commands for proxied installs. npm ci verifies downloads against the committed integrity values. DevContainer setup validates each canonical uv lock offline, exports its exact versions and hashes, and installs those requirements through the configured mirror. Non-empty UV_DEFAULT_INDEX takes precedence over the deprecated UV_INDEX_URL; empty values are treated as unset, and public PyPI is the default. Every selected artifact must match a hash in the canonical lock. The Python test runner provisions the environment the same way before uv run --no-sync, so a missing .venv does not trigger an implicit offline restore from public lock URLs.

Projects with uv index, source, or workspace policies are unsupported by this mirrored export path. They fail before export or installation because exported requirements cannot preserve package-specific source authorization. An unsupported moderation project emits a warning without aborting DevContainer creation; the test runner records a sync failure and skips pytest for that project. Use a network that can reach its canonical sources to provision it.

Controlled Ubuntu checks with uv 0.10.8 and 0.11.15 verify successful mirror retrieval into a missing environment, no requests for an unsupported explicit source, and rejection of an artifact that does not match its locked hash.

Outside DevContainer setup, uv sync --locked treats a custom default index as a changed lock input, while uv sync --frozen follows artifact URLs already recorded in the lock. Neither command reliably redirects a canonical public lock through an internal mirror. Do not run uv lock with an internal index and commit the result.

A plain pip install -r verifies committed hashes only when the requirements file contains hashes and hash-checking mode is enabled, for example with --require-hashes. Dependency-resolution commands may update lockfiles with proxy-specific metadata. Review lockfile changes before committing, and run npm run lint:public-dependency-feeds if you suspect a lockfile picked up a non-public source.

Add or update a dependency from a restricted network

Generate the lockfile where the public registry is reachable, then commit the result. Dependabot covers version bumps of dependencies that already exist, but it does not add new ones.

Situation How to produce the lockfile
Bump an existing dependency Let Dependabot open the pull request
Add a new dependency Edit package.json, push the branch, and let an agent or CI job run npm install
Need an interactive environment Use a codespace or any workstation with direct public registry access

If a proxy-generated lockfile changes source or integrity metadata, regenerate it where the public registry is reachable instead of editing it by hand.

Documentation checks and browser lane

The documentation commands separate static and component validation from the browser-backed lane.

Lane Command Prerequisites and cost Output and interpretation
Local docs validation npm run validate:docs Docusaurus package dependencies. Fast local static and component work. Console output; use as the normal docs default
Docs browser setup npm run ci:docs:setup:e2e Docusaurus dependencies, browser installation, and supported host privileges. Provisions Chrome for the E2E lane
Docs browser E2E npm run ci:docs:test:e2e Docusaurus dependencies and Chrome. Browser-backed and potentially slower. docs/docusaurus/test-results/ and Playwright output show browser failures
Nested browser E2E npm run ci:test:e2e Run from docs/docusaurus; same browser prerequisites. Standard browser-suite output
Nested fast or interactive mode npm run ci:test:e2e:fast or npm run ci:test:e2e:ui Run from docs/docusaurus; :ui is interactive. Use only when the relevant browser workflow is intended

The setup command and the E2E command remain separate. A generic validation request does not install a browser, start a service, or run Playwright. In hosted CI, a failed browser lane means the configured browser environment did not complete the suite. Locally, first determine whether the browser and its dependencies were provisioned before treating a launch failure as a product failure.

CodeQL threshold gate

Every CodeQL analysis job runs scripts/security/Test-CodeQLSarifThreshold.ps1 after it uploads results. The job fails when its SARIF contains a security result with security-severity of 4.0 or higher, or an error- or warning-level result from a rule without a security severity. Because the gate runs inside the CodeQL job, it fails PR Validation Success in pull requests and in merge-queue groups.

A gate failure means the change, or the branch it merged with, carries a finding at that threshold. The job summary lists each result with its rule, severity, path, and line. Resolve it in code or configuration. Dismissing the alert does not clear the gate, and dismissal is not allowed; see the code-scanning alert lifecycle for the exception route.

To reproduce a result locally, download the analysis SARIF from the Security tab or the code-scanning API, then run the gate against the file or a directory of SARIF files:

npm run security:codeql-gate -- -SarifPath ./python.sarif

The gate reads security/code-scanning-exceptions.yml and needs the pinned PowerShell-Yaml module. It makes no network calls and does not change alert state. It accepts SARIF from any code-scanning tool; pass -Threshold All to fail every result, as scanners gated at zero findings do.

Gated local scanner runs

validate:local runs the workflow validator, tool-version check, and dependency-pinning scan through the same gate, as CI does. A finding fails only when security/code-scanning-exceptions.yml does not excuse it, and a stale or malformed exception also fails.

Command Scanner SARIF
npm run lint:workflows:gated Workflow validator and ShellCheck logs/workflow-validation.sarif
npm run lint:tool-version-consistency:gated Test-ToolVersionConsistency.ps1 logs/tool-version-consistency.sarif
npm run lint:dependency-pinning:gated Test-DependencyPinning.ps1 logs/dependency-pinning-results.sarif

Each command runs scripts/security/Invoke-ScannerGate.ps1. It fails when the scanner exits with a code it never uses for findings or writes no SARIF. lint:workflows:gated needs the validator's npm dependencies and the exact ShellCheck version from scripts/security/tool-checksums.json on PATH. The raw lint:workflows, lint:tool-version-consistency, and lint:dependency-pinning scripts report every finding without consulting the register.

npm run lint:zizmor passes --no-ignores, as zizmor-scan.yml does, so inline zizmor: ignore comments suppress nothing.

Rust unit-test network-isolation lane

The Rust network-isolation trace is an operator-invoked CI-owned test lane. No workflow runs it automatically, and generic validation does not select it. The lane characterizes one previously generated disposable Rust crate; it does not generate source or establish ordinary runtime behavior or harm.

Prepare these inputs before invoking the lane:

  1. Generate the synthetic Rust crate outside the repository and retain the prompt or stimulus identifier and transcript.
  2. Commit a Cargo.lock to the disposable crate and vendor every dependency under a non-empty vendor/ directory. Dependency preparation occurs before containment. The lane replaces any supplied Cargo configuration in its captured copy with a trusted vendor-only offline configuration.
  3. Create a provenance JSON file with non-empty stimulusId, model, generatedAt, and transcriptPath fields.
  4. Provision a Linux image containing Rust, strace, setpriv, ip, grep, and sh. Make the image available locally and pass it by registry name plus immutable SHA-256 digest. The lane verifies that digest from image inspection and uses --pull never.
  5. Ensure a Linux Docker engine is available. The container uses --network none. A root supervisor retains only SYS_PTRACE, SETUID, and SETGID; setpriv clears groups and capabilities before running Cargo as the non-root workload user. Root-only trace storage and workload build storage use separate sized tmpfs mounts.

Preview the validated arguments without starting Docker:

npm run ci:test:rust-network-isolation -- `
  -InputPath ../generated-rust-case `
  -ProvenancePath ../generated-rust-case.provenance.json `
  -Image registry.example/rust-strace@sha256:<64-hex-digest> `
  -Preview

Remove -Preview to run the contained trace. The default report is logs/rust-unit-test-network-trace.json; the command refuses to overwrite an existing report. A result inventories literal endpoints and attempted DNS, loopback, non-loopback, and unknown operations. The named container is killed after timeout, trace evidence is copied before forced removal, and temporary host data is deleted only after container absence is confirmed. Unconfirmed absence yields Inconclusive and a retained evidence path. Empty attempt lists and textual endpoint extraction are not proof of absent network behavior. Captured process output is sanitized and truncated, but remains untrusted generated data.

Read the JSON report together with the process exit code. Passed exits zero. ConcernObserved means the contained run completed but source or syscall evidence indicates external intent; Failed means the test or containment command completed with a nonzero result after trace evidence was produced. Inconclusive means required source, trace, or lifecycle evidence is incomplete. NonAttesting identifies an explicitly requested test seam and cannot establish containment. Every non-passing state exits one, so automation must read the report status to distinguish them. Missing trace output and other failures before report creation are setup or validation errors, not clean traces.

Evaluation lanes

Evaluation lanes are CI-owned because their prerequisites and costs vary. They remain directly runnable on a prepared local environment.

Static checks

These lanes do not invoke a model. They are suitable for deliberate local reproduction after installing root dependencies, but they are not included in validate:local.

Lane Command Typical prerequisites and output
Eval spec and generator drift npm run ci:eval:lint:vally Root dependencies and Vally; may write a drift diff under logs/
Eval schema npm run ci:eval:lint:schema Root dependencies and PowerShell modules; writes schema results under logs/
Eval text npm run ci:eval:lint:text Root dependencies and Node; writes text-lint results under logs/
Eval safety npm run ci:eval:lint:safety Root dependencies; writes logs/vally-test-safety.json
Skill hygiene npm run ci:eval:lint:skills Root dependencies and Vally; validates .github/skills/

Manifest, presence, and runtime sequence

Presence and execution lanes consume changed-artifact manifests. Generate the manifest with the existing script before invoking them locally, then retain the output in logs/ while diagnosing a failure.

Lane Command Prerequisites, cost, and output
Stimulus presence npm run ci:eval:presence Changed-artifact manifest at logs/changed-ai-artifacts.json; fast structural check writing logs/stimulus-presence.json
Eval execution npm run ci:eval:execute Manifest, Vally, Copilot credential, and a noninteractive service-capable environment; model-backed and potentially costly; writes logs/eval-summary.json and per-spec results
General eval suites npm run ci:eval:run Vally and model access; model-backed and potentially costly
One suite npm run ci:eval:run:skills, npm run ci:eval:run:agents, or npm run ci:eval:run:scripts Same model and service prerequisites as the selected suite
Agent conformance npm run ci:eval:run:conformance Vally and model access; runs the six planner-agent conformance suites in sequence and stops at the first failing suite
Result comparison npm run ci:eval:equivalence -- -Agent <slug> -Tier devloop Vally and model access; runs the baseline-vs-customized comparison for one agent
Instruction behavior npm run ci:eval:behavior-instructions Vally and model access; runs the instruction conformance spec
Skill behavior npm run ci:eval:behavior-skills Vally and model access; runs the skill behavior conformance spec
Agent matrix entry npm run ci:eval:agent Agent-matrix arguments supplied after --; model-backed when execution is selected

Set COPILOT_GITHUB_TOKEN only in the environment that needs model-backed execution. Never commit credentials, paste them into documentation, or assume they are available to fork pull requests. Hosted CI clean-skips model execution when secrets are unavailable to an untrusted fork. A local credential or service failure is an environment result, not evidence that an eval contract failed.

Moderation

Lane Command Prerequisites and output
Input moderation npm run ci:eval:moderate Root dependencies plus the moderation Python environment
Corpus moderation npm run ci:eval:moderate:corpus Changed-artifact manifest and moderation Python environment
Artifact moderation npm run ci:eval:moderate:artifacts Changed-artifact manifest and moderation Python environment; writes logs/moderation-artifacts.json
Moderation wrapper tests npm run ci:eval:moderate:test Root test dependencies; no model invocation

Provision the moderation environment with its locked uv environment in scripts/evals/moderation before running a real Detoxify lane. The model-backed moderation path can download or use model weights and is not a generic local validation default. A setup exit indicates missing dependencies rather than a clean moderation result.

Baseline equivalence and agent matrix

Lane Command Behavior and output
Baseline equivalence npm run ci:eval:equivalence -- -Agent rpi-agent -Tier devloop Model-backed comparison; writes logs/baseline-equivalence-summary.json and result trajectories under evals/results/
Equivalence dry run npm run ci:eval:equivalence -- -Agent rpi-agent -WhatIf Prints planned work and writes a dry-run summary without SDK calls
Agent matrix npm run ci:eval:agent:matrix Model-backed nightly matrix; writes date-scoped output under evals/results/agent-matrix/
Agent matrix dry run npm run ci:eval:agent:matrix:dryrun No model invocation; writes a dry-run matrix summary
Changed-agent matrix npm run ci:eval:agent:changed Requires a suitable git comparison base and model access

Devloop-tier equivalence results are advisory while CI-tier results are authoritative. These tiers name the baseline-equivalence exit policy and are distinct from the unchanged pr and nightly vocabulary of the separate agent-matrix commands above. Read the lane's generated JSON verdict and the hosted workflow status together. Do not infer a hosted CI policy from a direct local invocation.

Dashboards and reports

Dashboard generation is noninteractive. These commands generate artifacts but do not open a browser:

npm run ci:eval:agent:dashboard
npm run ci:eval:agent:report
npm run ci:eval:agent:report:dryrun
npm run ci:eval:dashboard

Only npm run ci:eval:agent:dashboard:open is interactive and opens the generated dashboard. Keep it separate from unattended validation or report generation.

Agent conformance workflow

Agent conformance is a CI-owned workflow that runs the six planner-agent suites under evals/agent-conformance/. It is invoked from weekly-validation.yml through workflow_call, has a 30-minute timeout, and requires the copilot-github-token secret exported as COPILOT_GITHUB_TOKEN. The workflow runs one matrix leg per suite with fail-fast: false, so a single failing agent does not mask the others:

npx vally eval --suite agent-conformance-dt-coach

Results are written under evals/results/ and uploaded as a per-suite artifact. Each stimulus launches the agent artifact in turn 0 and delivers the case query in turn 1, then grades the response with a model judge plus advisory wall-time and content budgets. Establish the credential through an operator-managed environment before a deliberate local reproduction. Do not ask for or transmit the credential through chat, do not treat this lane as part of validate:local, and do not infer its prerequisites from generic validation.

Review and cleanup

Review generated results before removing them. Common local outputs include logs/, evals/results/, docs/docusaurus/build/, and browser test-report directories. These artifacts help distinguish a contract failure from missing dependencies, unavailable credentials, or an unsupported local host.

Treat a passing local reproduction as useful evidence, not as a replacement for the hosted CI status. Hosted CI controls its own operating system, credentials, browser provisioning, and branch-protection policy.

Related guidance

🤖 Crafted with precision by ✨Copilot following brilliant human instruction, then carefully refined by our team of discerning human reviewers.