diff --git a/.agents/skills/switchyard-docs/SKILL.md b/.agents/skills/switchyard-docs/SKILL.md index 731b1f7c4..00fe8b5db 100644 --- a/.agents/skills/switchyard-docs/SKILL.md +++ b/.agents/skills/switchyard-docs/SKILL.md @@ -1,46 +1,191 @@ --- name: switchyard-docs -description: Edit or debug the published Switchyard MkDocs site. Use for docs pages, mkdocs.yml navigation, mkdocs_hooks.py source links, strict build failures, local previews, or .github/workflows/docs.yml. +description: Use when adding or editing Switchyard docs or contributor guidance under `docs/`, changing Fern configuration or navigation, debugging `fern check`, previewing or publishing docs, or reviewing the Fern GitHub Actions workflows. --- -# Switchyard Documentation +# Switchyard Fern Docs -The published site is the subset of `docs/` selected by `mkdocs.yml`. Strict MkDocs warnings fail -CI, so fix warnings rather than weakening validation. +## Overview -## Source Of Truth +Published documentation is authored as Markdown at the top level of `docs/`. Fern infrastructure lives +under `docs/fern/`; do not put nightly content under `docs/fern/versions/nightly/pages/`. -| Concern | File | +`docs/README.md` is the contributor entry point. Scoped instructions live in `docs/AGENTS.md` and +`docs/fern/AGENTS.md`; neither guidance file is published. Keep those files synchronized with this +skill when the authoring, validation, redirect, or publishing workflow changes. + +The public page set, order, and labels come from `docs/fern/versions/nightly.yml`. Markdown files +under `docs/internal/` are intentionally unpublished design and operations notes. Do not add them to +the Fern navigation unless the content is deliberately being promoted to public documentation. + +`fern check` is the contract. Run it locally through `docs/Makefile`; CI runs the same CLI version +pinned in `docs/fern/fern.config.json`. + +## Quick Reference + +| Situation | Command | |---|---| -| Navigation, exclusions, theme | `mkdocs.yml` | -| Source-link rewriting | `mkdocs_hooks.py` | -| Local commands | `docs/Makefile` | -| Build, preview, deployment | `.github/workflows/docs.yml` | -| Documentation dependencies | `pyproject.toml` `docs` group | - -## Workflow - -1. Decide whether a new page is public. Public pages go in `nav`; internal notes go in - `exclude_docs`. -2. Match the surrounding file naming and documentation style. -3. Use relative links. For repository files outside `docs/`, use paths relative to the Markdown - file and let `mkdocs_hooks.py` produce the source URL. -4. Verify public examples use supported public imports and current CLI syntax. -5. Run the strict build: - -```bash -cd docs -make publish -``` - -## CI Constraints - -- Keep the docs workflow path-filtered. -- Keep default permissions read-only and grant write access only to deployment jobs. -- Keep PR previews limited to same-repository pull requests. -- Do not use `pull_request_target` to run untrusted documentation code with write permissions. -- Build once and pass the `site/` artifact to preview and deployment jobs. -- Preserve `keep_files: true` so a main deployment does not remove PR previews. - -Common strict-build fixes are direct: add or remove a missing nav entry, repair the relative link, -or explicitly exclude an internal page. Do not disable strict mode. +| Validate the site | `cd docs && make check` | +| Test routes and redirects | `uv run pytest tests/test_fern_docs.py -v` | +| Run a local preview | `cd docs && make preview` | +| Remove local Fern artifacts | `cd docs && make clean` | +| List published pages | `sed -n '1,240p' docs/fern/versions/nightly.yml` | +| Find unpublished notes | `find docs/internal -type f -name '*.md' -print` | +| Check Markdown safety | `cd docs && make check` | +| Check internal Fern links | `uv run pytest tests/test_fern_docs.py::test_internal_mdx_links_resolve_to_navigation_routes -v` | + +## Where Things Live + +- **Published content** → `docs/**/*.md`, excluding `docs/fern/`. +- **Unpublished notes** → `docs/internal/**/*.md`. +- **Docs contributor guide and scoped instructions** → `docs/README.md` and `docs/AGENTS.md`. +- **Fern infrastructure guide and scoped instructions** → `docs/fern/README.md` and + `docs/fern/AGENTS.md`. +- **Site and product configuration** → `docs/fern/docs.yml`. +- **Nightly navigation** → `docs/fern/versions/nightly.yml`. +- **Fern CLI pin** → `docs/fern/fern.config.json`. +- **Local commands** → `docs/Makefile`. +- **CI validation** → `.github/workflows/fern-docs-ci.yml`. +- **PR preview collection** → `.github/workflows/fern-docs-preview-build.yml`. +- **PR preview publishing and comment** → `.github/workflows/fern-docs-preview-comment.yml`. +- **Release publishing** → `.github/workflows/publish-fern-docs.yml`. +- **GitHub Pages redirect generator** → `docs/fern/generate_legacy_redirect_site.py`. +- **Route/redirect regression tests** → `tests/test_fern_docs.py`. +- **Shared NVIDIA presentation** → `global-theme: nvidia`; theme assets are not vendored here. + +## Adding a Published Page + +1. Create `docs/
/.md` with concise frontmatter, including `title` and `position`. +2. Add it to `docs/fern/versions/nightly.yml` in the intended sidebar location. Preserve existing + labels and order unless the change explicitly redesigns navigation. +3. Point `path:` from the version file back to the authored page with `../../`, for example: + + ```yaml + - page: Health-aware Routing + path: ../../routing_algorithms/health_aware_routing.md + ``` + +4. Use version-agnostic Fern URLs for published-page links. Fern URLs derive from the slugified + `page:` and `section:` labels, not filenames. For example, `page: Core Concepts` is + `/concepts/core-concepts` inside every version. +5. Link repository files outside `docs/` with an absolute GitHub URL. They are not part of Fern's + content tree. +6. Import examples from `switchyard`, not `switchyard.lib.*`; public examples must use exports from + `switchyard/__init__.py`. +7. Run `cd docs && make check` before pushing. + +## Editing Navigation + +- Treat `docs/fern/versions/nightly.yml` as the source of truth for public scope and order. +- Use `section:` for labeled groups with nested `contents:`. Do not combine `folder:` with + `contents:`; Fern rejects that schema. +- Keep Switchyard sections label-only. Do not add a `path:` to a section and then repeat that same + file as a child page; Fern publishes the section landing route instead of the intended child + route. Every authored page in this site belongs under `contents:` and resolves as + `/section/page`. +- Keep the first navigation entry as the home page. +- Paths in nightly navigation point to `../../*.md`. Frozen releases, when added, own independent + copies under `docs/fern/versions//pages/`. +- Do not add `latest.yml` as a copy of nightly. A future `latest` alias must point at a frozen GA + tree. + +## Links and MDX + +- MDX treats bare angle-bracket text as JSX. Escape comparisons such as `<100ms` as `<100ms` + and wrap placeholders such as `` in backticks. +- HTML void tags must self-close, for example ``. +- Use MDX comments (`{/* note */}`), not HTML comments. +- Built-in Fern components do not require imports. Add product-specific components only when a page + needs them, then register their directory in `docs/fern/docs.yml`. +- Do not link unpublished `docs/internal/` notes from published pages. +- Preserve titled MkDocs admonitions with the Fern callout `title` prop, for example + ``. +- Give every published page a non-empty `description` in frontmatter. + +## Redirects + +- Use full `/nemo/switchyard/...` paths in both `source` and `destination`. +- Add exact mappings when old filenames and new navigation slugs differ. A generic `index.html` + catch-all cannot translate underscores or changed section hierarchy. +- Cover both the directory route and its generated `index.html` form for every former MkDocs page. +- Keep `docs/fern/generate_legacy_redirect_site.py` synchronized with `docs/fern/docs.yml`. The + generator owns redirects on `nvidia-nemo.github.io/Switchyard`; Fern redirects only own requests + that already reach the Fern domain. +- Run `uv run pytest tests/test_fern_docs.py -v` after changing navigation, links, or redirects. + +## CI Hygiene + +The Fern workflows follow the same split as NeMo Curator: + +- `fern-docs-ci.yml` is a reusable `fern check` workflow called by the repository-wide CI. Its job + is included in the required `CI Success` aggregate, so Fern failures cannot merge as an optional + check. +- `fern-docs-preview-build.yml` runs only for same-repository PR branches, uses no secrets, and + uploads the complete `docs/` tree plus a changed-Markdown hint. Fork PRs stop after required Fern + validation. The complete tree is required because nightly navigation under `docs/fern/` + references pages through `../../` paths. It uses PR-numbered concurrency and cancels + superseded runs. +- `fern-docs-preview-comment.yml` runs through `workflow_run`, downloads the collector artifact, + rejects fork-originated runs, derives the target PR from the trusted event, reads the Fern version + from a separate default-branch checkout, and generates a PR-numbered preview with + `DOCS_FERN_TOKEN`. Commenting runs in a separate job with only pull-request write permission. It + must never check out the PR branch, run PR-provided scripts, or trust artifact metadata for the + target PR/tool version. +- `publish-fern-docs.yml` publishes only for `docs/v*` tags or manual dispatch, then replaces the + old GitHub Pages content with static redirects to the custom Fern domain. The workflow verifies + the custom-domain home page first; if it remains unavailable, GitHub Pages stays untouched. + Publishing is serialized, and the repository-write redirect job never receives the Fern token. + A merge to `main` validates the site but does not publish either destination. + +Every workflow installs the exact Fern CLI version from `docs/fern/fern.config.json` and runs Fern +from `docs/fern/`. The privileged preview reads that pin from the trusted default branch rather +than the artifact. Preview generation and publishing require the `DOCS_FERN_TOKEN` organization +secret. All Fern workflow actions use immutable commit pins and supported Node LTS; secret-bearing +jobs disable package-manager caching and validate the Fern version before installing it. Checkout +steps do not persist credentials. Do not collapse the collector and trusted preview phases or use +`pull_request_target`. + +## Failure Map + +| Symptom | Fix | +|---|---| +| `Path does not exist` | Correct the `../../` path in `versions/nightly.yml` or restore the referenced page file. | +| Navigation object does not match schema | Use `section` with `contents`, or a standalone `folder`; do not combine `folder` and `contents`. | +| Unsupported JSX tag | Replace it with a built-in Fern component or register the product component directory. | +| MDX parse error near `<` | Escape prose comparisons/placeholders or self-close the HTML element. | +| Link works on GitHub but not Fern | Rewrite it to the canonical Fern URL or an absolute GitHub URL for repository-only files. | +| Child page returns 404 but section works | Remove the duplicate section `path`; keep the authored page only under `contents`. | +| Legacy redirect lands on another 404 | Add an exact old-path → current-slug mapping before broad redirect patterns. | +| `global-theme` authentication failure | Verify NVIDIA Fern authentication; do not vendor a local copy of the shared theme. | + +## Anti-Patterns + +- Adding unpublished design notes to navigation because they happen to live under `docs/`. +- Adding `README.md` or `AGENTS.md` contributor guidance to published navigation. +- Recreating `docs/fern/versions/nightly/pages/`; nightly content is authored at `docs/*.md`. +- Deriving links from filenames instead of the navigation title/slug. +- Assigning the same page file to a section `path` and one of its child pages. +- Hardcoding `/nightly/` in shared page links. +- Vendoring NVIDIA theme CSS, logos, favicon, or footer components. +- Reintroducing MkDocs configuration, hooks, or a parallel docs build. +- Running a secret-bearing Fern preview directly from a `pull_request` job. +- Publishing production docs on every merge to `main` instead of a docs tag or manual dispatch. + +## References + +- `docs/fern/docs.yml` — Fern instance, theme, versions, and redirects. +- `docs/README.md` — contributor entry point for docs content. +- `docs/AGENTS.md` — scoped instructions for the full docs tree. +- `docs/fern/README.md` — Fern infrastructure and CI guide. +- `docs/fern/AGENTS.md` — scoped instructions for Fern configuration and redirects. +- `docs/fern/versions/nightly.yml` — public navigation and authored-page paths. +- `docs/fern/fern.config.json` — Fern CLI pin. +- `docs/Makefile` — local check and preview commands. +- `docs/fern/generate_legacy_redirect_site.py` — redirect-only GitHub Pages output. +- `tests/test_fern_docs.py` — navigation, link, metadata, and redirect invariants. +- `.github/workflows/fern-docs-ci.yml` — CI-equivalent validation. +- `.github/workflows/fern-docs-preview-build.yml` — same-repository preview source collection. +- `.github/workflows/fern-docs-preview-comment.yml` — trusted preview generation and PR comment. +- `.github/workflows/publish-fern-docs.yml` — tag/manual production publishing. +- [`switchyard-testing-ci`](../switchyard-testing-ci/SKILL.md) — broader repository validation. +- [`switchyard-codebase-exploration`](../switchyard-codebase-exploration/SKILL.md) — impact mapping for docs that cite code or APIs. diff --git a/.agents/skills/switchyard-testing-ci/SKILL.md b/.agents/skills/switchyard-testing-ci/SKILL.md index 9aed5cc4b..38a5bacae 100644 --- a/.agents/skills/switchyard-testing-ci/SKILL.md +++ b/.agents/skills/switchyard-testing-ci/SKILL.md @@ -1,80 +1,322 @@ --- name: switchyard-testing-ci -description: Select or debug Switchyard validation and GitHub CI. Use when asked which tests to run, whether a change is ready, why a CI job failed, how to reproduce CI, or whether live provider tests are appropriate. +description: Use when validating a Switchyard change, preparing or reviewing a PR, debugging CI failures (ruff, mypy, SPDX, pytest, slim-install), choosing which local tests to run, or diagnosing dependency, optional-extra, stale-name, CLI, server, translation, routing, stats, or live e2e failures. Triggers on phrases like "is this ready to merge", "CI is failing", "which tests should I run", or "how do I reproduce this locally". --- -# Switchyard Testing And CI +# Switchyard Testing and CI -Read `.github/workflows/ci.yml` before claiming local CI equivalence. Select tests from the current -diff and symbol references rather than a static subsystem inventory. +## Overview -## Validation Matrix +**Pick the smallest local gate that is genuinely equivalent to what CI will run, then map failures +to targeted fixes.** The two goals — move fast locally, never report a green run weaker than the CI +gate you are claiming — usually conflict; this skill resolves that conflict from the current diff +rather than from memory. -| Change | Minimum trustworthy validation | +If your change also touches code you have not yet read, run [`switchyard-codebase-exploration`](../switchyard-codebase-exploration/SKILL.md) first so the validation set covers the real impact, not just the diff surface. + +Rust crates under `crates/` (`switchyard-core`, `switchyard-translation`, +`switchyard-components`, `switchyard-server`, `switchyard-py`) are active on the Rust core branch. +For Rust-touching diffs, include `cargo fmt --all --check`, `cargo clippy --workspace +--all-targets -- -D warnings`, and the smallest trustworthy `cargo test` scope; +use `cargo test --workspace` before calling a broad Rust MR ready. + +## Quick Reference + +| Situation | Command | |---|---| -| Markdown or skill Markdown only | Parse skill frontmatter when applicable; `git diff --check` | -| Python implementation | Focused pytest, `uv run ruff check .`, `uv run mypy switchyard` | -| Broad Python or public API | Hermetic full pytest plus lint and mypy | -| One Rust crate | `cargo fmt --all --check`, workspace clippy, focused crate tests | -| Shared Rust/API/FFI | Rust workspace tests plus affected Python tests | -| Packaging, extras, top-level imports | The slim-install smoke commands from the current CI workflow | -| Live provider behavior | Focused integration test only after explicit authorization | +| Map the current diff | `git status -sb && git diff --stat && git diff --name-only` | +| Pre-PR hermetic gate (default) | `uv run ruff check . && uv run mypy switchyard && env -u OPENROUTER_API_KEY -u NVIDIA_API_KEY -u OPENAI_API_KEY -u ANTHROPIC_API_KEY uv run pytest tests/ -v -m "not integration"` | +| Mirror CI pytest with no live creds | `env -u OPENROUTER_API_KEY -u NVIDIA_API_KEY -u OPENAI_API_KEY -u ANTHROPIC_API_KEY uv run pytest tests/ -v -m "not integration"` | +| Rust component crate change | `cargo fmt --all --check && cargo clippy --workspace --all-targets -- -D warnings && cargo test -p switchyard-components` | +| Rust server crate change | `cargo fmt --all --check && cargo clippy --workspace --all-targets -- -D warnings && cargo test -p switchyard-server` | +| Broad Rust crate change | `cargo fmt --all --check && cargo clippy --workspace --all-targets -- -D warnings && cargo test --workspace` | +| Docs/Fern workflow change | `make -C docs check && uv run pytest tests/test_fern_docs.py -v -o addopts=` | +| Select focused validation from uncommitted paths | `python scripts/select_validation.py --changed` | +| Select validation for committed branch work | `python scripts/select_validation.py --base origin/main` | +| Any GitHub Actions workflow change | Run the selector, then `actionlint && zizmor --pedantic .github/workflows`; review `scripts/ci/README.md` | +| Slim-install regression guard | see [Slim-install smoke gate](#slim-install-smoke-gate) | +| Live e2e (only on explicit user request) | `NVIDIA_API_KEY=… uv run pytest tests/e2e/ -v -m integration -o addopts= --maxfail=10` | +| Skill/docs change only | YAML frontmatter check + `git diff --check` (see [Skill/docs-only gate](#skilldocs-only-gate)) | + +## Diff-Based Selection First + +From the repo root, inspect the diff and map each changed area to the focused gates below: + +```bash +cd "$(git rev-parse --show-toplevel)" +git status -sb +git diff --stat +``` + +Use the current diff as the starting plan, then add any tests revealed by code search or by the +failure map in this skill. + +## CI Gates and Local Equivalence + +The hard GitHub Actions gates live in `.github/workflows/ci.yml`: -## Standard Gates +- `uv run ruff check .` +- SPDX header check for every Python file found by CI, including `.agents/skills/**/scripts/*.py` +- `uv run pytest tests/ -v -m "not integration"` on Python 3.12 through 3.14 +- reusable Fern validation (`fern check`) called from `.github/workflows/ci.yml` and included in + the required `CI Success` aggregate +- Rust workspace gate: `cargo fmt --all --check`, `cargo clippy --workspace --all-targets -- -D warnings`, + and `cargo test --workspace` +- slim-install smoke: isolated `uv run --with` default install/import checks, heavy-package + absence, and CLI help checks with `nemo-switchyard[cli,server] @ file://...` +- `uv run mypy switchyard` is a signal job (`continue-on-error`) but should still be run for typed + package code, public APIs, profiles, route bundles, backends, request/response models, and translation changes. -Python pre-PR gate: +The local pre-commit config also runs the Rust fmt/clippy gate when Rust files, +`Cargo.toml`, or `Cargo.lock` change. Treat those hooks as hard failures; do not +bypass them with `--no-verify`. + +A CI-equivalent `ruff` claim means a clean checkout or generated artifacts removed. Do not treat +`uv run ruff check . --exclude docs/.venv-docs --exclude docs/_build` as canonical PR validation; +that is only a dirty-workspace workaround. + +If generated docs artifacts exist locally, prefer: + +```bash +rm -rf docs/.venv-docs docs/_build site/_build +git status -sb +uv run ruff check . +``` + +## Standard Command Sets + +### Fast local loop + +```bash +uv run pytest tests/test_.py -v +uv run pytest tests/test_.py::test_name -v +uv run ruff check . +``` + +Pytest uses `addopts = "-x"`; use `-o addopts= --maxfail=10` when finding all failures: + +```bash +uv run pytest tests/ -v -o addopts= --maxfail=10 -m "not integration" +``` + +### Pre-PR package-code gate + +Use the hermetic test gate by default so local credentials cannot trigger live provider calls: ```bash uv run ruff check . uv run mypy switchyard -env -u OPENROUTER_API_KEY -u NVIDIA_API_KEY -u OPENAI_API_KEY \ - -u ANTHROPIC_API_KEY uv run pytest tests/ -v -m "not integration" +env -u OPENROUTER_API_KEY -u NVIDIA_API_KEY -u OPENAI_API_KEY -u ANTHROPIC_API_KEY uv run pytest tests/ -v -m "not integration" ``` -Rust workspace gate: +If you need to mirror the CI pytest command locally, run from a clean checkout or explicitly remove +provider credentials first: ```bash -cargo fmt --all --check -cargo clippy --workspace --all-targets -- -D warnings -cargo test --workspace +env -u OPENROUTER_API_KEY -u NVIDIA_API_KEY -u OPENAI_API_KEY -u ANTHROPIC_API_KEY uv run pytest tests/ -v -m "not integration" ``` -Use `-o addopts=` when the repository's pytest defaults would stop failure collection too early. -Run focused tests first, then broaden only when the changed contract or shared ownership warrants it. +### Live integration/e2e gate + +Run this only when the user asked for live validation and accepts upstream API calls: + +```bash +NVIDIA_API_KEY=... uv run pytest tests/e2e/ -v -m integration -o addopts= --maxfail=10 +``` -## Live Tests +`tests/e2e/conftest.py` and per-file markers/skips own credential behavior. Do not run live e2e as +a default PR gate. -Integration tests can spend money, require credentials, and contact real services. Never run them -as an automatic pre-PR step. Provision `NVIDIA_API_KEY` through the environment or a secret manager -before running the user-authorized path, and state which provider was called: +### Slim-install smoke gate + +Run after touching `pyproject.toml`, lockfiles, top-level imports, optional extras, +`switchyard/__init__.py`, CLI/server exports, or anything that could pull heavy packages into the +default install: ```bash -uv run pytest tests/e2e/.py \ - -v -m integration -o addopts= +SWITCHYARD_ROOT="$(git rev-parse --show-toplevel)" +SWITCHYARD_DEFAULT_PACKAGE="nemo-switchyard @ file://${SWITCHYARD_ROOT}" +SWITCHYARD_EXTRAS_PACKAGE="nemo-switchyard[cli,server] @ file://${SWITCHYARD_ROOT}" +cd /tmp +uv run --isolated --no-project --python 3.12 \ + --with "${SWITCHYARD_DEFAULT_PACKAGE}" \ + python - <<'PY' +import importlib.util +import switchyard +print('import OK, version:', switchyard.__version__) +from switchyard import RandomRoutingPresets +print('preset import OK:', sorted(RandomRoutingPresets.PRESETS)) +forbidden = [ + 'torch', 'transformers', 'huggingface_hub', 'litellm', + 'datasets', 'tokenizers', 'safetensors', 'pyarrow', 'agents', 'mcp', 'docx', +] +extras = [m for m in forbidden if importlib.util.find_spec(m) is not None] +if extras: + raise SystemExit(f'FAIL: heavy packages pulled into slim install: {extras}') +print('OK: heavy packages absent') +PY +uv run --isolated --no-project \ + --python 3.12 \ + --with "${SWITCHYARD_EXTRAS_PACKAGE}" \ + switchyard launch claude --help >/dev/null +uv run --isolated --no-project \ + --python 3.12 \ + --with "${SWITCHYARD_EXTRAS_PACKAGE}" \ + switchyard launch claude --help 2>&1 \ + | grep -q -- '--routing-profiles' || { echo 'FAIL: --routing-profiles flag missing from help'; exit 1; } +uv run --isolated --no-project \ + --python 3.12 \ + --with "${SWITCHYARD_EXTRAS_PACKAGE}" \ + switchyard serve --help >/dev/null ``` -## Skill Validation +### Skill/docs-only gate -Each retained skill must have YAML frontmatter with `name` and `description`: +For `.agents/skills/` changes, validate YAML frontmatter, whitespace, and any Python helper scripts: ```bash -uv run python - <<'PY' +python - <<'PY' from pathlib import Path - import yaml - -for path in Path(".agents/skills").glob("*/SKILL.md"): - parts = path.read_text().split("---", 2) - data = yaml.safe_load(parts[1]) if len(parts) == 3 else None - if not isinstance(data, dict) or not data.get("name") or not data.get("description"): - raise SystemExit(f"{path}: missing name or description") -print("skill frontmatter is valid") +for p in Path('.agents/skills').glob('*/SKILL.md'): + text = p.read_text(encoding='utf-8') + if not text.startswith('---\n'): + raise SystemExit(f'{p}: missing frontmatter start') + end = text.find('\n---\n', 4) + if end == -1: + raise SystemExit(f'{p}: missing frontmatter end') + data = yaml.safe_load(text[4:end]) + if not isinstance(data, dict) or not data.get('name') or not data.get('description'): + raise SystemExit(f'{p}: missing name/description') +print('OK: skill frontmatter parses') PY git diff --check +find .agents/skills -path '*/scripts/*.py' -print +``` + +For any listed skill Python helper, run targeted lint and SPDX checks: + +```bash +.venv/bin/ruff check --isolated .agents/skills//scripts/.py +python - <<'PY' +from importlib.machinery import SourceFileLoader +from pathlib import Path +spdx = SourceFileLoader('spdx', '.hooks/add_spdx_headers.py').load_module() +paths = [Path('.agents/skills//scripts/.py')] +bad = [str(p) for p in paths if not spdx.has_required_header(p)] +if bad: + raise SystemExit('missing SPDX header: ' + ', '.join(bad)) +print('OK: SPDX headers present') +PY +``` + +`.agents` is gitignored, so stage new or modified skills explicitly: + +```bash +git status --ignored -sb .agents/skills +git add -f .agents/skills/ +``` + +## Failure Map + +### Ruff + +- Generated docs/venv paths (`docs/.venv-docs/`, `docs/_build/`, `site/_build/`) are local debris; + remove them rather than editing vendored/generated files. +- Use autofix narrowly, then inspect the diff: + + ```bash + uv run ruff check --fix . + uv run ruff check . + ``` + +### SPDX + +CI requires this exact header in every Python file, after an optional shebang: + +```python +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +``` + +Fix with: + +```bash +python .hooks/add_spdx_headers.py path/to/file.py +uv run ruff check . +``` + +### Pytest + +Classify before editing: + +- Optional package import (`torch`, `fastapi`, `prompt_toolkit`, `nemo_platform`) → keep + optional deps out of top-level/default install paths; use lazy imports and clear runtime errors. +- Unit test makes network call → mock with `respx`, `httpx.ASGITransport`, or a fake backend. +- Translation mismatch → run translation-focused tests before touching broad plumbing. +- Streaming failure → inspect streaming response accumulator, SSE helpers, and translation-stream tests. +- CLI/config regression → run launcher/config/model/verify tests. +- Routing/stats regression → run the random-routing suite. + +Useful focused groups: + +```bash +uv run pytest tests/test_anthropic_openai_translation.py tests/test_responses_openai_translation.py tests/test_translation_engine_chaos.py -v -o addopts= +uv run pytest tests/test_launch_claude.py tests/test_launch_codex.py tests/test_launch_route_builder.py tests/test_user_config.py tests/test_model_discovery.py tests/test_verify.py -v -o addopts= +uv run pytest tests/test_random_routing_config.py tests/test_random_routing_llm_backend.py tests/test_random_routing_presets.py tests/test_random_routing_profile.py -v -o addopts= +``` + +### Mypy + +Mypy is strict on `switchyard/` and ignores tests. Prefer concrete request/response subclasses after +`isinstance` narrowing, `TYPE_CHECKING` imports for circular or optional types, and aliases from +`switchyard/lib/roles.py` instead of broad `Any`. Use targeted ignores with error codes only when an +external incompatibility is localized. + +### Slim install / dependencies + +Default install must not pull heavyweight packages: + +- `torch`, `transformers`, `huggingface_hub`, `litellm`, `datasets`, `tokenizers`, + `safetensors`, `pyarrow`, `agents`, `mcp`, `docx` + +Keep them in optional extras or dev groups, not `[project.dependencies]`. Dependency changes require +explicit user intent; after changing dependencies run `uv lock`, `uv sync`, and the selected tests. + +### Stale package names + +The distribution package is `nemo-switchyard`; imports and the CLI are `switchyard`. Do not +introduce `nemo_switchyard` imports or publish docs that imply `pip install switchyard`. Check: + +```bash +uv run pytest tests/test_cli_stale_names.py tests/test_no_stale_module_paths.py tests/test_version_package_name.py -v -o addopts= +``` + +## Common Mistakes + +| Mistake | Reality | +|---|---| +| Running `uv run ruff check . --exclude docs/.venv-docs --exclude docs/_build` and calling it CI-equivalent | That is a dirty-workspace workaround, not the CI command. Either remove the generated artifacts (`rm -rf docs/.venv-docs docs/_build site/_build`) or run from a clean checkout. | +| Running `uv run pytest tests/ -v` with `OPENROUTER_API_KEY`/`NVIDIA_API_KEY`/`OPENAI_API_KEY`/`ANTHROPIC_API_KEY` set in your shell | Tests that mock providers can accidentally hit live endpoints. Default to `-m "not integration"`, or strip credentials with `env -u`. | +| Treating mypy as optional because CI marks it `continue-on-error` | Mypy still catches real bugs in `switchyard/` typed code. Run it for any change to profiles, route bundles, backends, request/response models, or translation. | +| Skipping the slim-install smoke gate after a dependency or top-level import change | This is the *actual* hard CI gate that catches `torch`/`transformers` accidentally landing in the default install. | +| Claiming validation from `scripts/select_validation.py --changed` after committing changes | `--changed` covers staged, unstaged, and untracked paths. Use `--base origin/main` to include committed branch changes, or pass `--path` explicitly. | +| Adding `# noqa` or per-line ignores to make ruff green | Ruff is a hard CI gate. Fix the code or, if the rule is wrong here, lift the ignore to the file or project level with a one-line justification. | +| Editing a generated artifact (e.g., `docs/.venv-docs/...`) because ruff complained about it | Delete the artifact instead. CI does not have it; you should not either. | + +## Report Format + +Report validation with exact commands and live-call status: + +```markdown +Validation: +- `uv run ruff check .` — pass +- `uv run mypy switchyard` — pass / not run () +- `uv run pytest tests/ -v -m "not integration"` — pass / not run () +- Slim install smoke — pass / not run () +- Live provider tests — not run / pass () ``` -## Reporting +## Related Skills -Report exact commands, pass/fail status, and whether any live network call occurred. Do not call a -focused test run "full CI", and do not treat mypy or clippy warnings as unrelated without checking -whether the diff introduced them. +- [`switchyard-codebase-exploration`](../switchyard-codebase-exploration/SKILL.md) — run this **before** the testing-ci skill on any non-trivial change, so the validation set you pick covers every file the change actually touches (importers, profile builders, tests), not just the diff surface. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d37e49106..760284276 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -132,6 +132,12 @@ jobs: - name: mypy run: uv run mypy switchyard + # Keep Fern validation inside the required CI Success aggregate so a docs + # configuration failure cannot be merged as an optional check. + fern-docs: + name: Fern docs + uses: ./.github/workflows/fern-docs-ci.yml + # Unit tests on supported Python versions. Integration/e2e tests are # intentionally deselected here because they may need Docker, live provider # credentials, or network egress. @@ -237,7 +243,7 @@ jobs: ci-success: name: CI Success if: always() - needs: [changes, lint, spdx-headers, rust, typecheck, test, slim-install-smoke] + needs: [changes, lint, spdx-headers, rust, typecheck, fern-docs, test, slim-install-smoke] runs-on: ubuntu-latest steps: - name: Verify all jobs succeeded diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml deleted file mode 100644 index d8611564a..000000000 --- a/.github/workflows/docs.yml +++ /dev/null @@ -1,100 +0,0 @@ -name: Documentation - -on: - pull_request: - paths: - - "docs/**" - - "mkdocs_hooks.py" - - "mkdocs.yml" - - "pyproject.toml" - - "uv.lock" - - ".github/workflows/docs.yml" - push: - branches: [main] - paths: - - "docs/**" - - "mkdocs_hooks.py" - - "mkdocs.yml" - - "pyproject.toml" - - "uv.lock" - - ".github/workflows/docs.yml" - -# Cancel superseded PR runs; keep main builds so gh-pages writes don't race. -concurrency: - group: ${{ github.workflow }}-${{ github.ref }} - cancel-in-progress: ${{ github.event_name == 'pull_request' }} - -env: - MKDOCS_SOURCE_REF: ${{ github.event.pull_request.head.sha || github.sha }} - -permissions: - contents: read - -jobs: - build: - name: Build docs - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: astral-sh/setup-uv@v6 - with: - python-version: "3.12" - enable-cache: true - cache-dependency-glob: "uv.lock" - - run: uv sync --only-group docs --locked - - name: Build documentation - run: uv run --only-group docs mkdocs build --strict - - name: Upload site artifact - uses: actions/upload-artifact@v4 - with: - name: docs-site - path: site/ - retention-days: 1 - - deploy: - name: Deploy to GitHub Pages - needs: build - if: github.event_name == 'push' && github.ref == 'refs/heads/main' - runs-on: ubuntu-latest - permissions: - contents: write - steps: - - name: Download site artifact - uses: actions/download-artifact@v4 - with: - name: docs-site - path: site/ - - name: Deploy to GitHub Pages - uses: peaceiris/actions-gh-pages@v4 - with: - github_token: ${{ secrets.GITHUB_TOKEN }} - publish_dir: ./site - keep_files: true - - # Fork PRs can't write to gh-pages with GITHUB_TOKEN, so the preview is - # gated to same-repo PRs. - preview: - name: Deploy PR preview - needs: build - if: >- - github.event_name == 'pull_request' && - github.event.pull_request.head.repo.full_name == github.repository - runs-on: ubuntu-latest - permissions: - contents: write - pull-requests: write - steps: - - uses: actions/checkout@v4 - - name: Download site artifact - uses: actions/download-artifact@v4 - with: - name: docs-site - path: site/ - - name: Deploy PR preview - uses: rossjrw/pr-preview-action@v1 - with: - source-dir: ./site - preview-branch: gh-pages - umbrella-dir: pr-preview - action: auto - qr-code: false diff --git a/.github/workflows/fern-docs-ci.yml b/.github/workflows/fern-docs-ci.yml new file mode 100644 index 000000000..12ee62036 --- /dev/null +++ b/.github/workflows/fern-docs-ci.yml @@ -0,0 +1,53 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Reusable Fern validation gate called by the repository CI workflow. + +name: Fern docs (check) + +on: + workflow_call: + +permissions: + contents: read + +jobs: + check: + name: Check Fern docs + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Setup Node.js + uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 + with: + node-version: "24" + package-manager-cache: false + + - name: Install Fern CLI + run: | # zizmor: ignore[adhoc-packages] Fern is exact-version pinned in fern.config.json. + FERN_VERSION="$(jq -r '.version' docs/fern/fern.config.json)" + if ! [[ "$FERN_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[A-Za-z0-9.]+)?$ ]]; then + echo "::error::Refusing to install unexpected Fern CLI version: $FERN_VERSION" + exit 1 + fi + npm install -g "fern-api@${FERN_VERSION}" + + - name: Validate Fern configuration + working-directory: ./docs/fern + run: fern check diff --git a/.github/workflows/fern-docs-preview-build.yml b/.github/workflows/fern-docs-preview-build.yml new file mode 100644 index 000000000..4de10827c --- /dev/null +++ b/.github/workflows/fern-docs-preview-build.yml @@ -0,0 +1,67 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Workflow 1 of 2 for Fern doc previews. +# +# Collects docs sources and changed-file metadata from same-repository PR +# branches. No secrets are used here. Fork PRs receive the required Fern check +# but do not enter the hosted-preview path. The companion workflow builds the +# preview and posts the PR comment. + +name: "Preview Fern Docs: Build" + +on: + pull_request: + paths: + - "docs/**" + - ".github/workflows/fern-docs-preview-build.yml" + +permissions: + contents: read + +concurrency: + group: fern-docs-preview-${{ github.event.pull_request.number }} + cancel-in-progress: true + +jobs: + collect: + name: Collect preview sources + if: ${{ github.event.pull_request.head.repo.full_name == github.repository }} + runs-on: ubuntu-latest + steps: + - name: Checkout PR + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + + - name: Save changed Markdown paths + env: + BASE_REF: ${{ github.base_ref }} + run: | + mkdir -p .preview-metadata + git diff --name-only "origin/${BASE_REF}...HEAD" -- 'docs/**/*.md' > .preview-metadata/changed_md_files 2>/dev/null || true + + # The nightly navigation reaches from docs/fern/ into docs/**/*.md, + # so the artifact must preserve the complete docs tree. + - name: Upload docs sources and metadata + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: fern-preview + path: | + docs/ + .preview-metadata/ + include-hidden-files: true + retention-days: 1 diff --git a/.github/workflows/fern-docs-preview-comment.yml b/.github/workflows/fern-docs-preview-comment.yml new file mode 100644 index 000000000..e8e0df671 --- /dev/null +++ b/.github/workflows/fern-docs-preview-comment.yml @@ -0,0 +1,191 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Workflow 2 of 2 for Fern doc previews. +# +# Runs after the same-repository collector succeeds, downloads only its artifact, +# creates a Fern preview with DOCS_FERN_TOKEN, and upserts a PR comment. PR +# identity and the Fern CLI version come from trusted GitHub/default-branch +# state. The workflow never checks out or executes scripts from the PR branch. + +name: "Preview Fern Docs: Comment" + +on: + workflow_run: # zizmor: ignore[dangerous-triggers] Same-repository runs only; identity/tooling are trusted. + workflows: ["Preview Fern Docs: Build"] + types: [completed] + +permissions: {} + +concurrency: + group: fern-docs-preview-${{ github.event.workflow_run.pull_requests[0].number }} + cancel-in-progress: true + +jobs: + preview: + name: Publish preview + runs-on: ubuntu-latest + if: >- + ${{ + github.event.workflow_run.conclusion == 'success' && + github.event.workflow_run.event == 'pull_request' && + github.event.workflow_run.head_repository.full_name == github.repository && + github.event.workflow_run.pull_requests[0].number != null + }} + permissions: + # Download the triggering artifact and read only the trusted default branch. + actions: read # Required to download the triggering workflow artifact. + contents: read + outputs: + page_links: ${{ steps.page-links.outputs.page_links }} + pr_number: ${{ steps.metadata.outputs.pr_number }} + preview_url: ${{ steps.generate-docs.outputs.preview_url }} + steps: + - name: Checkout trusted Fern configuration + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.event.repository.default_branch }} + path: trusted + persist-credentials: false + + - name: Download docs sources and metadata + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: fern-preview + run-id: ${{ github.event.workflow_run.id }} + github-token: ${{ secrets.GITHUB_TOKEN }} + path: preview-source + + - name: Read trusted workflow metadata + id: metadata + env: + PR_NUMBER: ${{ github.event.workflow_run.pull_requests[0].number }} + run: | + if ! [[ "$PR_NUMBER" =~ ^[0-9]+$ ]]; then + echo "::error::workflow_run did not identify a pull request." + exit 1 + fi + echo "pr_number=$PR_NUMBER" >> "$GITHUB_OUTPUT" + + - name: Read trusted Fern CLI version + id: fern-version + run: | + FERN_VERSION="$(jq -r '.version' trusted/docs/fern/fern.config.json)" + if ! [[ "$FERN_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[A-Za-z0-9.]+)?$ ]]; then + echo "::error::Refusing to install unexpected Fern CLI version: $FERN_VERSION" + exit 1 + fi + echo "version=$FERN_VERSION" >> "$GITHUB_OUTPUT" + + - name: Setup Node.js + uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 + with: + node-version: "24" + package-manager-cache: false + + - name: Install pinned Fern CLI + env: + FERN_VERSION: ${{ steps.fern-version.outputs.version }} + run: npm install -g "fern-api@$FERN_VERSION" # zizmor: ignore[adhoc-packages] Trusted exact version. + + - name: Generate preview URL + id: generate-docs + env: + FERN_TOKEN: ${{ secrets.DOCS_FERN_TOKEN }} + PR_NUMBER: ${{ steps.metadata.outputs.pr_number }} + working-directory: ./preview-source/docs/fern + run: | + OUTPUT=$(fern generate --docs --preview --id "pr-$PR_NUMBER" --force --no-prompt 2>&1) + echo "$OUTPUT" + URL=$(echo "$OUTPUT" | grep -oP 'Published docs to \K.*(?= \()') + if [ -z "$URL" ]; then + echo "::error::Failed to generate preview URL. See fern output above." + exit 1 + fi + echo "preview_url=$URL" >> "$GITHUB_OUTPUT" + + - name: Build page links for changed Markdown files + id: page-links + env: + FERN_TOKEN: ${{ secrets.DOCS_FERN_TOKEN }} + PREVIEW_URL: ${{ steps.generate-docs.outputs.preview_url }} + run: | + CHANGED_FILES="" + if [ -f preview-source/.preview-metadata/changed_md_files ]; then + CHANGED_FILES=$(cat preview-source/.preview-metadata/changed_md_files) + fi + + if [ -z "$CHANGED_FILES" ] || [ -z "$PREVIEW_URL" ]; then + echo "page_links=" >> "$GITHUB_OUTPUT" + exit 0 + fi + + BASE_URL=$(echo "$PREVIEW_URL" | grep -oP 'https?://[^/]+') + FILES_PARAM=$(echo "$CHANGED_FILES" | tr '\n' ',' | sed 's/,$//' \ + | python3 -c "import sys, urllib.parse; print(urllib.parse.quote(sys.stdin.read().strip(), safe=',/'))") + RESPONSE=$(curl -sf -H "FERN_TOKEN: $FERN_TOKEN" "${PREVIEW_URL}/api/fern-docs/get-slug-for-file?files=${FILES_PARAM}" 2>/dev/null) || { + echo "page_links=" >> "$GITHUB_OUTPUT" + exit 0 + } + + PAGE_LINKS=$(echo "$RESPONSE" | jq -r --arg url "$BASE_URL" \ + '.mappings[] | select(.slug != null) | "- [\(.slug)](\($url)/\(.slug))"') + + if [ -n "$PAGE_LINKS" ]; then + { echo "page_links<> "$GITHUB_OUTPUT" + else + echo "page_links=" >> "$GITHUB_OUTPUT" + fi + + comment: + name: Update pull request comment + needs: preview + runs-on: ubuntu-latest + permissions: + # The comment job receives no Fern token or repository content access. + pull-requests: write # Required to create or update the preview comment. + steps: + - name: Post or update PR comment + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + PAGE_LINKS: ${{ needs.preview.outputs.page_links }} + PREVIEW_URL: ${{ needs.preview.outputs.preview_url }} + PR_NUMBER: ${{ needs.preview.outputs.pr_number }} + REPOSITORY: ${{ github.repository }} + run: | + BODY=":herb: **Preview your docs:** <${PREVIEW_URL}>" + if [ -n "${PAGE_LINKS}" ]; then + BODY="${BODY} + + Here are the documentation pages you've updated: + ${PAGE_LINKS}" + fi + + MARKER="" + BODY="${BODY} + + ${MARKER}" + + COMMENT_ID=$(gh api "repos/$REPOSITORY/issues/$PR_NUMBER/comments" \ + --jq '.[] | select(.user.login == "github-actions[bot]" and (.body | contains(""))) | .id' \ + | head -1) + + if [ -n "$COMMENT_ID" ]; then + gh api "repos/$REPOSITORY/issues/comments/$COMMENT_ID" \ + -X PATCH -f body="$BODY" + else + gh api "repos/$REPOSITORY/issues/$PR_NUMBER/comments" \ + -f body="$BODY" + fi diff --git a/.github/workflows/getting-started.yml b/.github/workflows/getting-started.yml index 6b7511b02..b1db74048 100644 --- a/.github/workflows/getting-started.yml +++ b/.github/workflows/getting-started.yml @@ -12,6 +12,8 @@ on: paths: - "docs/getting_started.md" - "crates/**" + - "switchyard/**" + - "switchyard_rust/**" - "tests/getting_started/**" - "Cargo.toml" - "Cargo.lock" @@ -24,6 +26,8 @@ on: paths: - "docs/getting_started.md" - "crates/**" + - "switchyard/**" + - "switchyard_rust/**" - "tests/getting_started/**" - "Cargo.toml" - "Cargo.lock" diff --git a/.github/workflows/publish-fern-docs.yml b/.github/workflows/publish-fern-docs.yml new file mode 100644 index 000000000..890a74fd8 --- /dev/null +++ b/.github/workflows/publish-fern-docs.yml @@ -0,0 +1,112 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Publishes the Fern site for docs/v* tags or a manual workflow dispatch. +# Required configuration: organization secret DOCS_FERN_TOKEN. + +name: Publish Fern Docs + +on: + push: + tags: + - "docs/v*" + workflow_dispatch: + +permissions: {} + +concurrency: + group: fern-docs-website + cancel-in-progress: false + +jobs: + publish: + name: Publish Fern site + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - name: Checkout repository + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Setup Node.js + uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 + with: + node-version: "24" + package-manager-cache: false + + - name: Install pinned Fern CLI + run: | # zizmor: ignore[adhoc-packages] Fern is exact-version pinned in fern.config.json. + FERN_VERSION="$(jq -r '.version' docs/fern/fern.config.json)" + if ! [[ "$FERN_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[A-Za-z0-9.]+)?$ ]]; then + echo "::error::Refusing to install unexpected Fern CLI version: $FERN_VERSION" + exit 1 + fi + npm install -g "fern-api@${FERN_VERSION}" + + - name: Publish docs + env: + FERN_TOKEN: ${{ secrets.DOCS_FERN_TOKEN }} + working-directory: ./docs/fern + run: fern generate --docs + + - name: Verify custom-domain publication + run: | + for attempt in {1..12}; do + if curl --fail --silent --show-error --location --max-time 20 \ + --output /dev/null https://docs.nvidia.com/nemo/switchyard/home; then + exit 0 + fi + echo "Custom domain is not ready (attempt ${attempt}/12); retrying in 10 seconds." + sleep 10 + done + echo "::error::Fern published, but the custom domain did not become ready." + exit 1 + + - name: Generate legacy GitHub Pages redirects + run: python docs/fern/generate_legacy_redirect_site.py --output legacy-docs-site + + - name: Upload legacy redirects + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: legacy-docs-site + path: legacy-docs-site/ + include-hidden-files: true + retention-days: 1 + + redirects: + name: Deploy legacy redirects + needs: publish + runs-on: ubuntu-latest + permissions: + # This job receives the generated artifact, but never the Fern token. + actions: read # Required to download redirects generated by the publish job. + contents: write # Required only to replace the gh-pages branch. + steps: + - name: Download legacy redirects + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: legacy-docs-site + path: legacy-docs-site + + - name: Deploy legacy redirects to GitHub Pages + uses: peaceiris/actions-gh-pages@84c30a85c19949d7eee79c4ff27748b70285e453 # v4 + with: + github_token: ${{ secrets.GITHUB_TOKEN }} + publish_branch: gh-pages + publish_dir: ./legacy-docs-site + keep_files: false + full_commit_message: "docs: redirect GitHub Pages to Fern" diff --git a/AGENTS.md b/AGENTS.md index 4ce6f7e47..0cc7b4b35 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -128,7 +128,7 @@ testing, and review do not require loading a skill. |---|---| | `publish-python-release` | Python wheel artifacts, PyPI releases, and release workflow changes | | `switchyard-coding-agent-launchers` | Claude Code, Codex, or OpenClaw launcher behavior | -| `switchyard-docs` | Published MkDocs pages, strict builds, previews, and docs CI | +| `switchyard-docs` | Published Fern docs, `fern check`, previews, and docs CI | | `switchyard-rust-review` | Focused review of Rust, PyO3, async, streaming, and crate boundaries | | `switchyard-stage-router-scorer` | Replaying benchmark trajectories through the stage-router scorer and picker | | `switchyard-testing-ci` | Selecting non-obvious validation or diagnosing CI failures | @@ -354,7 +354,7 @@ uvicorn.run(build_switchyard_app(switchyard), port=4000) - Export new public classes from `switchyard/__init__.py` with `__all__`. - Write unit tests for new roles and bug fixes. - Use `ProxyContext.metadata` for cross-component state within a request. -- In a new `LLMBackend`, map upstream context-window 4xx to `SwitchyardError::ContextWindowExceeded` (Rust) — the chain executor uses it for evict-and-retry. See [Context-Window Handling](docs/context_window.md). +- In a new `LLMBackend`, map upstream context-window 4xx to `SwitchyardError::ContextWindowExceeded` (Rust) — the chain executor uses it for evict-and-retry. See [Context-Window Handling](docs/operations/context_window.md). ### Ask first - Modifying `pyproject.toml` dependencies. diff --git a/CHANGELOG.md b/CHANGELOG.md index 63195a865..26468fa3d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -46,7 +46,7 @@ traffic that sits between client applications and LLM backends. - **Four-role chain** — `RequestProcessor → LLMBackend → ResponseProcessor → TranslationEngine`, executed by the Rust-backed core. See - [Architecture](docs/ARCHITECTURE.md). + [Architecture](docs/architecture.md). - **Protocol translation** — convert between OpenAI Chat Completions, Anthropic Messages, and OpenAI Responses wire formats, so each client keeps speaking its native API regardless of the upstream backend. @@ -55,7 +55,7 @@ traffic that sits between client applications and LLM backends. `passthrough`, `random_routing`, `stage_router`, `deterministic` (LLM-as-classifier), `latency_service`, and `noop`. - **Routing strategies** — weighted random split, signal-driven **stage-router** - escalation (see [Stage-Router Routing](docs/stage_router_routing.md)), + escalation (see [Stage-Router Routing](docs/routing_algorithms/stage_router_routing.md)), LLM-as-classifier strong/weak routing, and latency-aware multi-endpoint failover. - **One-command launchers** — `switchyard launch claude`, `launch codex`, and @@ -66,7 +66,7 @@ traffic that sits between client applications and LLM backends. `--list-models`), and `verify` / `launch --smoke` round-trip checks. - **Observability** — Prometheus `/metrics`, a JSON `/v1/stats` (`/v1/routing/stats` alias), and per-request cost/token/latency stats. See - [Metrics Reference](docs/METRICS_REFERENCE.md). + [Metrics Reference](docs/internal/metrics_reference.md). - **Python library** — `ProfileSwitchyard` driven by typed profile configs (`PassthroughProfileConfig`, `RandomRoutingProfileConfig`, `StageRouterProfileConfig`, …) and typed `ChatRequest` / `ChatResponse` diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2e199e652..d8ad912c8 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -81,6 +81,12 @@ These commands run in CI on every push. **Fix linting errors locally before push uv run ruff check --fix . ``` +For a path-based validation plan, run `python scripts/select_validation.py --changed` (or use +`--base origin/main` after committing). Maintainers adding or changing workflows should also use +the [CI options and workflow design guide](scripts/ci/README.md), which covers required-check +aggregation, fork trust boundaries, concurrency, permissions, immutable action pins, and workflow +security checks. + ### 3. Commit messages Commit messages must follow diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index e0826286b..9b9fe50d5 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -5,7 +5,7 @@ Switchyard itself. If you only want to **use** the package, see [README](README.md). For deeper architectural docs, see [Agents](AGENTS.md) and -[Architecture](docs/ARCHITECTURE.md). +[Architecture](docs/architecture.mdx). ## Setup diff --git a/docs/AGENTS.md b/docs/AGENTS.md new file mode 100644 index 000000000..b898ae794 --- /dev/null +++ b/docs/AGENTS.md @@ -0,0 +1,48 @@ +# Documentation instructions + +These instructions apply to everything under `docs/`. More specific rules in +`docs/fern/AGENTS.md` also apply inside `docs/fern/`. + +## Before editing + +- Read `.agents/skills/switchyard-docs/SKILL.md` completely. +- Read `.agents/skills/switchyard-codebase-exploration/SKILL.md` when documentation cites code, + commands, configuration, or public APIs. +- Treat `docs/fern/versions/nightly.yml` as the source of truth for the published page set, order, + labels, and routes. + +## Content boundaries + +- Published pages are `docs/**/*.md`, excluding `docs/fern/`. +- `docs/internal/**/*.md` files are unpublished. Do not add them to Fern navigation or link to them + from published pages unless the task explicitly promotes that content. +- Keep contributor guidance in `README.md` and `AGENTS.md`; do not add either file to Fern + navigation. +- Use public `switchyard` imports in examples, not `switchyard.lib.*` internals. +- Link repository files outside `docs/` with absolute GitHub URLs. + +## Fern authoring + +- Give every published page non-empty `title` and `description` frontmatter. +- Keep navigation sections label-only and place authored pages under `contents:`. Do not assign the + same page to both a section `path:` and a child page. +- Use canonical, version-agnostic Fern routes derived from navigation labels for internal links. +- Do not link unpublished internal notes from published pages. +- Use Fern callout components instead of GitHub admonition syntax. + +## Validation + +Run the smallest relevant checks after every docs change: + +```bash +cd docs && make check +uv run pytest tests/test_fern_docs.py -v +``` + +Before pushing, also follow the repository-wide validation requirements in the root `AGENTS.md`. +If a docs workflow, route invariant, or ownership boundary changes, update +`.agents/skills/switchyard-docs/SKILL.md` in the same change. + +Fern validation is part of the required `CI Success` aggregate. Keep preview concurrency scoped to +the PR number, and never trust PR-artifact metadata for the target PR or Fern CLI version. Hosted +previews are limited to same-repository branches; fork PRs receive validation only. diff --git a/docs/Makefile b/docs/Makefile index c710ed75b..ed98c7a15 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -1,25 +1,19 @@ -.PHONY: env html publish live clean help +.PHONY: check preview clean help -REPO_ROOT := $(dir $(abspath $(lastword $(MAKEFILE_LIST)))).. +FERN_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))fern +FERN_VERSION := $(shell jq -r .version $(FERN_DIR)/fern.config.json) +FERN := npx -y fern-api@$(FERN_VERSION) help: - @echo "make env : sync the docs dependency group via uv" - @echo "make html : build HTML site (incremental)" - @echo "make publish : build HTML site (strict, fails on warnings)" - @echo "make live : live-reload server at http://127.0.0.1:8000" - @echo "make clean : remove build artefacts" + @echo "make check : validate Fern configuration and content" + @echo "make preview : serve a local Fern preview" + @echo "make clean : remove local Fern build artifacts" -env: - cd $(REPO_ROOT) && uv sync --only-group docs +check: + cd $(FERN_DIR) && $(FERN) check -html: env - cd $(REPO_ROOT) && uv run --only-group docs mkdocs build - -publish: env - cd $(REPO_ROOT) && uv run --only-group docs mkdocs build --strict - -live: env - cd $(REPO_ROOT) && uv run --only-group docs mkdocs serve +preview: + cd $(FERN_DIR) && $(FERN) docs dev clean: - rm -rf $(REPO_ROOT)/site + rm -rf $(FERN_DIR)/.fern $(FERN_DIR)/node_modules diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 000000000..fa20092f3 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,53 @@ +# NeMo Switchyard docs + +Switchyard documentation is authored as [Fern](https://buildwithfern.com/) Markdown in this directory. +The current Fern site is available at +**[nemo-switchyard.docs.buildwithfern.com/nemo/switchyard](https://nemo-switchyard.docs.buildwithfern.com/nemo/switchyard/home)**. + +## Where to make changes + +| Change | Location | +|---|---| +| Published page | `docs/**/*.md` | +| Public navigation | [`fern/versions/nightly.yml`](fern/versions/nightly.yml) | +| Site settings and redirects | [`fern/docs.yml`](fern/docs.yml) | +| Internal, unpublished notes | `docs/internal/**/*.md` | +| Fern build and publishing details | [`fern/README.md`](fern/README.md) | + +Markdown files under `internal/` are intentionally excluded from the public navigation. Promote one +to Markdown only when it is ready to become public documentation. + +## Common commands + +Run these from the repository root: + +```bash +cd docs && make check +uv run pytest tests/test_fern_docs.py -v +cd docs && make preview +``` + +`make check` uses the Fern CLI version pinned in +[`fern/fern.config.json`](fern/fern.config.json). `make preview` starts a local server and does not +publish the site. + +## Add or edit a page + +1. Add or update the Markdown page under `docs/`. +2. Add new pages to [`fern/versions/nightly.yml`](fern/versions/nightly.yml) in the intended order. +3. Use canonical Fern routes derived from the navigation labels for internal links. +4. Run the Fern check and route regression tests above. + +Every pull request runs Fern validation inside the required `CI Success` aggregate. Docs changes +from repository branches also start a PR-numbered hosted preview: the collector uploads content +without secrets, while the trusted workflow derives PR identity and the Fern CLI version from +GitHub/default-branch state before publishing. Preview generation and PR commenting use separate +permission scopes. Fork PRs receive Fern validation but do not publish hosted previews. + +Legacy GitHub Pages paths are handled in two places: Fern redirects live in +[`fern/docs.yml`](fern/docs.yml), while +[`fern/generate_legacy_redirect_site.py`](fern/generate_legacy_redirect_site.py) builds the static +redirect-only GitHub Pages site. Keep both mappings synchronized. + +For agent-specific editing rules, see [`AGENTS.md`](AGENTS.md) and the +[`switchyard-docs`](../.agents/skills/switchyard-docs/SKILL.md) skill. diff --git a/docs/architecture.md b/docs/architecture.md index e77dab115..d1e0f3b0f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,5 +1,8 @@ -# Switchyard Architecture - +--- +title: "Architecture" +description: "Understand how NeMo Switchyard routes requests, translates provider formats, and handles responses end to end." +position: 2 +--- Switchyard is an LLM traffic proxy that sits between clients and model backends. It keeps the client-facing API stable while applying routing policy, translating provider formats, and handling configured fallbacks. @@ -36,7 +39,7 @@ flowchart TB Routing policy determines which model or endpoint receives a request. Depending on the selected strategy, that decision can use fixed weights, a classifier, request signals, or conversation affinity. See the -[Routing Overview](routing_algorithms/overview.md) for the available strategies. +[Routing Overview](/routing/overview) for the available strategies. ## Backend Wire Format @@ -92,6 +95,6 @@ upstream format. ## Related Documentation -- [Getting Started](getting_started.md): install Switchyard and run a first request -- [Routing Overview](routing_algorithms/overview.md): choose and configure a routing strategy -- [CLI Reference](cli_reference.md): configure and operate Switchyard from the command line +- [Getting Started](/get-started/getting-started): install Switchyard and run a first request +- [Routing Overview](/routing/overview): choose and configure a routing strategy +- [CLI Reference](/reference/cli-reference): configure and operate Switchyard from the command line diff --git a/docs/cli_reference.md b/docs/cli_reference.md index f0417ddf9..c31efe91a 100644 --- a/docs/cli_reference.md +++ b/docs/cli_reference.md @@ -1,5 +1,8 @@ -# CLI reference - +--- +title: "CLI Reference" +description: "Reference the NeMo Switchyard launch command, options, and deployment configuration path." +position: 1 +--- The documented CLI workflow launches a coding agent through the native Rust server. @@ -22,4 +25,5 @@ switchyard launch --model [--config ] [-- str: + """Return a static redirect page for one canonical Fern destination.""" + target = f"{CANONICAL_BASE}{destination}" + escaped_target = html.escape(target, quote=True) + javascript_target = json.dumps(target) + return f""" + + + + + + + NeMo Switchyard documentation moved + + + +

This documentation moved to {escaped_target}.

+ + +""" + + +def generate_redirect_site(output: Path) -> None: + """Write one GitHub Pages redirect document for every former MkDocs route.""" + output.mkdir(parents=True, exist_ok=True) + (output / ".nojekyll").touch() + for legacy_path, destination in LEGACY_REDIRECTS.items(): + page = output / legacy_path / "index.html" + page.parent.mkdir(parents=True, exist_ok=True) + page.write_text(render_redirect_page(destination), encoding="utf-8") + + +def main() -> None: + """Parse CLI arguments and generate the redirect-only GitHub Pages site.""" + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--output", type=Path, required=True) + args = parser.parse_args() + generate_redirect_site(args.output) + + +if __name__ == "__main__": + main() diff --git a/docs/fern/versions/nightly.yml b/docs/fern/versions/nightly.yml new file mode 100644 index 000000000..eacf1609c --- /dev/null +++ b/docs/fern/versions/nightly.yml @@ -0,0 +1,40 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +navigation: + - page: "Home" + path: ../../index.md + - section: "Get Started" + contents: + - page: "Getting Started" + path: ../../getting_started.md + - page: "Known Issues in 0.1.0" + path: ../../known_issues.md + - section: "Concepts" + contents: + - page: "Core Concepts" + path: ../../core_concepts.md + - page: "Architecture" + path: ../../architecture.md + - section: "Routing" + contents: + - page: "Overview" + path: ../../routing_algorithms/overview.md + - page: "Random Routing" + path: ../../routing_algorithms/random_routing.md + - page: "LLM Classifier Routing" + path: ../../routing_algorithms/llm_classifier_routing.md + - page: "Sticky Routing" + path: ../../routing_algorithms/sticky_routing.md + - page: "Stage-Router Routing" + path: ../../routing_algorithms/stage_router_routing.md + - page: "Escalation-Router Routing" + path: ../../routing_algorithms/escalation_router_routing.md + - section: "Operations" + contents: + - page: "Context-Window Handling" + path: ../../operations/context_window.md + - section: "Reference" + contents: + - page: "CLI Reference" + path: ../../cli_reference.md diff --git a/docs/getting_started.md b/docs/getting_started.md index e48eaf263..73284b0e0 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -1,5 +1,8 @@ -# Getting Started - +--- +title: "Getting Started" +description: "Install NeMo Switchyard and launch a coding agent through the packaged OpenRouter deployment." +position: 1 +--- ## Install ```bash @@ -28,4 +31,6 @@ To use another deployment: switchyard launch claude --model my-route --config routes.toml ``` -See `crates/switchyard-server/README.md` for the TOML schema. +See +[`crates/switchyard-server/README.md`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/README.md) +for the TOML schema. diff --git a/docs/index.md b/docs/index.md index 70ba539bc..ebabf2851 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,5 +1,8 @@ -# Switchyard - +--- +title: "Home" +description: "Route and translate LLM traffic for coding agents and API clients with NVIDIA NeMo Switchyard." +position: 1 +--- Switchyard routes and translates LLM traffic for coding agents and API clients. It supports OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages. @@ -17,12 +20,13 @@ Use a custom native deployment when needed: switchyard launch codex --model my-route --config routes.toml ``` -The TOML schema is maintained in `crates/switchyard-server/README.md`. +The TOML schema is maintained in +[`crates/switchyard-server/README.md`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/README.md). ## Read More -- [Getting Started](getting_started.md) -- [CLI Reference](cli_reference.md) -- [Routing Overview](routing_algorithms/overview.md) -- [Architecture](architecture.md) -- [Context-Window Handling](operations/context_window.md) +- [Getting Started](/get-started/getting-started) +- [CLI Reference](/reference/cli-reference) +- [Routing Overview](/routing/overview) +- [Architecture](/concepts/architecture) +- [Context-Window Handling](/operations/context-window-handling) diff --git a/docs/known_issues.md b/docs/known_issues.md index d2b3b361e..a157af5d8 100644 --- a/docs/known_issues.md +++ b/docs/known_issues.md @@ -1,5 +1,8 @@ -# Known Issues - +--- +title: "Known Issues in 0.1.0" +description: "Review known limitations and workarounds for NeMo Switchyard 0.1.0." +position: 2 +--- ## 0.1.0 1. Completed Codex Responses tasks may record `0` token usage in `GET /v1/stats` and `GET /v1/routing/stats`. 2. Tool-bearing Codex requests may fail when Switchyard routes them to an upstream that accepts only a fixed set of tool names or schemas. diff --git a/docs/operations/context_window.md b/docs/operations/context_window.md index a6504e157..66d7cd716 100644 --- a/docs/operations/context_window.md +++ b/docs/operations/context_window.md @@ -1,5 +1,8 @@ -# Context-Window Handling - +--- +title: "Context-Window Handling" +description: "Handle upstream context-window overflows with target eviction, fallback routing, and one retry." +position: 1 +--- When an upstream rejects a request because the prompt exceeds the model's context window, Switchyard evicts that target for the current request, reroutes to the configured fallback target, and retries once. If the fallback diff --git a/docs/routing_algorithms/escalation_router_routing.md b/docs/routing_algorithms/escalation_router_routing.md index 90ee9731a..a78fa21a3 100644 --- a/docs/routing_algorithms/escalation_router_routing.md +++ b/docs/routing_algorithms/escalation_router_routing.md @@ -1,4 +1,8 @@ -# Escalation-Router Routing +--- +title: "Escalation-Router Routing" +description: "Start agent tasks on a weak model and latch them to a strong model when an LLM judge detects sustained trouble." +position: 6 +--- Escalation-router routing starts each conversation on a cheaper `weak` model. An LLM judge watches how the work progresses and moves the conversation to a @@ -9,7 +13,7 @@ the task. Use it for multi-turn agent workloads where a weak model can handle routine work but may need rescue after repeated errors, loops, drift, false progress, or premature completion. Unlike -[LLM Classifier Routing](llm_classifier_routing.md), which predicts how +[LLM Classifier Routing](/routing/llm-classifier-routing), which predicts how difficult a request looks, escalation routing judges whether the run is actually going well. @@ -18,7 +22,7 @@ actually going well. The router keeps one state value per conversation: whether that conversation has escalated. Conversations are identified from the stable system prompt and first user message, using the same bounded in-memory store described in -[Sticky Routing](sticky_routing.md). +[Sticky Routing](/routing/sticky-routing). For each turn, Switchyard: @@ -112,7 +116,7 @@ a client-selectable model. If the selected tier exceeds its context window, Switchyard retries once on `fallback_target_on_evict`, which must match one of the configured tier ids (`strong` / `weak` unless the targets set their own `id`). See -[Context-Window Handling](../operations/context_window.md). +[Context-Window Handling](/operations/context-window-handling). ## Useful options @@ -177,12 +181,12 @@ mid-session prefix rewrites can change a deep key and lose the existing latch. ## When not to use escalation routing - **One-shot requests.** There is no trajectory to judge. Use - [LLM Classifier Routing](llm_classifier_routing.md) when the initial request + [LLM Classifier Routing](/routing/llm-classifier-routing) when the initial request should determine the tier. - **Fixed traffic experiments.** Use - [Random Routing](random_routing.md) for A/B splits and gradual traffic ramps. + [Random Routing](/routing/random-routing) for A/B splits and gradual traffic ramps. - **Per-turn stage optimization.** Use - [Stage-Router Routing](stage_router_routing.md) when tool-result signals + [Stage-Router Routing](/routing/stage-router-routing) when tool-result signals should move individual turns in both directions. - **Latency-critical traffic.** Eligible unlatched turns wait for the judge before the selected backend call. @@ -191,8 +195,8 @@ mid-session prefix rewrites can change a deep key and lose the existing latch. ## Related -- [Routing Overview](overview.md): compare all supported routing strategies. -- [Sticky Routing](sticky_routing.md): session-key derivation and affinity +- [Routing Overview](/routing/overview): compare all supported routing strategies. +- [Sticky Routing](/routing/sticky-routing): session-key derivation and affinity behavior. -- [Architecture](../architecture.md): the end-to-end request lifecycle and +- [Architecture](/concepts/architecture): the end-to-end request lifecycle and system boundaries. diff --git a/docs/routing_algorithms/llm_classifier_routing.md b/docs/routing_algorithms/llm_classifier_routing.md index 676e60b73..ce76228cb 100644 --- a/docs/routing_algorithms/llm_classifier_routing.md +++ b/docs/routing_algorithms/llm_classifier_routing.md @@ -1,5 +1,8 @@ -# LLM Classifier Routing - +--- +title: "LLM Classifier Routing" +description: "Use an LLM classifier to route each request to a strong or weak model target." +position: 3 +--- LLM classifier routing asks a classifier model to evaluate each request, then sends the request to a `weak` or `strong` backend. Use it when routing should depend on request content, tool use, context needs, or risk level instead of a @@ -87,7 +90,7 @@ prompt determine the verdict. For a self-hosted strong, weak, or classifier target, configure it like any other OpenAI-compatible endpoint. See -[Self-hosted targets](overview.md#self-hosted-targets). +[Self-hosted targets](/routing/overview#self-hosted-targets). ## Session affinity @@ -99,10 +102,10 @@ reuse that tier before classification, so they skip the classifier call; abstain, low-confidence, missing-signal, and fail-open decisions do not pin. Configure these fields on the `type: deterministic` entry in the `routes:` -bundle. See [Session Affinity](sticky_routing.md) for YAML and -[How session affinity composes](overview.md#how-session-affinity-composes) for +bundle. See [Session Affinity](/routing/sticky-routing) for YAML and +[How session affinity composes](/routing/overview#how-session-affinity-composes) for the interaction with routing decisions. If the per-request classifier cost is too high, use -[Stage-Router Routing](stage_router_routing.md), which can route many turns from tool and +[Stage-Router Routing](/routing/stage-router-routing), which can route many turns from tool and agent-progress signals without an extra classifier call. diff --git a/docs/routing_algorithms/overview.md b/docs/routing_algorithms/overview.md index f6fcc8efe..dbcbbcb5c 100644 --- a/docs/routing_algorithms/overview.md +++ b/docs/routing_algorithms/overview.md @@ -1,5 +1,8 @@ -# Routing Overview - +--- +title: "Overview" +description: "Choose and configure passthrough, random, classifier, stage-router, escalation-router, and sticky routing behavior." +position: 1 +--- The Python CLI loads routing policies from a YAML bundle. Each key under `routes:` becomes a model ID available through OpenAI Chat Completions, Anthropic Messages, and OpenAI Responses requests: @@ -15,13 +18,15 @@ configuration and tuning. | Strategy | Use it when | Route `type` | |---|---|---| -| [Random Routing](random_routing.md) | You need a fixed strong/weak split for A/B tests, baselines, or cost experiments. | `random_routing` | -| [LLM Classifier Routing](llm_classifier_routing.md) | Request content should decide whether a turn needs the weak or strong tier. | `deterministic` | -| [Stage-Router Routing](stage_router_routing.md) | Tool-result and agent-progress signals should route most turns without an extra classifier call. | `stage_router` | -| [Escalation-Router Routing](escalation_router_routing.md) | Start every task on the weak tier and escalate to strong when an LLM judge detects trouble. | `escalation_router` | +| [Random Routing](/routing/random-routing) | You need a fixed strong/weak split for A/B tests, baselines, or cost experiments. | `random_routing` | +| [LLM Classifier Routing](/routing/llm-classifier-routing) | Request content should decide whether a turn needs the weak or strong tier. | `deterministic` | +| [Stage-Router Routing](/routing/stage-router-routing) | Tool-result and agent-progress signals should route most turns without an extra classifier call. | `stage_router` | +| [Escalation-Router Routing](/routing/escalation-router-routing) | Start every task on the weak tier and escalate to strong when an LLM judge detects trouble. | `escalation_router` | -[Session Affinity (Sticky Routing)](sticky_routing.md) is an opt-in feature of -LLM classifier routing, not a standalone routing strategy. +[Session Affinity (Sticky Routing)](/routing/sticky-routing) is an opt-in feature of +LLM classifier routing, not a standalone routing strategy. See +[How session affinity composes](#how-session-affinity-composes) for the exact +behavior. ## Common route shape @@ -145,7 +150,7 @@ decisions never pin. Later turns reuse the tier and skip the classifier call. Pins use a bounded in-process LRU keyed from the stable conversation prefix. They are not shared across workers or restarts. See -[Sticky Routing](sticky_routing.md) for configuration and key derivation. +[Sticky Routing](/routing/sticky-routing) for configuration and key derivation. Random and stage-router routing make a fresh routing decision for each request. The escalation router instead uses affinity as a one-way escalation latch: only @@ -156,5 +161,5 @@ the strong tier is pinned, and the judge runs until that latch fires. The `switchyard-server` binary has a separate TOML schema that explicitly constructs LLM clients, targets, and libsy algorithms. It does not load Python route bundles. See the -[Rust server README](../../crates/switchyard-server/README.md) for its supported +[Rust server README](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/README.md) for its supported configuration. diff --git a/docs/routing_algorithms/random_routing.md b/docs/routing_algorithms/random_routing.md index 17da19eb0..9f8a335c4 100644 --- a/docs/routing_algorithms/random_routing.md +++ b/docs/routing_algorithms/random_routing.md @@ -1,5 +1,8 @@ -# Random Routing - +--- +title: "Random Routing" +description: "Split traffic between two or more model targets with fixed weights in a TOML deployment." +position: 2 +--- Random routing selects one configured target for each request without inspecting the prompt. It supports two or more targets. @@ -77,7 +80,7 @@ therefore use different targets. ## Run the route -After [building the Rust server](../getting_started.md#install), export the +After [installing Switchyard](/get-started/getting-started#install), export the provider credential, validate the configuration, and start the release binary: ```bash @@ -98,8 +101,8 @@ curl http://localhost:4000/v1/chat/completions \ ## Experimental LiteLLM integration The package's -[developer guide](../../examples/experimental/litellm/README.md) is the -canonical documentation for integrating Switchyard with a Dockerized LiteLLM +[developer guide](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/examples/experimental/litellm/README.md) +is the canonical documentation for integrating Switchyard with a Dockerized LiteLLM gateway. Its source-installable client adapts libsy's dictionary contract and demonstrates weighted random routing between model aliases backed by OpenRouter. The guide covers setup, application usage, live end-to-end testing, diff --git a/docs/routing_algorithms/stage_router_routing.md b/docs/routing_algorithms/stage_router_routing.md index f21d0ad0d..e62685f42 100644 --- a/docs/routing_algorithms/stage_router_routing.md +++ b/docs/routing_algorithms/stage_router_routing.md @@ -1,5 +1,8 @@ -# Stage-Router Routing - +--- +title: "Stage-Router Routing" +description: "Route agent turns with progress signals, tool results, and an optional LLM classifier fallback." +position: 5 +--- Stage-router routing sends each request to either a **capable** model or a cheaper **efficient** one, depending on where the agent is in its run. The goal is to spend the capable model on the turns that need it (exploration, error @@ -11,7 +14,7 @@ plus an optional LLM classifier. If the selected backend hits a context-window overflow, the router retries once against `fallback_target_on_evict`; a second overflow surfaces a -context-pool-exhausted error (see [Context-Window Handling](../operations/context_window.md)). +context-pool-exhausted error (see [Context-Window Handling](/operations/context-window-handling)). ## How it works @@ -290,7 +293,7 @@ went the way it did. It appears in per-tier metrics tagged on the decision: - **Single-model deployments.** Use a `model` route instead. - **Probabilistic A/B splits.** Use - [Random Routing](random_routing.md) (`type = "random"`). + [Random Routing](/routing/random-routing) (`type = "random"`). The stage-router's signals are wasted on a fixed traffic ratio. - **No tool-result history.** Stage-router needs meaningful tool-call traffic to populate the tool-result signal. For pure chat-completion workloads every @@ -298,5 +301,5 @@ went the way it did. It appears in per-tier metrics tagged on the decision: ## Related -- [Architecture](../architecture.md): the end-to-end request lifecycle and +- [Switchyard Architecture](/concepts/architecture): the end-to-end request lifecycle and system boundaries. diff --git a/docs/routing_algorithms/sticky_routing.md b/docs/routing_algorithms/sticky_routing.md index 404d798aa..8865d4810 100644 --- a/docs/routing_algorithms/sticky_routing.md +++ b/docs/routing_algorithms/sticky_routing.md @@ -1,5 +1,8 @@ -# Sticky Routing - +--- +title: "Sticky Routing" +description: "Keep multi-turn conversations on one model tier to preserve upstream prompt and KV caches." +position: 4 +--- By default, routing is recomputed on every request, so a single task can hop between models mid-conversation. This throws away the upstream prompt/KV cache and, on the classifier router, pays for an LLM classifier call every turn. diff --git a/docs/stylesheets/nvidia.css b/docs/stylesheets/nvidia.css deleted file mode 100644 index 69d0270e1..000000000 --- a/docs/stylesheets/nvidia.css +++ /dev/null @@ -1,15 +0,0 @@ -/* NVIDIA brand colors */ -:root, -[data-md-color-scheme="default"] { - --md-primary-fg-color: #76B900; - --md-primary-fg-color--light: #93d400; - --md-primary-fg-color--dark: #5a8c00; - --md-accent-fg-color: #76B900; -} - -[data-md-color-scheme="slate"] { - --md-primary-fg-color: #76B900; - --md-primary-fg-color--light: #93d400; - --md-primary-fg-color--dark: #5a8c00; - --md-accent-fg-color: #76B900; -} diff --git a/mkdocs.yml b/mkdocs.yml deleted file mode 100644 index 74f6cc2f4..000000000 --- a/mkdocs.yml +++ /dev/null @@ -1,104 +0,0 @@ -site_name: Switchyard -site_description: Typed, composable LLM routing and format translation for Python -repo_url: https://github.com/NVIDIA-NeMo/Switchyard -repo_name: NVIDIA-NeMo/Switchyard -edit_uri: edit/main/docs/ -extra: - source_ref: !ENV [MKDOCS_SOURCE_REF, main] -copyright: "Copyright © 2025-2026 NVIDIA Corporation" - -docs_dir: docs -site_dir: site -use_directory_urls: true -strict: true - -hooks: - - mkdocs_hooks.py - -nav: - - Home: index.md - - Get Started: - - Getting Started: getting_started.md - - Known Issues: known_issues.md - - Concepts: - - Core Concepts: core_concepts.md - - Architecture: architecture.md - - Routing: - - Overview: routing_algorithms/overview.md - - Random Routing: routing_algorithms/random_routing.md - - LLM Classifier Routing: routing_algorithms/llm_classifier_routing.md - - Sticky Routing: routing_algorithms/sticky_routing.md - - Stage-Router Routing: routing_algorithms/stage_router_routing.md - - Escalation-Router Routing: routing_algorithms/escalation_router_routing.md - - Operations: - - Context-Window Handling: operations/context_window.md - - Reference: - - CLI Reference: cli_reference.md - -exclude_docs: | - internal/** - research/** - vllm-serve-hidden-state.md - -theme: - name: material - palette: - - media: "(prefers-color-scheme: light)" - scheme: default - primary: custom - accent: custom - toggle: - icon: material/weather-sunny - name: Switch to dark mode - - media: "(prefers-color-scheme: dark)" - scheme: slate - primary: custom - accent: custom - toggle: - icon: material/weather-night - name: Switch to light mode - font: - text: Inter - code: JetBrains Mono - features: - - navigation.top - - navigation.sections - - search.suggest - - search.highlight - - content.code.copy - - toc.follow - -plugins: - - search: - lang: en - -markdown_extensions: - - admonition - - pymdownx.details - - pymdownx.superfences: - custom_fences: - - name: mermaid - class: mermaid - format: !!python/name:pymdownx.superfences.fence_code_format - - pymdownx.highlight: - anchor_linenums: true - line_spans: __span - pygments_lang_class: true - - pymdownx.inlinehilite - - pymdownx.snippets: - base_path: [docs] - check_paths: true - - pymdownx.emoji: - emoji_index: !!python/name:material.extensions.emoji.twemoji - emoji_generator: !!python/name:material.extensions.emoji.to_svg - - attr_list - - md_in_html - - tables - - toc: - permalink: true - title: On this page - - pymdownx.tasklist: - custom_checkbox: true - -extra_css: - - stylesheets/nvidia.css diff --git a/mkdocs_hooks.py b/mkdocs_hooks.py deleted file mode 100644 index d9f7bdd1a..000000000 --- a/mkdocs_hooks.py +++ /dev/null @@ -1,310 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -"""MkDocs hooks for repository-relative source links.""" - -import re -from collections.abc import Mapping -from pathlib import Path -from typing import Protocol -from urllib.parse import quote, unquote, urlsplit - -_FENCE_RE = re.compile( - r"^(?P[> \t]*)(?:(?P[-+*]|\d{1,9}[.)])" - r"(?P[ \t]{1,4}))?(?P`{3,}|~{3,})(?P[^\r\n]*)" -) -_BACKTICK_RUN_RE = re.compile(r"`+") -_LINK_DESTINATION_RE = re.compile( - r"(?P\]\(\s*)(?:<(?P[^>\r\n]+)>|(?P[^\s)\r\n]+))" -) - - -class _PageFile(Protocol): - abs_src_path: str - - -class _Page(Protocol): - file: _PageFile - - -class _MkDocsConfig(Protocol): - config_file_path: str - - def __getitem__(self, key: str) -> object: ... - - -def _target_url( - destination: str, - *, - source_path: Path, - docs_dir: Path, - repo_root: Path, - repo_url: str, - source_ref: str, -) -> str | None: - parsed = urlsplit(destination) - if parsed.scheme or parsed.netloc or not parsed.path or parsed.path.startswith(("/", "\\")): - return None - - try: - target = (source_path.parent / unquote(parsed.path)).resolve() - except (OSError, RuntimeError): - return None - - if target.is_relative_to(docs_dir) or not target.is_relative_to(repo_root): - return None - if not target.exists(): - return None - - object_type = "tree" if target.is_dir() else "blob" - relative_target = quote(target.relative_to(repo_root).as_posix(), safe="/") - encoded_ref = quote(source_ref, safe="/") - suffix = destination[len(parsed.path) :] - return f"{repo_url.rstrip('/')}/{object_type}/{encoded_ref}/{relative_target}{suffix}" - - -def _rewrite_link_destinations( - markdown: str, - *, - source_path: Path, - docs_dir: Path, - repo_root: Path, - repo_url: str, - source_ref: str, -) -> str: - def replace(match: re.Match[str]) -> str: - destination = match.group("angled") or match.group("plain") - rewritten = _target_url( - destination, - source_path=source_path, - docs_dir=docs_dir, - repo_root=repo_root, - repo_url=repo_url, - source_ref=source_ref, - ) - if rewritten is None: - return match.group(0) - if match.group("angled") is not None: - rewritten = f"<{rewritten}>" - return f"{match.group('prefix')}{rewritten}" - - return _LINK_DESTINATION_RE.sub(replace, markdown) - - -def _rewrite_outside_code_spans( - markdown: str, - *, - source_path: Path, - docs_dir: Path, - repo_root: Path, - repo_url: str, - source_ref: str, -) -> str: - def rewrite(text: str) -> str: - return _rewrite_link_destinations( - text, - source_path=source_path, - docs_dir=docs_dir, - repo_root=repo_root, - repo_url=repo_url, - source_ref=source_ref, - ) - - output: list[str] = [] - cursor = 0 - while opening := _BACKTICK_RUN_RE.search(markdown, cursor): - output.append(rewrite(markdown[cursor : opening.start()])) - closing = next( - ( - candidate - for candidate in _BACKTICK_RUN_RE.finditer(markdown, opening.end()) - if len(candidate.group(0)) == len(opening.group(0)) - ), - None, - ) - if closing is None: - output.append(rewrite(markdown[opening.start() :])) - return "".join(output) - output.append(markdown[opening.start() : closing.end()]) - cursor = closing.end() - output.append(rewrite(markdown[cursor:])) - return "".join(output) - - -def _container_position(line: str, quote_depth: int) -> tuple[int, int] | None: - cursor = 0 - for _ in range(quote_depth): - padding = 0 - while cursor < len(line) and line[cursor] == " " and padding < 3: - cursor += 1 - padding += 1 - if cursor >= len(line) or line[cursor] != ">": - return None - cursor += 1 - if cursor < len(line) and line[cursor] in " \t": - cursor += 1 - - indentation_start = cursor - while cursor < len(line) and line[cursor] in " \t": - cursor += 1 - indentation = len(line[indentation_start:cursor].expandtabs(4)) - return indentation, cursor - - -def _fence_container(match: re.Match[str]) -> tuple[int, int] | None: - prefix = match.group("prefix") - quote_depth = prefix.count(">") - position = _container_position(prefix, quote_depth) - if position is None: - return None - - indentation, _ = position - list_marker = match.group("list_marker") - if list_marker is not None: - indentation += len(list_marker) + len(match.group("list_padding").expandtabs(4)) - elif indentation <= 3: - indentation = 0 - return quote_depth, indentation - - -def _line_is_in_container(line: str, quote_depth: int, indentation: int) -> bool: - if not line.strip(): - return True - position = _container_position(line, quote_depth) - if position is None: - return False - line_indentation, content_start = position - return not line[content_start:].strip() or line_indentation >= indentation - - -def _is_closing_fence( - match: re.Match[str], - *, - fence_character: str, - fence_length: int, - quote_depth: int, - indentation: int, -) -> bool: - marker = match.group("marker") - if ( - match.group("list_marker") is not None - or marker[0] != fence_character - or len(marker) < fence_length - or match.group("rest").strip() - or match.group("prefix").count(">") != quote_depth - ): - return False - - position = _container_position(match.group("prefix"), quote_depth) - if position is None: - return False - closing_indentation, _ = position - return closing_indentation <= 3 if indentation == 0 else closing_indentation == indentation - - -def rewrite_repository_links( - markdown: str, - *, - source_path: Path, - docs_dir: Path, - repo_root: Path, - repo_url: str, - source_ref: str, -) -> str: - """Rewrite valid links outside docs_dir to repository source URLs.""" - source_path = source_path.resolve() - docs_dir = docs_dir.resolve() - repo_root = repo_root.resolve() - output: list[str] = [] - pending_markdown: list[str] = [] - fence_lines: list[str] = [] - fence_character: str | None = None - fence_length = 0 - fence_quote_depth = 0 - fence_indentation = 0 - - def flush_pending_markdown() -> None: - if not pending_markdown: - return - output.append( - _rewrite_outside_code_spans( - "".join(pending_markdown), - source_path=source_path, - docs_dir=docs_dir, - repo_root=repo_root, - repo_url=repo_url, - source_ref=source_ref, - ) - ) - pending_markdown.clear() - - for line in markdown.splitlines(keepends=True): - if fence_character is not None and not _line_is_in_container( - line, fence_quote_depth, fence_indentation - ): - pending_markdown.extend(fence_lines) - fence_lines.clear() - fence_character = None - fence_length = 0 - fence_quote_depth = 0 - fence_indentation = 0 - - fence = _FENCE_RE.match(line) - if fence_character is None: - container = _fence_container(fence) if fence is not None else None - if fence is None or container is None: - pending_markdown.append(line) - continue - flush_pending_markdown() - marker = fence.group("marker") - fence_character = marker[0] - fence_length = len(marker) - fence_quote_depth, fence_indentation = container - fence_lines.append(line) - continue - - fence_lines.append(line) - if fence is not None and _is_closing_fence( - fence, - fence_character=fence_character, - fence_length=fence_length, - quote_depth=fence_quote_depth, - indentation=fence_indentation, - ): - output.extend(fence_lines) - fence_lines.clear() - fence_character = None - fence_length = 0 - fence_quote_depth = 0 - fence_indentation = 0 - - output.extend(fence_lines) - flush_pending_markdown() - return "".join(output) - - -def on_page_markdown( - markdown: str, - *, - page: _Page, - config: _MkDocsConfig, - files: object, -) -> str: - """Rewrite repository-relative links before MkDocs validates Markdown links.""" - del files - docs_dir = config["docs_dir"] - repo_url = config["repo_url"] - extra = config["extra"] - if not isinstance(docs_dir, str) or not isinstance(repo_url, str): - raise ValueError("MkDocs docs_dir and repo_url must be configured") - if not isinstance(extra, Mapping) or not isinstance(extra.get("source_ref"), str): - raise ValueError("MkDocs extra.source_ref must be configured") - - return rewrite_repository_links( - markdown, - source_path=Path(page.file.abs_src_path), - docs_dir=Path(docs_dir), - repo_root=Path(config.config_file_path).resolve().parent, - repo_url=repo_url, - source_ref=extra["source_ref"], - ) diff --git a/scripts/ci/README.md b/scripts/ci/README.md new file mode 100644 index 000000000..f4098bb04 --- /dev/null +++ b/scripts/ci/README.md @@ -0,0 +1,106 @@ +# CI Options and Workflow Design + +This directory is the maintainer-facing entry point for choosing local validation and designing +GitHub Actions workflows. The defaults below incorporate patterns used in NVIDIA NeMo Data +Designer and NeMo Curator, adapted to Switchyard's required `CI Success` aggregate and public-fork +threat model. + +## Pattern Provenance + +- **NeMo Curator** informed the two-stage preview model: build from the pull request without + credentials, then perform privileged work from trusted workflow metadata. +- **NeMo Data Designer** informed same-repository preview policy, PR-scoped cancellation, bot + comment filtering, serialized publishing, and keeping action/runtime pins current. +- **Switchyard** adds the stable `CI Success` aggregate, job-level permission separation, trusted + default-branch tool configuration, and regression tests for workflow security invariants. + +## Choose a Validation Scope + +Use the selector to turn changed paths into the smallest relevant local command set: + +```bash +# Uncommitted work, including staged and untracked files +python scripts/select_validation.py --changed + +# The whole branch, including committed work +git fetch origin main +python scripts/select_validation.py --base origin/main + +# A proposed path before editing +python scripts/select_validation.py --path .github/workflows/example.yml +``` + +`--json` provides the same plan to editor or agent integrations. The selector never schedules live +provider tests; those require explicit intent and credentials. Before pushing, use the full hard +gate in `.github/workflows/ci.yml` when the change crosses multiple ownership areas or affects +packaging, shared runtime behavior, or the required aggregate. + +## Choose a Workflow Shape + +| Need | Preferred option | Why | +| --- | --- | --- | +| Required PR validation | Call a reusable workflow from `ci.yml` and add its job to `CI Success.needs` | Branch protection keeps one stable required check while new gates cannot become accidentally optional. | +| Path-filtered or advisory validation | Use a separate `pull_request` workflow | It can skip irrelevant changes without making the required aggregate wait for a check that never appears. | +| Secretless preview build | Run on `pull_request` with read-only permissions | Fork code can be checked without exposing secrets or write tokens. | +| Secret-bearing preview or PR comment | Use a trusted `workflow_run` follow-up for same-repository PRs only | Untrusted PR code and trusted credentials never execute in the same job. | +| Production publishing | Trigger from the default branch or a release tag and serialize runs | Prevents an older deployment from racing and overwriting a newer one. | +| Expensive release matrix | Use tag or manual dispatch | Keeps routine PR feedback fast while preserving an explicit full-matrix option. | + +## Required Design Checks + +### Stable required aggregation + +- Every hard gate must be a direct or transitive dependency of `CI Success`. +- Keep branch protection pointed at `CI Success`, not a changing list of matrix jobs. +- Do not put path filters on a required workflow unless a stable always-running aggregate accounts + for the skipped result. + +### Trust boundaries + +- Treat pull-request source, artifacts, cache contents, and artifact filenames as untrusted. +- Derive PR identity, repository identity, and tool versions from trusted event metadata or the + default-branch checkout—not from a downloaded artifact. +- Restrict secret-bearing previews to same-repository PRs. Fork PRs receive validation only. +- A `workflow_run` workflow must exist on the default branch before GitHub will trigger it. + +### Concurrency + +- For PR work, group by workflow plus PR number and cancel superseded runs. +- For production publishing, use one stable deployment group and do not cancel an active publish. +- Avoid grouping only by branch when multiple PRs or deployment targets can share it. + +### Permissions and secrets + +- Start privileged workflows with `permissions: {}` and grant only the permissions each job needs. +- Separate generation, commenting, and repository deployment jobs when they use different secrets + or write scopes. +- Keep `pull-requests: write`, `contents: write`, and publishing tokens out of jobs that execute PR + code. +- When upserting bot comments, match both a unique marker and the expected bot author. + +### Supply chain and toolchains + +- Pin third-party actions to immutable commit SHAs and retain a version comment for update tools and + reviewers. +- Pin command-line tools exactly when their output or hosted behavior is release-sensitive. +- Use runtime versions still supported by the action ecosystem; upgrade intentionally and test the + full workflow path. +- Run `actionlint` for workflow syntax and `zizmor --pedantic .github/workflows` for security. Keep + suppressions narrow, inline, and justified. + +## Review Checklist + +Before merging a workflow change, answer all of these: + +1. Is the check required, advisory, privileged follow-up, or release-only? +2. If required, does failure reach `CI Success`? +3. Can fork-controlled code, metadata, artifacts, or caches reach a secret or write permission? +4. Is superseded PR work cancelled and production work serialized? +5. Are permissions job-scoped and external actions immutable? +6. Are tool versions selected from trusted configuration? +7. Do regression tests encode the security and aggregation invariants? +8. Do `actionlint`, `zizmor`, and the path-selected local validations pass? + +Workflow-specific operational guidance remains with its owner, such as the +[release workflow](../../docs/internal/release_workflow.md) for package publication and the +[Switchyard docs skill](../../.agents/skills/switchyard-docs/SKILL.md) for Fern publishing. diff --git a/scripts/select_validation.py b/scripts/select_validation.py new file mode 100644 index 000000000..56005a69f --- /dev/null +++ b/scripts/select_validation.py @@ -0,0 +1,225 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Select local validation commands from changed repository paths.""" + +from __future__ import annotations + +import argparse +import json +import subprocess +import sys +from collections.abc import Iterable +from dataclasses import asdict, dataclass +from pathlib import Path + + +@dataclass(frozen=True) +class Validation: + """One local validation command and the CI behavior it covers.""" + + name: str + command: str + reason: str + + +VALIDATIONS = { + "python": Validation( + "Python quality gates", + "uv run ruff check . && uv run mypy switchyard", + "Python, tests, scripts, or package metadata changed.", + ), + "tests": Validation( + "Offline test suite", + 'env -u OPENROUTER_API_KEY -u NVIDIA_API_KEY -u OPENAI_API_KEY ' + '-u ANTHROPIC_API_KEY uv run pytest tests/ -v -m "not integration"', + "Runtime code, tests, or package metadata changed.", + ), + "rust": Validation( + "Rust quality gates", + "cargo fmt --all --check && cargo clippy --workspace --all-targets -- -D warnings " + "&& cargo test --workspace", + "Rust sources or Cargo metadata changed.", + ), + "fern": Validation( + "Fern docs", + "make -C docs check && uv run pytest tests/test_fern_docs.py -v -o addopts=", + "Fern content, configuration, or docs workflows changed.", + ), + "readme": Validation( + "README examples", + "OPENAI_API_KEY=sk-test NVIDIA_API_KEY=nvapi-test ANTHROPIC_API_KEY=sk-ant-test " + "uv run pytest tests/readme --markdown-docs README.md -v", + "README examples or their executable coverage changed.", + ), + "getting-started": Validation( + "Getting Started examples", + "OPENAI_API_KEY=sk-test NVIDIA_API_KEY=nvapi-test ANTHROPIC_API_KEY=sk-ant-test " + "uv run pytest tests/getting_started --markdown-docs docs/getting_started.md -v", + "The Getting Started guide or its executable coverage changed.", + ), + "workflow-security": Validation( + "Workflow syntax and security", + "actionlint && zizmor --pedantic .github/workflows", + "A GitHub Actions workflow changed.", + ), + "package": Validation( + "Package build", + "uv build", + "Package metadata, lockfiles, native code, or release automation changed.", + ), +} + + +def _normalize(path: str) -> str: + """Return a repository-relative path with POSIX separators.""" + value = Path(path).as_posix().removeprefix("./") + return value.rstrip("/") + + +def select_validations(paths: Iterable[str]) -> list[Validation]: + """Return the stable, deduplicated validation plan for ``paths``.""" + normalized = {_normalize(path) for path in paths if _normalize(path)} + selected: set[str] = set() + + python_changed = any( + path.endswith(".py") + or path in {"pyproject.toml", "uv.lock"} + or path.startswith("switchyard/") + or path.startswith("switchyard_rust/") + for path in normalized + ) + if python_changed: + selected.update({"python", "tests"}) + + if any( + path in {"Cargo.toml", "Cargo.lock"} + or path.startswith("crates/") + or path.startswith("switchyard_rust/") + for path in normalized + ): + selected.add("rust") + + if any(path == "README.md" or path.startswith("tests/readme/") for path in normalized): + selected.add("readme") + + if any( + path == "docs/getting_started.md" or path.startswith("tests/getting_started/") + for path in normalized + ): + selected.add("getting-started") + + fern_workflows = { + ".github/workflows/ci.yml", + ".github/workflows/fern-docs-ci.yml", + ".github/workflows/fern-docs-preview-build.yml", + ".github/workflows/fern-docs-preview-comment.yml", + ".github/workflows/publish-fern-docs.yml", + } + if any( + path.startswith("docs/") + or path == "tests/test_fern_docs.py" + or path in fern_workflows + for path in normalized + ): + selected.add("fern") + + if any(path.startswith(".github/workflows/") for path in normalized): + selected.add("workflow-security") + + if any( + path in {"pyproject.toml", "uv.lock", "Cargo.toml", "Cargo.lock"} + or path.startswith("crates/") + or path.startswith("scripts/release/") + or path in { + ".github/workflows/package-portability.yml", + ".github/workflows/publish.yml", + } + for path in normalized + ): + selected.add("package") + + return [validation for key, validation in VALIDATIONS.items() if key in selected] + + +def _git_paths(*args: str) -> set[str]: + """Return paths printed by one Git command.""" + result = subprocess.run( + ("git", *args), + check=True, + capture_output=True, + text=True, + ) + return {line for line in result.stdout.splitlines() if line} + + +def changed_paths(base: str | None) -> set[str]: + """Return committed, staged, unstaged, and untracked paths for this checkout.""" + paths: set[str] = set() + if base is not None: + paths.update(_git_paths("diff", "--name-only", f"{base}...HEAD")) + paths.update(_git_paths("diff", "--name-only", "HEAD")) + paths.update(_git_paths("ls-files", "--others", "--exclude-standard")) + return paths + + +def main(argv: list[str] | None = None) -> int: + """Print a focused validation plan for explicit or Git-derived paths.""" + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--path", action="append", default=[], help="Changed path; repeat as needed") + parser.add_argument( + "--changed", + action="store_true", + help="Include staged, unstaged, and untracked paths", + ) + parser.add_argument( + "--base", + help="Also include committed changes since the merge base with this ref", + ) + parser.add_argument("--json", action="store_true", help="Emit machine-readable JSON") + args = parser.parse_args(argv) + + if not args.path and not args.changed and args.base is None: + parser.error("pass --path, --changed, or --base") + + paths = {_normalize(path) for path in args.path} + try: + if args.changed or args.base is not None: + paths.update(changed_paths(args.base)) + except subprocess.CalledProcessError as exc: + print(exc.stderr.strip() or "git path discovery failed", file=sys.stderr) + return 2 + + validations = select_validations(paths) + if args.json: + print( + json.dumps( + { + "paths": sorted(paths), + "validations": [asdict(validation) for validation in validations], + }, + indent=2, + ) + ) + return 0 + + if not paths: + print("No changed paths found.") + return 0 + + print("Changed paths:") + for path in sorted(paths): + print(f" - {path}") + print("\nRecommended validation:") + if not validations: + print(" - No focused gate mapped; review scripts/ci/README.md before pushing.") + for validation in validations: + print(f" - {validation.name}: {validation.command}") + print(f" {validation.reason}") + print("\nLive provider tests are never selected automatically.") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/switchyard/lib/profiles/stage_router_config.py b/switchyard/lib/profiles/stage_router_config.py index 0092bb598..4ae3346ac 100644 --- a/switchyard/lib/profiles/stage_router_config.py +++ b/switchyard/lib/profiles/stage_router_config.py @@ -1,7 +1,10 @@ # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -"""Declarative config for stage_router profiles. See ``docs/stage_router_routing.md``.""" +"""Declarative config for stage_router profiles. + +See ``docs/routing_algorithms/stage_router_routing.mdx``. +""" from __future__ import annotations diff --git a/tests/test_fern_docs.py b/tests/test_fern_docs.py new file mode 100644 index 000000000..0458a730c --- /dev/null +++ b/tests/test_fern_docs.py @@ -0,0 +1,228 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Regression tests for Fern navigation, links, metadata, and legacy redirects.""" + +from __future__ import annotations + +import re +import runpy +from pathlib import Path +from typing import Any + +import yaml + +REPO_ROOT = Path(__file__).resolve().parents[1] +DOCS_ROOT = REPO_ROOT / "docs" +FERN_ROOT = DOCS_ROOT / "fern" +NAVIGATION_PATH = FERN_ROOT / "versions" / "nightly.yml" +DOCS_CONFIG_PATH = FERN_ROOT / "docs.yml" +GENERATOR_PATH = FERN_ROOT / "generate_legacy_redirect_site.py" +BASE_PATH = "/nemo/switchyard" +WORKFLOW_ROOT = REPO_ROOT / ".github" / "workflows" +CI_WORKFLOW_PATH = WORKFLOW_ROOT / "ci.yml" +FERN_CI_WORKFLOW_PATH = WORKFLOW_ROOT / "fern-docs-ci.yml" +FERN_PREVIEW_BUILD_PATH = WORKFLOW_ROOT / "fern-docs-preview-build.yml" +FERN_PREVIEW_COMMENT_PATH = WORKFLOW_ROOT / "fern-docs-preview-comment.yml" +FERN_PUBLISH_PATH = WORKFLOW_ROOT / "publish-fern-docs.yml" + + +def _load_yaml(path: Path) -> dict[str, Any]: + """Load one YAML mapping used by the Fern project.""" + data = yaml.safe_load(path.read_text(encoding="utf-8")) + assert isinstance(data, dict) + return data + + +def _load_workflow(path: Path) -> dict[str, Any]: + """Load a GitHub workflow without treating the `on` key as a YAML boolean.""" + data = yaml.load(path.read_text(encoding="utf-8"), Loader=yaml.BaseLoader) + assert isinstance(data, dict) + return data + + +def _slug(value: str) -> str: + """Return the Fern-style slug for the navigation labels used in this site.""" + return re.sub(r"[^a-z0-9]+", "-", value.lower()).strip("-") + + +def _published_routes() -> set[str]: + """Derive unversioned published routes from nightly navigation.""" + navigation = _load_yaml(NAVIGATION_PATH)["navigation"] + routes: set[str] = set() + for item in navigation: + if "page" in item: + routes.add(f"/{_slug(item['page'])}") + continue + + assert "section" in item + assert "path" not in item, "section paths must not duplicate child page paths" + section_slug = _slug(item["section"]) + for child in item["contents"]: + assert "page" in child + routes.add(f"/{section_slug}/{_slug(child['page'])}") + return routes + + +def _published_doc_paths() -> list[Path]: + """Return authored page paths referenced by nightly navigation.""" + navigation = _load_yaml(NAVIGATION_PATH)["navigation"] + paths: list[Path] = [] + version_dir = NAVIGATION_PATH.parent + + def add_path(relative_path: str) -> None: + paths.append((version_dir / relative_path).resolve()) + + for item in navigation: + if "page" in item: + add_path(item["path"]) + continue + for child in item["contents"]: + add_path(child["path"]) + return paths + + +def test_internal_mdx_links_resolve_to_navigation_routes() -> None: + """Every root-relative doc link must name a route that Fern publishes.""" + routes = _published_routes() + broken: list[str] = [] + patterns = (r"\]\((/[^)\s]+)", r'href="(/[^"]+)"') + for path in _published_doc_paths(): + text = path.read_text(encoding="utf-8") + for pattern in patterns: + for target in re.findall(pattern, text): + route = target.split("#", 1)[0] + if route not in routes: + broken.append(f"{path.relative_to(REPO_ROOT)}: {target}") + assert not broken, "broken Fern links:\n" + "\n".join(broken) + + +def test_published_pages_have_descriptions_and_titled_callouts() -> None: + """Published pages must retain SEO descriptions and converted admonition titles.""" + failures: list[str] = [] + for path in _published_doc_paths(): + text = path.read_text(encoding="utf-8") + frontmatter = text.split("---", 2)[1] + metadata = yaml.safe_load(frontmatter) + if not metadata.get("description"): + failures.append(f"{path.relative_to(REPO_ROOT)}: missing description") + if not metadata.get("title"): + failures.append(f"{path.relative_to(REPO_ROOT)}: missing title") + for tag in re.findall(r"<(?:Note|Warning)(?:\s+[^>]*)?>", text): + if "title=" not in tag: + failures.append(f"{path.relative_to(REPO_ROOT)}: untitled {tag}") + assert not failures, "\n".join(failures) + + +def test_fern_redirects_cover_every_github_pages_route() -> None: + """Fern must map every former MkDocs route to its current navigation route.""" + generator = runpy.run_path(str(GENERATOR_PATH)) + legacy_redirects = generator["LEGACY_REDIRECTS"] + configured = { + item["source"]: item["destination"] + for item in _load_yaml(DOCS_CONFIG_PATH)["redirects"] + } + routes = _published_routes() + + for legacy_path, destination in legacy_redirects.items(): + assert destination in routes + full_destination = f"{BASE_PATH}{destination}" + if not legacy_path: + assert configured[f"{BASE_PATH}/index.html"] == full_destination + continue + source = f"{BASE_PATH}/{legacy_path}" + assert configured[source] == full_destination + assert configured[f"{source}/index.html"] == full_destination + + +def test_legacy_github_pages_site_generation(tmp_path: Path) -> None: + """The generated GitHub Pages site must redirect every legacy route to Fern.""" + generator = runpy.run_path(str(GENERATOR_PATH)) + generator["generate_redirect_site"](tmp_path) + + assert (tmp_path / ".nojekyll").is_file() + for legacy_path, destination in generator["LEGACY_REDIRECTS"].items(): + page = tmp_path / legacy_path / "index.html" + target = f"{generator['CANONICAL_BASE']}{destination}" + assert page.is_file() + content = page.read_text(encoding="utf-8") + assert f'' in content + assert "window.location.search + window.location.hash" in content + + +def test_fern_validation_is_part_of_required_ci_success() -> None: + """The required CI aggregate must fail when reusable Fern validation fails.""" + ci = _load_workflow(CI_WORKFLOW_PATH) + jobs = ci["jobs"] + + assert jobs["fern-docs"]["uses"] == "./.github/workflows/fern-docs-ci.yml" + assert "fern-docs" in jobs["ci-success"]["needs"] + + fern_ci = _load_workflow(FERN_CI_WORKFLOW_PATH) + assert set(fern_ci["on"]) == {"workflow_call"} + + +def test_fern_preview_uses_trusted_identity_and_pr_scoped_concurrency() -> None: + """Preview artifacts must not choose the target PR, preview identifier, or tool version.""" + build = _load_workflow(FERN_PREVIEW_BUILD_PATH) + comment = _load_workflow(FERN_PREVIEW_COMMENT_PATH) + build_text = FERN_PREVIEW_BUILD_PATH.read_text(encoding="utf-8") + comment_text = FERN_PREVIEW_COMMENT_PATH.read_text(encoding="utf-8") + + assert build["concurrency"] == { + "group": "fern-docs-preview-${{ github.event.pull_request.number }}", + "cancel-in-progress": "true", + } + assert comment["concurrency"] == { + "group": "fern-docs-preview-${{ github.event.workflow_run.pull_requests[0].number }}", + "cancel-in-progress": "true", + } + assert "github.event.workflow_run.pull_requests[0].number" in comment_text + assert "github.event.pull_request.head.repo.full_name == github.repository" in build_text + assert "github.event.workflow_run.head_repository.full_name == github.repository" in comment_text + assert "github.event.repository.default_branch" in comment_text + assert "working-directory: ./preview-source/docs/fern" in comment_text + assert '--id "pr-$PR_NUMBER"' in comment_text + assert ".preview-metadata/pr_number" not in comment_text + assert ".preview-metadata/head_ref" not in comment_text + assert '.user.login == "github-actions[bot]"' in comment_text + + +def test_fern_publish_serializes_runs_and_isolates_write_permission() -> None: + """Only the redirect deployment may receive repository write permission.""" + workflow = _load_workflow(FERN_PUBLISH_PATH) + jobs = workflow["jobs"] + + assert workflow["permissions"] == {} + assert workflow["concurrency"] == { + "group": "fern-docs-website", + "cancel-in-progress": "false", + } + assert jobs["publish"]["permissions"] == {"contents": "read"} + assert jobs["redirects"]["permissions"] == { + "actions": "read", + "contents": "write", + } + assert "DOCS_FERN_TOKEN" in yaml.safe_dump(jobs["publish"]) + assert "DOCS_FERN_TOKEN" not in yaml.safe_dump(jobs["redirects"]) + + +def test_fern_workflows_pin_actions_and_use_supported_node() -> None: + """Secret-bearing and docs validation workflows use immutable action pins and Node 24.""" + paths = ( + FERN_CI_WORKFLOW_PATH, + FERN_PREVIEW_BUILD_PATH, + FERN_PREVIEW_COMMENT_PATH, + FERN_PUBLISH_PATH, + ) + action_pattern = re.compile(r"^\s*uses:\s+([^#\s]+)", re.MULTILINE) + + for path in paths: + text = path.read_text(encoding="utf-8") + for action in action_pattern.findall(text): + if action.startswith("./"): + continue + assert re.fullmatch(r"[^@]+@[0-9a-f]{40}", action), f"{path}: {action}" + + for path in (FERN_CI_WORKFLOW_PATH, FERN_PREVIEW_COMMENT_PATH, FERN_PUBLISH_PATH): + assert 'node-version: "24"' in path.read_text(encoding="utf-8") diff --git a/tests/test_mkdocs_repository_links.py b/tests/test_mkdocs_repository_links.py deleted file mode 100644 index 38952c681..000000000 --- a/tests/test_mkdocs_repository_links.py +++ /dev/null @@ -1,209 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -"""Tests for repository-relative links in MkDocs pages.""" - -import re -from pathlib import Path -from types import SimpleNamespace - -from mkdocs_hooks import on_page_markdown, rewrite_repository_links - -REPO_ROOT = Path(__file__).resolve().parents[1] -REPO_URL = "https://github.com/example/switchyard" -SOURCE_REF = "docs-ref" - - -class _Config(dict[str, object]): - def __init__(self, repo_root: Path, docs_dir: Path) -> None: - super().__init__( - docs_dir=str(docs_dir), - repo_url=REPO_URL, - extra={"source_ref": SOURCE_REF}, - ) - self.config_file_path = str(repo_root / "mkdocs.yml") - - -def _repository( - tmp_path: Path, page_relative_path: str = "docs/guides/page.md" -) -> tuple[Path, Path, Path]: - repo_root = tmp_path / "repo" - docs_dir = repo_root / "docs" - page_path = repo_root / page_relative_path - page_path.parent.mkdir(parents=True) - page_path.write_text("# Page\n", encoding="utf-8") - (repo_root / "AGENTS.md").write_text("# Instructions\n", encoding="utf-8") - (repo_root / "examples" / "prometheus").mkdir(parents=True) - (docs_dir / "index.md").write_text("# Home\n", encoding="utf-8") - return repo_root, docs_dir, page_path - - -def _rewrite(markdown: str, repo_root: Path, docs_dir: Path, page_path: Path) -> str: - return rewrite_repository_links( - markdown, - source_path=page_path, - docs_dir=docs_dir, - repo_root=repo_root, - repo_url=REPO_URL, - source_ref=SOURCE_REF, - ) - - -def test_repository_file_and_directory_links_use_blob_and_tree_urls(tmp_path: Path) -> None: - repo_root, docs_dir, page_path = _repository(tmp_path) - markdown = "[file](../../AGENTS.md)\n[directory](../../examples/prometheus/)\n" - - rewritten = _rewrite(markdown, repo_root, docs_dir, page_path) - - assert rewritten == ( - f"[file]({REPO_URL}/blob/{SOURCE_REF}/AGENTS.md)\n" - f"[directory]({REPO_URL}/tree/{SOURCE_REF}/examples/prometheus)\n" - ) - - -def test_hook_resolves_links_from_a_nested_page_and_uses_mkdocs_config(tmp_path: Path) -> None: - repo_root, docs_dir, page_path = _repository(tmp_path, "docs/routing/deep/page.md") - page = SimpleNamespace(file=SimpleNamespace(abs_src_path=str(page_path))) - - rewritten = on_page_markdown( - "[instructions](../../../AGENTS.md)", - page=page, - config=_Config(repo_root, docs_dir), - files=object(), - ) - - assert rewritten == f"[instructions]({REPO_URL}/blob/{SOURCE_REF}/AGENTS.md)" - - -def test_repository_links_preserve_query_strings_and_fragments(tmp_path: Path) -> None: - repo_root, docs_dir, page_path = _repository(tmp_path) - - rewritten = _rewrite( - "[instructions](../../AGENTS.md?plain=1&view=raw#setup)", - repo_root, - docs_dir, - page_path, - ) - - assert rewritten == ( - f"[instructions]({REPO_URL}/blob/{SOURCE_REF}/AGENTS.md?plain=1&view=raw#setup)" - ) - - -def test_repository_links_in_fenced_code_blocks_are_not_rewritten(tmp_path: Path) -> None: - repo_root, docs_dir, page_path = _repository(tmp_path) - markdown = ( - "```markdown\n" - "[backtick fence](../../AGENTS.md)\n" - "```\n" - "~~~markdown\n" - "[tilde fence](../../AGENTS.md)\n" - "~~~\n" - "[live link](../../AGENTS.md)\n" - ) - - rewritten = _rewrite(markdown, repo_root, docs_dir, page_path) - - assert rewritten == markdown.replace( - "[live link](../../AGENTS.md)", - f"[live link]({REPO_URL}/blob/{SOURCE_REF}/AGENTS.md)", - ) - - -def test_repository_links_in_blockquote_fences_are_not_rewritten(tmp_path: Path) -> None: - repo_root, docs_dir, page_path = _repository(tmp_path) - markdown = ( - "> ~~~markdown\n" - "> [fenced link](../../AGENTS.md)\n" - "> ~~~\n" - "> Ordinary prose with ~~~ inline.\n" - "> [live link](../../AGENTS.md)\n" - ) - - rewritten = _rewrite(markdown, repo_root, docs_dir, page_path) - - assert rewritten == markdown.replace( - "[live link](../../AGENTS.md)", - f"[live link]({REPO_URL}/blob/{SOURCE_REF}/AGENTS.md)", - ) - - -def test_repository_links_in_list_item_fences_are_not_rewritten(tmp_path: Path) -> None: - repo_root, docs_dir, page_path = _repository(tmp_path) - markdown = ( - "- Example:\n" - "\n" - " ~~~markdown\n" - " [fenced link](../../AGENTS.md)\n" - " ~~~\n" - "\n" - " Ordinary prose with ~~~ inline.\n" - " [live link](../../AGENTS.md)\n" - ) - - rewritten = _rewrite(markdown, repo_root, docs_dir, page_path) - - assert rewritten == markdown.replace( - "[live link](../../AGENTS.md)", - f"[live link]({REPO_URL}/blob/{SOURCE_REF}/AGENTS.md)", - ) - - -def test_indented_fence_markers_do_not_hide_outdented_prose_links(tmp_path: Path) -> None: - repo_root, docs_dir, page_path = _repository(tmp_path) - markdown = ( - " ~~~\n" - "[live link](../../AGENTS.md)\n" - " ~~~\n" - ) - - rewritten = _rewrite(markdown, repo_root, docs_dir, page_path) - - assert rewritten == markdown.replace( - "[live link](../../AGENTS.md)", - f"[live link]({REPO_URL}/blob/{SOURCE_REF}/AGENTS.md)", - ) - - -def test_missing_repository_target_remains_relative(tmp_path: Path) -> None: - repo_root, docs_dir, page_path = _repository(tmp_path) - markdown = "[missing](../../missing.md?plain=1#section)" - - assert _rewrite(markdown, repo_root, docs_dir, page_path) == markdown - - -def test_existing_target_outside_repository_remains_relative(tmp_path: Path) -> None: - repo_root, docs_dir, page_path = _repository(tmp_path) - (tmp_path / "outside.md").write_text("# Outside\n", encoding="utf-8") - markdown = "[outside](../../../outside.md)" - - assert _rewrite(markdown, repo_root, docs_dir, page_path) == markdown - - -def test_links_inside_docs_dir_and_non_relative_links_are_unchanged(tmp_path: Path) -> None: - repo_root, docs_dir, page_path = _repository(tmp_path) - markdown = ( - "[docs page](../index.md)\n" - "[external](https://example.com/docs)\n" - "[page anchor](#section)\n" - ) - - assert _rewrite(markdown, repo_root, docs_dir, page_path) == markdown - - -def test_docs_do_not_hard_code_same_repository_blob_or_tree_links() -> None: - config = (REPO_ROOT / "mkdocs.yml").read_text(encoding="utf-8") - repo_url_match = re.search(r"^repo_url:\s*(\S+)\s*$", config, flags=re.MULTILINE) - assert repo_url_match is not None - repo_url = repo_url_match.group(1).strip("'\"").rstrip("/") - source_link = re.compile(rf"{re.escape(repo_url)}/(?:blob|tree)/") - markdown_paths = [REPO_ROOT / "README.md", *(REPO_ROOT / "docs").rglob("*.md")] - - offenders = [ - f"{path.relative_to(REPO_ROOT)}:{line_number}: {line.strip()}" - for path in markdown_paths - for line_number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), start=1) - if source_link.search(line) - ] - - assert not offenders, "Hard-coded same-repository source links:\n" + "\n".join(offenders) diff --git a/tests/test_select_validation.py b/tests/test_select_validation.py new file mode 100644 index 000000000..0a5010c48 --- /dev/null +++ b/tests/test_select_validation.py @@ -0,0 +1,68 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Tests for path-based local validation selection and its contributor guidance.""" + +from __future__ import annotations + +import runpy +from pathlib import Path +from typing import Any + +REPO_ROOT = Path(__file__).resolve().parents[1] +SCRIPT_PATH = REPO_ROOT / "scripts" / "select_validation.py" +GUIDE_PATH = REPO_ROOT / "scripts" / "ci" / "README.md" + + +def _load_script() -> dict[str, Any]: + """Load the validation selector without making scripts a Python package.""" + return runpy.run_path(str(SCRIPT_PATH)) + + +def _names(*paths: str) -> set[str]: + """Return validation names selected for the supplied paths.""" + module = _load_script() + return {item.name for item in module["select_validations"](paths)} + + +def test_fern_workflow_selects_docs_and_security_validation() -> None: + """Fern workflow edits need both product checks and workflow security checks.""" + names = _names(".github/workflows/fern-docs-preview-comment.yml") + assert names == {"Fern docs", "Workflow syntax and security"} + + +def test_native_or_package_changes_select_all_affected_gates() -> None: + """Native package metadata must cover Python, Rust, tests, and package construction.""" + names = _names("Cargo.lock", "switchyard_rust/__init__.py") + assert names == { + "Offline test suite", + "Package build", + "Python quality gates", + "Rust quality gates", + } + + +def test_docs_only_change_does_not_select_runtime_or_live_tests() -> None: + """Ordinary docs changes stay focused and never opt into provider calls.""" + validations = _load_script()["select_validations"](["docs/operations/context_window.md"]) + assert [item.name for item in validations] == ["Fern docs"] + assert all("integration" not in item.command for item in validations) + + +def test_ci_guide_records_required_security_invariants() -> None: + """The guide must retain the cross-repository workflow design lessons.""" + guide = GUIDE_PATH.read_text(encoding="utf-8") + for invariant in ( + "Stable required aggregation", + "Trust boundaries", + "Concurrency", + "Permissions and secrets", + "Supply chain and toolchains", + "same-repository PRs", + "CI Success", + "actionlint", + "zizmor", + "NeMo Curator", + "NeMo Data Designer", + ): + assert invariant in guide