microsoft · danielmeppiel · May 10, 2026 · May 10, 2026 · May 10, 2026 · May 10, 2026
@@ -148,7 +148,12 @@ it via tool calls before emitting it as a finding. The procedure:
 5. **Probe the test tree** with `view` / `grep` / `glob`:
    - Look in `tests/unit/<area>/` for unit tests on the touched module.
    - Look in `tests/integration/` for integration tests on the touched
-     command or flow.
+     command or flow. New integration tests must follow the marker
+     placement contract in
+     [`.apm/instructions/tests.instructions.md`](../instructions/tests.instructions.md);
+     flag ungated live-network or runtime-binary calls in
+     `tests/integration/` as `recommended` regardless of whether the
+     test self-skips at runtime.
    - Search for the specific symbol, error string, or flag name being
      changed. Absence of ANY hit on the changed symbol is a strong
      signal of a coverage gap.

@@ -107,3 +107,68 @@ production code must follow (see
 - **Targeted runs during iteration.** Run the specific test file first
   (`uv run pytest tests/unit/install/test_X.py -x`) before running the
   full suite (`uv run pytest tests/unit tests/test_console.py`).
+
+## Integration tests: placement and markers
+
+The integration suite uses **declarative gating** via pytest markers,
+not per-file orchestrator enumeration. Adding a new integration test
+is two steps.
+
+### Procedure
+
+1. Drop the file under `tests/integration/test_<feature>.py`.
+2. At the top of the module, declare the runtime / network / E2E
+   prerequisites as a single `pytestmark`:
+
+   ```python
+   import pytest
+
+   pytestmark = pytest.mark.requires_network_integration
+   # OR for multiple prerequisites:
+   pytestmark = [
+       pytest.mark.requires_e2e_mode,
+       pytest.mark.requires_runtime_codex,
+   ]
+   ```
+
+That is it. The orchestrator (`scripts/test-integration.sh`) and the
+CI integration job collect everything under `tests/integration/` in
+a single `pytest` invocation; markers are honored automatically.
+
+### Marker selection
+
+Pick the marker that matches the **strongest** prerequisite the test
+has. The full registry lives in `pyproject.toml` under
+`[tool.pytest.ini_options].markers` and is documented (with the
+opt-in commands) in
+[`docs/src/content/docs/contributing/integration-testing.md`](../../docs/src/content/docs/contributing/integration-testing.md).
+Quick map for the common cases:
+
+| Test prerequisite                            | Marker                          |
+|----------------------------------------------|---------------------------------|
+| Real HTTP to APM-owned services              | `requires_network_integration`  |
+| Real codex / copilot / llm runtime binary    | `requires_runtime_<name>`       |
+| Downloads runtimes; full E2E flow            | `requires_e2e_mode`             |
+| GitHub / ADO token required                  | `requires_github_token` / `requires_ado_pat` |
+| Paid or third-party external service         | `live` (deselected by default)  |
+| Performance measurement                      | `benchmark` (deselected by default) |
+| Hermetic (mocks all I/O)                     | *no marker required*            |
+
+Need a marker that does not exist yet? Register it in
+`pyproject.toml` AND add a row to the docs registry table in the
+same PR. Both must stay in sync.
+
+### Anti-patterns (will land as `recommended` findings on review)
+
+- **Editing `scripts/test-integration.sh` per file.** The orchestrator
+  enumerates the directory, not the files. Per-file blocks are drift
+  by construction.
+- **Runtime self-skips inside the test body.** A bare
+  `if not os.getenv("APM_E2E_TESTS"): pytest.skip(...)` runs before
+  collection-time gating and weakens the contract. Use
+  module-level `pytestmark` instead -- declarative gating is the
+  single source of truth.
+- **Reading the gate env var inside test logic.** If your test
+  reads `APM_RUN_INTEGRATION_TESTS` to branch behaviour, the marker
+  is wrong (or missing). The marker is the gate; the test body
+  should assume the gate already passed.
@@ -148,7 +148,12 @@ it via tool calls before emitting it as a finding. The procedure:
 5. **Probe the test tree** with `view` / `grep` / `glob`:
    - Look in `tests/unit/<area>/` for unit tests on the touched module.
    - Look in `tests/integration/` for integration tests on the touched
-     command or flow.
+     command or flow. New integration tests must follow the marker
+     placement contract in
+     [`.apm/instructions/tests.instructions.md`](../../.apm/instructions/tests.instructions.md);
+     flag ungated live-network or runtime-binary calls in
+     `tests/integration/` as `recommended` regardless of whether the
+     test self-skips at runtime.
    - Search for the specific symbol, error string, or flag name being
      changed. Absence of ANY hit on the changed symbol is a strong
      signal of a coverage gap.

@@ -1,6 +1,6 @@
 <!-- Generated by APM CLI from .apm/ primitives -->
-<!-- Build ID: 3a5991d7ca52 -->
-<!-- APM Version: 0.11.0 -->
+<!-- Build ID: 791e09362ff0 -->
+<!-- APM Version: 0.12.4 -->
 
 <!-- Source: .apm/instructions/linting.instructions.md -->
 # Linting (canonical contract)
@@ -49,4 +49,4 @@ skills; honor this instruction before invoking them.
 
 ---
 *This file was generated by APM CLI. Do not edit manually.*
-*To regenerate: `specify apm compile`*
+*To regenerate: `apm compile`*
@@ -107,3 +107,68 @@ production code must follow (see
 - **Targeted runs during iteration.** Run the specific test file first
   (`uv run pytest tests/unit/install/test_X.py -x`) before running the
   full suite (`uv run pytest tests/unit tests/test_console.py`).
+
+## Integration tests: placement and markers
+
+The integration suite uses **declarative gating** via pytest markers,
+not per-file orchestrator enumeration. Adding a new integration test
+is two steps.
+
+### Procedure
+
+1. Drop the file under `tests/integration/test_<feature>.py`.
+2. At the top of the module, declare the runtime / network / E2E
+   prerequisites as a single `pytestmark`:
+
+   ```python
+   import pytest
+
+   pytestmark = pytest.mark.requires_network_integration
+   # OR for multiple prerequisites:
+   pytestmark = [
+       pytest.mark.requires_e2e_mode,
+       pytest.mark.requires_runtime_codex,
+   ]
+   ```
+
+That is it. The orchestrator (`scripts/test-integration.sh`) and the
+CI integration job collect everything under `tests/integration/` in
+a single `pytest` invocation; markers are honored automatically.
+
+### Marker selection
+
+Pick the marker that matches the **strongest** prerequisite the test
+has. The full registry lives in `pyproject.toml` under
+`[tool.pytest.ini_options].markers` and is documented (with the
+opt-in commands) in
+[`docs/src/content/docs/contributing/integration-testing.md`](../../docs/src/content/docs/contributing/integration-testing.md).
+Quick map for the common cases:
+
+| Test prerequisite                            | Marker                          |
+|----------------------------------------------|---------------------------------|
+| Real HTTP to APM-owned services              | `requires_network_integration`  |
+| Real codex / copilot / llm runtime binary    | `requires_runtime_<name>`       |
+| Downloads runtimes; full E2E flow            | `requires_e2e_mode`             |
+| GitHub / ADO token required                  | `requires_github_token` / `requires_ado_pat` |
+| Paid or third-party external service         | `live` (deselected by default)  |
+| Performance measurement                      | `benchmark` (deselected by default) |
+| Hermetic (mocks all I/O)                     | *no marker required*            |
+
+Need a marker that does not exist yet? Register it in
+`pyproject.toml` AND add a row to the docs registry table in the
+same PR. Both must stay in sync.
+
+### Anti-patterns (will land as `recommended` findings on review)
+
+- **Editing `scripts/test-integration.sh` per file.** The orchestrator
+  enumerates the directory, not the files. Per-file blocks are drift
+  by construction.
+- **Runtime self-skips inside the test body.** A bare
+  `if not os.getenv("APM_E2E_TESTS"): pytest.skip(...)` runs before
+  collection-time gating and weakens the contract. Use
+  module-level `pytestmark` instead -- declarative gating is the
+  single source of truth.
+- **Reading the gate env var inside test logic.** If your test
+  reads `APM_RUN_INTEGRATION_TESTS` to branch behaviour, the marker
+  is wrong (or missing). The marker is the gate; the test body
+  should assume the gate already passed.
@@ -22,6 +22,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Integration tests now use marker-driven discovery: 21 `pytestmark = pytest.mark.skipif(...)` chains across `tests/integration/` are replaced with declarative `requires_*` markers, with precondition logic centralized in `tests/integration/conftest.py` and auto-skipping at collection time. PR1 of #1166. (#1167)
 - Integration test apm-binary resolution now prefers the local build (`./dist/apm-<os>-<arch>/apm`) over a system-wide `apm` on `PATH`, so contributors validating the binary under test are not silently shadowed by a global install; the bearer-token marker (`requires_ado_bearer`) discards the captured JWT immediately and persists only the boolean outcome. (#1167)
 - `scripts/test-integration.sh` is now a thin orchestrator: it builds/locates the apm binary, sets up runtimes and tokens, then invokes `pytest tests/integration/` exactly once. The 28 per-file pytest enumerations were removed; the marker registry handles per-test gating, and new test files dropped into `tests/integration/` are picked up automatically. PR2 of #1166. (#1247)
+- Integration-test marker procedure codified as `.apm/instructions/tests.instructions.md` (wired into `test-coverage-expert` persona) and guarded by a regression-trap test that asserts `pyproject.toml`, `tests/integration/conftest.py::_MARKER_CHECKS`, the docs registry table, and the instructions rule stay in sync. (#1166)
 
 ### Fixed
 

@@ -45,7 +45,7 @@ local_deployed_file_hashes:
   .github/agents/oss-growth-hacker.agent.md: sha256:1cd56bb78ab37d52c50e45ab69d759f775cd49cdf35981b3dc6c4004315c6b83
   .github/agents/python-architect.agent.md: sha256:7587ee7c684c61046a83dfa1b7e39d1345f2f119c3395478e3ca2dbbaaaff0e9
   .github/agents/supply-chain-security-expert.agent.md: sha256:8fb8cc426d6af17ba084a28b3f026c2b475b62e3ca63ed2f88b83bd823f877af
-  .github/agents/test-coverage-expert.agent.md: sha256:4df107de6179b5237fa9300582921e7fcb439928649601f0fef2d3b9275cea40
+  .github/agents/test-coverage-expert.agent.md: sha256:bc588d89530362469502bfbea788df892a9a0b00e630cd0f3926d3dfd2c2a9e2
   .github/instructions/changelog.instructions.md: sha256:1e51ec4c74e847967962bd279dc4c6e582c5d3578490b3c28d5f3acd3e05f73e
   .github/instructions/cicd.instructions.md: sha256:9c0fafc74f743aa97e5adba2168d66c9e3a327b135065e3b804bdbb5f04cda5d
   .github/instructions/cli.instructions.md: sha256:8e39e8d5047ce88575cb02f87c2bcede584dfef258bd86f7466c7badf136541a
@@ -54,4 +54,4 @@ local_deployed_file_hashes:
   .github/instructions/integrators.instructions.md: sha256:b151e0438088d2c0b636dfc28532ecf43c3b51e5f1070a354b8d5b57c345e335
   .github/instructions/linting.instructions.md: sha256:312acd32353567834ec9f4f246710a47a991729a11c0380aa6a010b63de607eb
   .github/instructions/python.instructions.md: sha256:45173f778eddc126c37c7ace96acd0e17adb1895031eec134ec0754638d3ba37
-  .github/instructions/tests.instructions.md: sha256:4c6335e3373f9735778a05913f2d8ef250d118f8c5305e70ba407e578a525ef7
+  .github/instructions/tests.instructions.md: sha256:b527ccaecf0e92f74d300fc9027f1bc49bb43d8ddcdd36338c1556fcde0d8b2d