refactor(hooks)!: retire the telemetry hook in favor of opt-in copilot-otel-metrics capture

[Bill Berry (WilliamBerryiii)]

Pull Request

Description

This retires the local telemetry hook and makes the copilot-otel-metrics skill the single supported path for Copilot telemetry capture.

The privacy argument is the reason for the change, not a side effect of it. The hook manifest registered one command against eleven lifecycle events, so installing the package enrolled the developer in collection across every prompt, every tool call, and every subagent dispatch. Consent was a property of installation rather than a decision the developer made about their own session data.

The replacement inverts that. Nothing is captured until the developer enables an editor setting, and the skill presents the exact settings diff before writing it. Someone who installs the package and never opts in emits nothing at all.

The hook also wrote a raw input artifact to local disk before any filtering decision, which put the privacy boundary after collection. The skill puts it first: otel-collector-local.yaml applies a fail-closed redaction processor across the trace, metric, and log pipelines, with allow_all_keys: false making the allow-list authoritative and blocked_values running as a second independent pass over whatever survives. An attribute introduced by a future extension release is dropped by default rather than stored by default.

That filtering is measured rather than asserted. test_collector_carriers.py starts the pinned Collector with the shipped configuration, places a distinct marker in every OTLP carrier the signal model can transport, and records what actually survives. A carrier that opens fails the test rather than passing quietly.

Residual exposure is documented rather than papered over. SECURITY.md is a STRIDE model across seven trust boundaries with sixteen registered gaps. G-INF-1 states plainly that spans still carry prompt text, tool call arguments and results, and system instructions on a configuration left at its documented default, and that the plaintext loopback hop remains outside the skill's control.

There is a reliability dimension as well. Hooks are fail-closed, so a telemetry defect could deny every tool call in a session, which is what #2496 reports. Ancillary observability code held the power to halt all work, and removing it from the hook path removes that class of failure.

Hook retirement

Removed telemetry.json and its entire owned runtime: three PowerShell entry points, three bash entry points, the Python core, the static report template, and the test package including its fuzz corpus. The repository now ships zero hook manifests, and the validator accepts that state.

• Deleted the dedicated local telemetry guide, with copilot-otel-metrics.md as the replacement
• Updated hooks.md to state that the repository ships no hook manifest today while keeping the generic contract supported
• Rewrote the telemetry paragraph in TRANSPARENCY-NOTE.md to describe opt-in capture

Skill as the supported path

Added the files that complete the skill package, including pyproject.toml, uv.lock, the input-policy and settings-upsert helpers, the local Collector configuration, and a six-file test package covering carrier behavior, local configuration, helper guard placement, Azure templates, settings mutation, and fuzzing.

• Committed uv.lock so dependency scanning can resolve and patch the skill
• Pinned both container images by manifest digest rather than mutable tag

Packaging and fixtures

• Dropped the hooks key and its maturity entry from the affected package recipes
• Registered copilot-otel-metrics in the experimental package
• Generalized plugin and extension test fixtures from the real manifest to a synthetic sample-hook fixture

Related Issue(s)

Resolves #2496
Refs #2717
Supersedes #2562

Type of Change

Select all that apply:

Code & Documentation:

• Bug fix (non-breaking change fixing an issue)
• New feature (non-breaking change adding functionality)
• Breaking change (fix or feature causing existing functionality to change)
• Documentation update

Infrastructure & Configuration:

• GitHub Actions workflow
• Linting configuration (markdown, PowerShell, etc.)
• Security configuration
• DevContainer configuration
• Dependency update

AI Artifacts:

• Reviewed contribution with hve-builder and addressed all actionable findings
• Copilot instructions (.github/instructions/*.instructions.md)
• Copilot prompt (.github/prompts/*.prompt.md)
• Copilot agent (.github/agents/*.agent.md)
• Copilot skill (.github/skills/*/SKILL.md)
• Copilot hook (.github/hooks/*/*.json)
• Eval spec added/updated for changed AI artifacts (evals/)

Note for AI Artifact Contributors:

• Agents: Research, indexing/referencing other project (using standard VS Code GitHub Copilot/MCP tools), planning, and general implementation agents likely already exist. Review .github/agents/ before creating new ones.
• Skills: Must include both bash and PowerShell scripts. See Skills.
• Model Versions: Contributions MUST target models listed in the model catalog (scripts/linting/model-catalog.json) whose provider appears in providerAllowlist and whose status is ga or preview. Run npm run lint:models to validate references.
• See Agents Not Accepted and Model Version Requirements.

Other:

• Script/automation (.ps1, .sh, .py)
• Other (please describe):

Sample Prompts (for AI Artifact Contributions)

User Request:

Set up local Copilot telemetry capture so I can see my token usage.

Execution Flow:

The skill resolves one of four modes: local-setup, local-stack, org-distribution, or azure-capture. For local capture it explains what the editor setting enables, presents the exact settings.json diff before writing, and applies the change through settings_upsert.py, which validates the schema, refuses the write when unrelated settings would change or the result would not parse, takes a backup first, restores on failure, and records a redacted audit entry. It then brings up the Collector and Grafana stack from compose.yaml, both images pinned by digest, with the fail-closed redaction allow-list applied before anything is stored.

Output Artifacts:

An updated global settings.json with a backup alongside it, a running local stack, and a Grafana dashboard. No repository files are created.

Success Indicators:

verify.py confirms the pipeline is receiving signals, baseline.py captures a snapshot of metric and service names, and the Grafana dashboard renders usage and cost panels. Nothing is captured before the setting is enabled.

For detailed contribution requirements, see:

• Common Standards: docs/contributing/ai-artifacts-common.md - Shared standards for XML blocks, markdown quality, RFC 2119, validation, and testing
• Agents: docs/contributing/custom-agents.md - Agent configurations with tools and behavior patterns
• Prompts: docs/contributing/prompts.md - Workflow-specific guidance with template variables
• Instructions: docs/contributing/instructions.md - Technology-specific standards with glob patterns
• Skills: docs/contributing/skills.md - Task execution utilities with cross-platform scripts

Testing

Automated validation run locally:

Check	Result
npm run lint:hooks	Passed, zero manifests found
npm run plugin:validate	Passed, 10 plugins with no orphaned hook entries
npm run validate:skills	Passed, 72 skills with 0 errors and 0 warnings
npm run docs:generate:check	Passed, 0 updates and 236 unchanged
component-copy.Tests.ps1	Passed, 45 of 45
PluginHelpers.Materialization.Tests.ps1	Passed

Diff-based assessment: searched the repository for references to every deleted path and filename. Three matches were found and all three are false positives naming the otel-collector service and the copilot-otel-collector container rather than the deleted collector script. No real dangling references remain.

Security analysis: no secrets appear in the diff. The Application Insights connection string is documented as operator-supplied and never written into a generated file. Both container images are pinned by manifest digest. Helper scripts route HTTP through a shared input policy that refuses non-http schemes, credentials in the authority, and non-loopback hosts without an explicit opt-in.

Two pre-existing failures in scripts/tests/docs/Generate-AssetDocs.Tests.ps1 were reproduced on a clean checkout of the base commit in an isolated worktree, confirming they are not introduced here. Four failures in the wider scripts/tests/plugins/ directory remain unattributed; the suite modified by this PR passes in isolation.

Manual testing was not performed.

Checklist

Required Checks

• Documentation is updated (if applicable)
• Files follow existing naming conventions
• Changes are backwards compatible (if applicable)
• Tests added for new functionality (if applicable)

AI Artifact Contributions

• Used hve-builder review mode to review contribution
• Addressed all actionable findings from the hve-builder review
• Verified contribution follows common standards and type-specific requirements

Required Local Checks

The following local-safe validation commands must pass before merging:

• Local validation aggregate: npm run validate:local
• Documentation validation (if docs changed): npm run validate:docs
• Spell checking: npm run spell-check
• Link validation: npm run lint:md-links

Security Considerations

• This PR does not contain any sensitive or NDA information
• Any new dependencies have been reviewed for security issues
• Security-related scripts follow the principle of least privilege

GHCP Maturity

Warning

This PR includes experimental GHCP artifacts that may have breaking changes.

• .github/skills/experimental/copilot-otel-metrics/SKILL.md

Additional Notes

Two capabilities are removed rather than replaced, and the change should be reviewed with that in mind:

• Automatic hook-driven collection across session lifecycle events. Capture is now opt-in through an editor setting.
• Local JSONL to static HTML reporting. Reporting moves to Grafana dashboards and query helpers.

Hook events are application-level and OTel traces are instrumentation-level, so the replacement is not a like-for-like substitute for lifecycle event capture. Fleet and organization capture through the Azure path is new capability with no hook equivalent.

This supersedes #2562, which proposed keeping a reduced three-hook spine with sessionStart as the opt-in consent gate. The position taken here is that an editor setting is a better consent gate than a hook, because it is owned by the developer, visible in their own settings, and revocable without touching the package. The trade-off that issue raises still applies and is accepted: some report surfaces become dependent on what the developer chose to enable, which is the intended consequence of moving from ambient to opt-in collection.

.github/plugin/marketplace.json is also modified by open PR #2712, so one of the two will need a conflict resolution pass.

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

Scorecard details

Package	Version	Score	Details
pip/atheris	3.1.0	🟢 6.1	Details Check Score Reason Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Packaging ⚠️ -1 packaging workflow not detected Token-Permissions 🟢 10 GitHub workflow tokens follow principle of least privilege Code-Review 🟢 7 Found 23/30 approved changesets -- score normalized to 7 Maintained ⚠️ 2 3 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 2 SAST ⚠️ 0 no SAST tool detected Binary-Artifacts 🟢 10 no binaries found in the repo CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected Fuzzing 🟢 10 project is fuzzed Signed-Releases ⚠️ -1 no releases found License 🟢 10 license file detected Pinned-Dependencies ⚠️ 0 dependency not pinned by hash detected -- score normalized to 0 Branch-Protection ⚠️ 0 branch protection not enabled on development/release branches Security-Policy 🟢 10 security policy file detected
pip/colorama	0.4.6	Unknown	Unknown
pip/coverage	7.15.4	Unknown	Unknown
pip/iniconfig	2.3.0	Unknown	Unknown
pip/packaging	26.3	Unknown	Unknown
pip/pluggy	1.6.0	Unknown	Unknown
pip/pygments	2.20.0	Unknown	Unknown
pip/pytest	9.1.1	Unknown	Unknown
pip/pytest-cov	7.1.0	Unknown	Unknown
pip/pyyaml	6.0.3	Unknown	Unknown
pip/ruff	0.16.3	Unknown	Unknown
pip/tomli	2.4.1	Unknown	Unknown

Scanned Files

• .github/hooks/shared/telemetry/uv.lock
• .github/skills/experimental/copilot-otel-metrics/uv.lock

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.91%. Comparing base (5b2119e) to head (a22b48c).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2719      +/-   ##
==========================================
- Coverage   83.17%   82.91%   -0.26%     
==========================================
  Files         180      181       +1     
  Lines       32201    32553     +352     
  Branches       25       25              
==========================================
+ Hits        26782    26992     +210     
- Misses       5416     5558     +142     
  Partials        3        3              

Flag	Coverage Δ
docusaurus	89.92% <ø> (ø)
pester	83.30% <100.00%> (-0.91%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
scripts/plugins/Sync-PluginManifest.ps1	87.86% <100.00%> (+3.88%)	⬆️

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Eval Execution

✅ Status: Passed — no merge-blocking failures (46 advisory assertion failure(s) present)

• Artifacts evaluated: 4
• Specs run: 4
• Assertions passed: 20
• Assertions failed (blocking): 0
• Assertions failed (advisory): 46
• Failed specs (merge-blocking): 0

Artifact	Kind	Status	Specs	Passed	Failed (blocking)	Failed (advisory)
disclaimer-language	instruction	❌ fail	1	0	0	0
copilot-otel-metrics	skill	⚠️ advisory-fail	1	6	0	3
outcome-hypothesis	skill	⚠️ advisory-fail	1	11	0	43
requirements-author	skill	✅ pass	1	3	0	0

Legend — ✅ clean · ⚠️ advisory failures only (non-blocking) · ⏭️ skipped · ❌ merge-blocking failure

Only Failed specs (merge-blocking) gates this PR. Advisory assertion failures are signal-quality checks captured during iteration; review them, but they do not block merge and may be acceptable.

[Katrien De Graeve (katriendg)]

Thanks for this — the privacy argument for retiring the hook is convincing, and the carrier harness that measures what the Collector filter actually reaches, rather than asserting it, is the strongest thing in the change. The threat model retracting the claims that harness disproved is exactly the right instinct.

Full findings with file:line references and suggested patches are in the review comments. Happy to re-review once the generated docs are regenerated.

---

Findings in files this PR does not touch

These could not be attached inline because the files carry no diff hunk. They are the documentation the hook retirement did not reach.

High — Customization packages guide states both packages include the telemetry hook

docs/customization/packages.md lines 25, 47-49

Two places contradict the new catalog. Line 25 gives 'the telemetry hook' as a reason not to install both packages together, and the dedicated ## Hooks section at lines 47-49 asserts that hve-core and hve-core-all each include it and that extension users must configure hook locations manually. Both statements are false against the current .github/plugin/marketplace.json, and the manual-configuration instruction is unactionable.

Suggested fix

Line 25: "Do not install `hve-core` and `hve-core-all` together. Both include overlapping content."

Lines 47-49: "## Hooks\n\nHooks are per-plugin declarations through the optional `hooks` field. No package currently declares one. VS Code has no declarative hook contribution point, so a package that adds a hook requires extension users to configure its location manually. Hooks are not copied during selective installation."

High — Extension install guidance directs users to configure a hook location that no longer ships

docs/getting-started/install.md lines 112

The installation guide tells extension users to perform a manual configuration step for a component the repository no longer contains. A reader following this instruction after the merge will look for a hook manifest path to enter into their VS Code configuration and find nothing, and any reader who already wired .github/hooks/shared/telemetry.json into their settings from a previous release keeps a dangling path with no documented removal step.

Suggested fix

Delete the sentence, or replace it with a removal note: "HVE Core ships no hook manifest. If a previous release left a hook path in your VS Code configuration, remove it."

High — Package selection guide still declares the telemetry hook and its manual setup step

docs/getting-started/packages.md lines 109

The Selective Clone Adoption section states that both packages declare the telemetry hook and instructs extension users to configure hook locations manually. Neither package declares a hooks field any more, so the declaration is false and the instruction cannot be completed.

Suggested fix

Remove the paragraph. The surrounding text at line 111 already covers the installer behaviour ("never copies hooks"), which stays correct as a generic contract statement.

High — Generated package documents still advertise the deleted telemetry hook

docs/plugins/hve-core.md lines 131-135 (and docs/plugins/hve-core-all.md 272-276)

The PR removes the 'hooks' key and its componentMaturity entry from both the hve-core and hve-core-all recipes in .github/plugin/marketplace.json, but neither package document was regenerated. Both files still carry a generator-owned '### Hooks' table inside the AUTO-GENERATED ARTIFACTS region listing 'telemetry - experimental - Records Copilot session lifecycle events to local telemetry for reporting.' The repository convention is that a marketplace recipe change is followed by 'npm run docs:generate' with the regenerated pages committed, and 'npm run docs:generate:check' validates the region against the catalog. As shipped, both package documents describe a component the catalog no longer declares and the repository no longer contains, and the drift is inside a region a human is not permitted to hand-edit. This is the only place in the repository where the hook removal is functionally incomplete.

Also raised by the Readiness perspective: Both package landing pages carry a ### Hooks table inside the <!-- BEGIN AUTO-GENERATED ARTIFACTS --> region that lists a telemetry component as an included artifact of the package. .github/plugin/marketplace.json now contains no hooks key at all (grep for hooks in that file returns nothing) and git ls-files .github/hooks is empty, so the table advertises a component the package cannot deliver. These blocks are written by Update-PluginDocumentationSource in scripts/plugins/Generate-Plugins.ps1 (the section list at line 163 includes @{ Title = 'Hooks'; Kind = 'hook' }), which runs only under npm run plugin:generate. The validations named in the PR do not regenerate them: plugin:validate is an alias for lint:marketplace, docs:generate covers docs/reference/ only, and scripts/plugins/Modules/PluginHelpers.psm1:454 checks that the markers exist rather than that their content is current. The stale table therefore passes every check that was run.

Suggested fix

Run `npm run docs:generate` and commit the regenerated `docs/plugins/hve-core.md` and `docs/plugins/hve-core-all.md`; the `### Hooks` section should disappear with the recipe key. Confirm with `npm run docs:generate:check` and `npm run plugin:validate`. Do not hand-edit the generated region.

Medium — Architecture reference states both plugin entries include the telemetry hook

docs/architecture/ai-artifacts.md lines 272

The Distribution section asserts that both plugin entries include the telemetry hook and that extension users configure hook locations manually. This is the architecture page a contributor reads to understand what a package contains, so a stale claim here propagates into further work. The neighbouring statements about the optional hooks field (line 213) and the closure model (line 222) remain correct as generic contract text.

Suggested fix

Choose the catalog entry that matches the required scope. Do not install `hve-core` and `hve-core-all` together because their content overlaps. No catalog entry currently declares a hook. VS Code has no declarative hook contribution point, so a package that adds one requires extension users to configure its location manually.

Medium — Package landing page claims hve-core-all includes the shared telemetry hook

docs/plugins/hve-core-all.md lines 12, 15

Same defect as finding 8 on the aggregate package page: the membership sentence ends "It also includes the shared telemetry hook" and the CAUTION callout repeats the hook as shared content.

Suggested fix

Its membership spans planning, backlog integrations, data science, Design Thinking, security, accessibility, privacy, RPI, HVE Builder, and supporting operational tooling.

> [!CAUTION]
> Do not install `hve-core` and `hve-core-all` together. Both include shared content, so install one package based on the scope you need.

Medium — Package landing page claims hve-core includes the shared telemetry hook

docs/plugins/hve-core.md lines 12, 15

The hand-written intro and the CAUTION callout both name the shared telemetry hook as package content. The callout also uses it as a reason not to install both packages together. Neither claim holds after the hooks key was removed from the recipe.

Suggested fix

It combines lifecycle coordination, prompt-engineering authoring and validation, documentation, Git and pull request workflows, and code review.

> [!CAUTION]
> Do not install `hve-core` and `hve-core-all` together. Both include shared content, so install one package based on the scope you need.

---

_{AI-assisted review (Functional, Standards, Security, Readiness at comprehensive depth): 0 Critical, 10 High, 29 Medium, 18 Low across 57 findings. 50 posted inline, 7 above. PR description accuracy and mergeability were excluded from scope at the reviewer's request. Findings are suggestions for a human reviewer to weigh, not engineering sign-off.}

[Emmeline Hoops (chaosdinosaur)]

Runtime and security (Medium)

• Validate settings policy against the merged effective configuration, not only values passed in the current invocation (settings_upsert.py:340).
• Add an upgrade path that rotates persisted Grafana admin credentials when reusing copilot-otel-data; verify the supplied credential succeeds and admin/admin fails (compose.yaml:72-79).
• Disable ambient proxy discovery for local credentialed requests, or apply a separately validated explicit proxy policy. The current build_opener can route localhost Basic credentials through HTTP_PROXY (_input_policy.py:189-191).

Please add split-invocation settings tests, fresh/preinitialized Grafana volume tests, and an ambient-proxy regression test.

[Emmeline Hoops (chaosdinosaur)]

Engineering standards and tests (Medium)

• Add focused Pester coverage for hook-present, hook-absent, and stale-hook manifest synchronization states (Sync-PluginManifest.ps1:368-373).
• Bring the new Python suite under the repository BDD naming and Arrange/Act/Assert conventions; this is a suite-level pattern, not one isolated test (test_collector_carriers.py:761-795).
• Type the exported open_url response contract and remove the ANN201 suppression (_input_policy.py:179-193).

[Emmeline Hoops (chaosdinosaur)]

Accessibility and migration (Medium)

• Expose the eight-step local setup as a semantic Markdown ordered list rather than numbered shell comments inside one code block (examples/README.md:43-70).
• Add retirement guidance for existing hook users. The deleted tooling leaves legacy .copilot-tracking/telemetry, ~/.hve/telemetry-dirs, generated launchers, and raw capture files untouched (TRANSPARENCY-NOTE.md:59-60).

[Emmeline Hoops (chaosdinosaur)]

PR scope and readiness (Medium)

• The hook retirement is coupled to a broad replacement-platform expansion. Split it, or add a reviewer-facing change map with independent validation and rollback boundaries (SECURITY.md:18).
• Expand the PR description to cover mandatory fleet bearer authentication/TLS, per-environment deployment, agent-host loss behavior, and the relay requirement.
• Complete or accurately disposition required checklist items: backwards compatibility, validate:local, validate:docs, spell check, and Markdown links. Do not claim compatibility for an explicitly breaking change.
• Update the branch from main and resolve the currently failing required checks before merge.

[Emmeline Hoops (chaosdinosaur)]

Documentation accuracy (4 Medium, 1 Low)

• Correct the claim that the test suite starts no container; test_collector_carriers.py starts disposable Collector containers (SECURITY.md:81).
• Replace the stale tag-pinning narrative with the actual digest-pinning posture while retaining signature/provenance verification as unresolved (SECURITY.md:232).
• Complete the public residual-carrier inventory with instrumentation scope name/version and resource/scope schema URLs (docs/customization/copilot-otel-metrics.md:151-155).
• Remove remaining architecture-page statements that say a telemetry hook ships or must be configured manually (docs/architecture/ai-artifacts.md:231).
• Correct the open residual-gap count from 10 to 11, or derive it from normalized register statuses (SECURITY.md:47).

[Emmeline Hoops (chaosdinosaur)]

The hook retirement resolves the Windows fail-closed tool-call denial in #2496, but changes are requested before merge. The fleet workflow requires a workstation relay that is not shipped, existing Grafana volumes can retain stale credentials, settings validation can miss conflicts in the effective configuration, and local credentialed requests can traverse ambient proxies. Please also address the zero-hook regression coverage, migration guidance, documentation inconsistencies, and current branch/checklist blockers detailed in the inline and categorized comments.