Skip to content

fix(health-plugin): pluginUsage counts hook fires, so usage telemetry cannot judge plugin value #2245

Description

@laurigates

What

pluginUsage.usageCount in ~/.claude.json counts hook fires in the same number as skill, agent, and command deliveries. The hook term dominates by two to four orders of magnitude, so the counter cannot answer "is this plugin earning its catalog cost?" — which is the question it looks like it answers.

Measured on one machine, 427 sessions in the trailing 30 days, cross-referenced against each plugin's declared hooks in .claude-plugin/plugin.json:

Plugin Declared hooks usageCount
hooks-plugin 19 entries across 8 events 164,694
feedback-plugin PreToolUse×1 50,180
kubernetes-plugin PreToolUse×2 16,851
session-plugin SessionStart×1, Stop×1 4,592
configure-plugin SessionStart×1 1,292
taskwarrior-plugin SessionStart×1 1,280
health-plugin SessionStart×1 1,263
evaluate-plugin SessionStart×1 1,178
project-plugin none 64
agents-plugin none 44
comfyui-plugin none 18
tools-plugin none 6

The counter tracks hook cadence and nothing else. Plugins with no hooks sit at or below 64 — that residue is genuine skill and agent delivery. Every plugin in the thousands has hooks; per-tool-call hooks reach six figures.

A supporting detail: the four SessionStart-only plugins share an identical lastUsedAt millisecond, which is one session-start event dispatching four plugin hooks in lockstep. Their counts are close but unequal (1,178–1,292), consistent with the same event source and different enable dates.

Why

health-plugin:health-check reads usage telemetry as one input to plugin cleanup recommendations. On this data that input is inverted: a plugin that has delivered zero skills or agents but fires one SessionStart hook outranks a hookless plugin that delivered 64 times, by roughly 18×.

The worked example is evaluate-plugin. Its counter reads 1,178, which looks like heavy use. Across 427 sessions and a full-transcript pass over 29,071 tool calls:

  • zero of its 7 skills ever dispatched
  • zero of its 3 agents ever dispatched (subagent_type matching evaluate* returns nothing)
  • no slash-command invocations
  • no tool call carries AttributionPlugin: evaluate-plugin

It ships no commands/ and no .mcp.json, so the delivery surfaces reduce to skills, agents, and one hook. The entire counter is the hook.

The probe script is correct and should stay as written. hooks/evaluate-drift-probe.sh bounds its find to depth 5, excludes node_modules and .git, and no-ops cleanly when no eval-results/ directory exists. This issue is about what the counter means, not about the script.

How

  • Document in .claude/rules/ that pluginUsage.usageCount is hook-dominated and is not a delivery signal, so nothing reads it as one.
  • Normalize in health-plugin:health-check: a plugin whose count is close to sessions × hooks-per-session is hook-dominated. Report hook-driven and delivery-driven activity as separate figures rather than one total, or discount the hook term before ranking.
  • Prefer per-call attribution where a delivery signal is actually needed. Tool calls carry AttributionPlugin / AttributionSkill, which measures delivery directly. Note AttributionAgent was empty across all 29,071 calls sampled, so agent dispatch needs a different source.
  • Decide separately whether evaluate-plugin earns its 10 always-resident catalog entries (7 skills + 3 agents) given zero delivery in 427 sessions. That is a catalog-cost question, and it is only visible once the counter stops reading as 1,178.

Verification

The mechanism was established by eliminating the other delivery surfaces plus the cadence ladder above, not by direct instrumentation. The direct test: record a SessionStart-only plugin's usageCount, start one throwaway session, re-read. Expect +1 with a fresh lastUsedAt shared by the other SessionStart-only plugins.

Observed incidentally while this analysis ran: evaluate-plugin moved 1,151 → 1,178 over a working session, tracking session starts rather than any use of the plugin.

References

Metadata

Metadata

Assignees

Labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions