Documentation-only skills can never pass validate: not_applicable analyzers are counted as missing security evidence

[aunysillyme]

Summary

skillevaluator validate reports the security scan INCOMPLETE and exits 1 for any skill that ships no scripts and no MCP configuration, i.e. a documentation-only skill. Nothing is wrong with the skill and nothing the operator can configure changes it.

The cause is a mismatch between the analyzers SkillEvaluator treats as required evidence and the analyzers SkillSpector reports as not_applicable when there is no code to analyze.

Versions

• skillevaluator 0.2.1 (skillevaluator[all] from git, commit ff349e0)
• skillspector 2.11.2 (installed via uv tool install git+https://github.com/NVIDIA/SkillSpector.git)
• macOS (Darwin 25.6.0), arm64, Python 3.13
• semgrep and gitleaks 8.30.1 both present

Reproduce

A minimal markdown-only skill:

my-skill/SKILL.md      # frontmatter with name, description, metadata.author; an H1; some prose

skillevaluator validate ./skills/my-skill --external --no-dedup

│ Security Scan │ INCOMPLETE │ skillspector did not complete (1 errors) │

Errors:
  • skillspector JSON field 'analysis_completeness' reports incomplete analysis
    (status 'partial'); security scan did not complete
Error: security scan failed          # exit 1

skillspector scan on the same directory returns:

"analysis_completeness": {
  "coverage_percent": 100.0,
  "is_complete": false,
  "status": "partial",
  "limitations": [],
  "analyzer_statuses": [
    {"analyzer_id": "behavioral_ast", "status": "not_applicable"},
    {"analyzer_id": "behavioral_taint_tracking", "status": "not_applicable"},
    {"analyzer_id": "bundled_execution_surface", "status": "not_applicable"},
    {"analyzer_id": "mcp_least_privilege", "status": "not_applicable"},
    {"analyzer_id": "meta_analyzer", "status": "not_applicable"}
  ]
}

Coverage is 100% and there are no limitations. The analyzers simply had nothing to look at.

The separating probe

Add one trivial python file to an otherwise identical skill:

my-skill/SKILL.md
my-skill/scripts/hello.py     # def main(): print("hello")

Same command, same flags:

passed: True | incomplete_scans: [] | exit 0

So the variable is the presence of a script, not anything about the skill's content or quality.

Why the semantic analyzers are not the cause

validators/security.py already handles those:

_SKILLSPECTOR_OPTIONAL_ANALYZERS = _SKILLSPECTOR_SEMANTIC_ANALYZERS | {"meta_analyzer"}
# SkillSpector 2.10+ can omit semantic analyzers when no provider is available.
_SKILLSPECTOR_2_10_REQUIRED_ANALYZERS = _SKILLSPECTOR_COMMON_REQUIRED_ANALYZERS | {"artifact_integrity"}

but _SKILLSPECTOR_COMMON_REQUIRED_ANALYZERS still requires behavioral_ast, behavioral_taint_tracking and mcp_least_privilege, which are precisely the ones that cannot run without code. The failure persists with SKILLSPECTOR_PROVIDER=ollama configured, so it is unrelated to the semantic analyzers or to any missing API key.

Suggested fix

Treat not_applicable as satisfied rather than missing, for the script and MCP analyzers, when the scanned skill contains no scripts and no MCP configuration. analysis_completeness.status == "partial" with coverage_percent == 100.0, empty limitations and no fatal ledger_exceptions is a complete scan of everything there was to scan.

Alternatively, keep the strictness but distinguish it in the output, so that "no code to analyze" does not present identically to a scanner that crashed. Both currently surface as incomplete_scans: ["skillspector"], which means a consumer cannot tell them apart. That matters: anyone accepting the label to work around this bug also silently accepts a crashed or timed-out scan.

Impact

Documentation-only skills are common: prompt-and-procedure skills with no bundled code. For a collection of 15 such skills, validate exits 1 on all 15 with no actionable finding, so the exit code cannot be used as a CI gate. The recommended keyless gate in the CI Integration docs is affected.