Skip to content

fix(security): accept fully covered documentation scans - #143

Open
deepujain wants to merge 2 commits into
NVIDIA:mainfrom
deepujain:fix/137-doc-only-skillspector-completeness
Open

deepujain wants to merge 2 commits into
NVIDIA:mainfrom
deepujain:fix/137-doc-only-skillspector-completeness

Conversation

@deepujain

Copy link
Copy Markdown

Summary

  • accept SkillSpector partial completeness only when the scan has 100% coverage and every analyzer is either completed or not applicable
  • restrict the exception to reports that declare no executable scripts
  • preserve fail-closed behavior for limitations, ledger exceptions, partial or uninspected files, executable skills, unavailable analyzers, and missing evidence
  • add positive documentation-only coverage plus adversarial regression cases

Closes #137.

Verification

  • .venv/bin/python -m pytest -q tests/validators/test_security.py — 342 passed
  • make PYTHON=.venv/bin/python lint — passed
  • .venv/bin/python -m build — sdist and wheel built successfully
  • git diff --check origin/main...HEAD — passed
  • make PYTHON=.venv/bin/python test — 6,061 passed, 18 skipped, 4 deselected, 4 unrelated baseline failures: one CLI golden-surface drift and three Anthropic SDK httpx2 compatibility expectations

Release Impact

Documentation-only skills can pass security validation when SkillSpector inspected the complete component inventory and reports code-dependent analyzers as not applicable. Reports with any incomplete evidence continue to fail closed.

Signed-off-by: Deepak Jain <deepujain@gmail.com>
and not ledger_exceptions
and not limitations
and report_metadata.get("has_executable_scripts") is False
and "not_applicable" in analyzer_states

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Verify which analyzers are not applicable. Reducing all evidence to a set of states lets any required zero-work analyzer activate this exception. Starting from the new docs-only payload, change mcp_tool_poisoning to status="not_applicable" with planned_work=completed=0; validation still passes, although SkillSpector 2.11.2 normally runs that analyzer over SKILL.md with planned/completed work of 1. This can turn missing MCP security analysis plus an explicit partial verdict into a clean scan. Match the exact allowed docs-only N/A analyzer IDs (and reason codes/version), and require every other required analyzer to be completed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Documentation-only skills can never pass validate: not_applicable analyzers are counted as missing security evidence

2 participants