diff --git a/AGENTS.md b/AGENTS.md index a982b82fae..00e3d370f2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -36,6 +36,7 @@ Use these skills for detailed patterns on-demand: | `prowler-test-api` | API testing (pytest-django + RLS) | [SKILL.md](skills/prowler-test-api/SKILL.md) | | `prowler-test-ui` | E2E testing (Playwright) | [SKILL.md](skills/prowler-test-ui/SKILL.md) | | `prowler-compliance` | Compliance framework structure | [SKILL.md](skills/prowler-compliance/SKILL.md) | +| `prowler-compliance-review` | Review compliance framework PRs | [SKILL.md](skills/prowler-compliance-review/SKILL.md) | | `prowler-provider` | Add new cloud providers | [SKILL.md](skills/prowler-provider/SKILL.md) | | `prowler-pr` | Pull request conventions | [SKILL.md](skills/prowler-pr/SKILL.md) | | `prowler-docs` | Documentation style guide | [SKILL.md](skills/prowler-docs/SKILL.md) | diff --git a/skills/prowler-compliance-review/SKILL.md b/skills/prowler-compliance-review/SKILL.md new file mode 100644 index 0000000000..47a270b410 --- /dev/null +++ b/skills/prowler-compliance-review/SKILL.md @@ -0,0 +1,187 @@ +--- +name: prowler-compliance-review +description: > + Reviews Pull Requests that add or modify compliance frameworks. + Trigger: When reviewing PRs with compliance framework changes, CIS/NIST/PCI-DSS additions, or compliance JSON files. +license: Apache-2.0 +metadata: + author: prowler-cloud + version: "1.0" +allowed-tools: Read, Edit, Write, Glob, Grep, Bash, WebFetch, WebSearch, Task +--- + +## When to Use + +- Reviewing PRs that add new compliance frameworks +- Reviewing PRs that modify existing compliance frameworks +- Validating compliance framework JSON structure before merge + +--- + +## Review Checklist (Critical) + +| Check | Command/Method | Pass Criteria | +|-------|----------------|---------------| +| JSON Valid | `python3 -m json.tool file.json` | No syntax errors | +| All Checks Exist | Run validation script | 0 missing checks | +| No Duplicate IDs | Run validation script | 0 duplicate requirement IDs | +| CHANGELOG Entry | Manual review | Present under correct version | +| Dashboard File | Compare with existing | Follows established pattern | +| Framework Metadata | Manual review | All required fields populated | + +--- + +## Commands + +```bash +# 1. Validate JSON syntax +python3 -m json.tool prowler/compliance/{provider}/{framework}.json > /dev/null \ + && echo "Valid JSON" || echo "INVALID JSON" + +# 2. Run full validation script +python3 skills/prowler-compliance-review/assets/validate_compliance.py \ + prowler/compliance/{provider}/{framework}.json + +# 3. Compare dashboard with existing (find similar framework) +diff dashboard/compliance/{new_framework}.py \ + dashboard/compliance/{existing_framework}.py +``` + +--- + +## Decision Tree + +``` +JSON Valid? +├── No → FAIL: Fix JSON syntax errors +└── Yes ↓ + All Checks Exist in Codebase? + ├── Missing checks → FAIL: Add missing checks or remove from framework + └── All exist ↓ + Duplicate Requirement IDs? + ├── Yes → FAIL: Fix duplicate IDs + └── No ↓ + CHANGELOG Entry Present? + ├── No → REQUEST CHANGES: Add CHANGELOG entry + └── Yes ↓ + Dashboard File Follows Pattern? + ├── No → REQUEST CHANGES: Fix dashboard pattern + └── Yes ↓ + Framework Metadata Complete? + ├── No → REQUEST CHANGES: Add missing metadata + └── Yes → APPROVE +``` + +--- + +## Framework Structure Reference + +Compliance frameworks are JSON files in: `prowler/compliance/{provider}/{framework}.json` + +```json +{ + "Framework": "CIS", + "Name": "CIS Provider Benchmark vX.Y.Z", + "Version": "X.Y", + "Provider": "AWS|Azure|GCP|...", + "Description": "Framework description...", + "Requirements": [ + { + "Id": "1.1", + "Description": "Requirement description", + "Checks": ["check_name_1", "check_name_2"], + "Attributes": [ + { + "Section": "1 Section Name", + "SubSection": "1.1 Subsection (optional)", + "Profile": "Level 1|Level 2", + "AssessmentStatus": "Automated|Manual", + "Description": "...", + "RationaleStatement": "...", + "ImpactStatement": "...", + "RemediationProcedure": "...", + "AuditProcedure": "...", + "AdditionalInformation": "...", + "References": "...", + "DefaultValue": "..." + } + ] + } + ] +} +``` + +--- + +## Common Issues + +| Issue | How to Detect | Resolution | +|-------|---------------|------------| +| Missing checks | Validation script reports missing | Add check implementation or remove from Checks array | +| Duplicate IDs | Validation script reports duplicates | Ensure each requirement has unique ID | +| Empty Checks for Automated | AssessmentStatus is Automated but Checks is empty | Add checks or change to Manual | +| Wrong file location | Framework not in `prowler/compliance/{provider}/` | Move to correct directory | +| Missing dashboard file | No corresponding `dashboard/compliance/{framework}.py` | Create dashboard file following pattern | +| CHANGELOG missing | Not under correct version section | Add entry to prowler/CHANGELOG.md | + +--- + +## Dashboard File Pattern + +Dashboard files must be in `dashboard/compliance/` and follow this exact pattern: + +```python +import warnings + +from dashboard.common_methods import get_section_containers_cis + +warnings.filterwarnings("ignore") + + +def get_table(data): + + aux = data[ + [ + "REQUIREMENTS_ID", + "REQUIREMENTS_DESCRIPTION", + "REQUIREMENTS_ATTRIBUTES_SECTION", + "CHECKID", + "STATUS", + "REGION", + "ACCOUNTID", + "RESOURCEID", + ] + ].copy() + + return get_section_containers_cis( + aux, "REQUIREMENTS_ID", "REQUIREMENTS_ATTRIBUTES_SECTION" + ) +``` + +--- + +## Testing the Compliance Framework + +After validation passes, test the framework with Prowler: + +```bash +# Verify framework is detected +poetry run python prowler-cli.py {provider} --list-compliance | grep {framework} + +# Run a quick test with a single check from the framework +poetry run python prowler-cli.py {provider} --compliance {framework} --check {check_name} + +# Run full compliance scan (dry-run with limited checks) +poetry run python prowler-cli.py {provider} --compliance {framework} --checks-limit 5 + +# Generate compliance report in multiple formats +poetry run python prowler-cli.py {provider} --compliance {framework} -M csv json html +``` + +--- + +## Resources + +- **Validation Script**: See [assets/validate_compliance.py](assets/validate_compliance.py) +- **Related Skills**: See [prowler-compliance](../prowler-compliance/SKILL.md) for creating frameworks +- **Documentation**: See [references/review-checklist.md](references/review-checklist.md) diff --git a/skills/prowler-compliance-review/assets/validate_compliance.py b/skills/prowler-compliance-review/assets/validate_compliance.py new file mode 100644 index 0000000000..92a5809b0f --- /dev/null +++ b/skills/prowler-compliance-review/assets/validate_compliance.py @@ -0,0 +1,218 @@ +#!/usr/bin/env python3 +""" +Prowler Compliance Framework Validator + +Validates compliance framework JSON files for: +- JSON syntax validity +- Check existence in codebase +- Duplicate requirement IDs +- Required field completeness +- Assessment status consistency + +Usage: + python validate_compliance.py + +Example: + python validate_compliance.py prowler/compliance/azure/cis_5.0_azure.json +""" + +import json +import os +import sys +from pathlib import Path + + +def find_project_root(): + """Find the Prowler project root directory.""" + current = Path(__file__).resolve() + for parent in current.parents: + if (parent / "prowler" / "providers").exists(): + return parent + return None + + +def get_existing_checks(project_root: Path, provider: str) -> set: + """Find all existing checks for a provider in the codebase.""" + checks = set() + services_path = project_root / "prowler" / "providers" / provider.lower() / "services" + + if not services_path.exists(): + return checks + + for service_dir in services_path.iterdir(): + if service_dir.is_dir() and not service_dir.name.startswith("__"): + for check_dir in service_dir.iterdir(): + if check_dir.is_dir() and not check_dir.name.startswith("__"): + check_file = check_dir / f"{check_dir.name}.py" + if check_file.exists(): + checks.add(check_dir.name) + + return checks + + +def validate_compliance_framework(json_path: str) -> dict: + """Validate a compliance framework JSON file.""" + results = { + "valid": True, + "errors": [], + "warnings": [], + "stats": {} + } + + # 1. Check file exists + if not os.path.exists(json_path): + results["valid"] = False + results["errors"].append(f"File not found: {json_path}") + return results + + # 2. Validate JSON syntax + try: + with open(json_path, "r") as f: + data = json.load(f) + except json.JSONDecodeError as e: + results["valid"] = False + results["errors"].append(f"Invalid JSON syntax: {e}") + return results + + # 3. Check required top-level fields + required_fields = ["Framework", "Name", "Version", "Provider", "Description", "Requirements"] + for field in required_fields: + if field not in data: + results["valid"] = False + results["errors"].append(f"Missing required field: {field}") + + if not results["valid"]: + return results + + # 4. Extract provider + provider = data.get("Provider", "").lower() + + # 5. Find project root and existing checks + project_root = find_project_root() + if project_root: + existing_checks = get_existing_checks(project_root, provider) + else: + existing_checks = set() + results["warnings"].append("Could not find project root - skipping check existence validation") + + # 6. Validate requirements + requirements = data.get("Requirements", []) + all_checks = set() + requirement_ids = [] + automated_count = 0 + manual_count = 0 + empty_automated = [] + + for req in requirements: + req_id = req.get("Id", "UNKNOWN") + requirement_ids.append(req_id) + + # Collect checks + checks = req.get("Checks", []) + all_checks.update(checks) + + # Check assessment status + attributes = req.get("Attributes", [{}]) + if attributes: + status = attributes[0].get("AssessmentStatus", "Unknown") + if status == "Automated": + automated_count += 1 + if not checks: + empty_automated.append(req_id) + elif status == "Manual": + manual_count += 1 + + # 7. Check for duplicate IDs + seen_ids = set() + duplicates = [] + for req_id in requirement_ids: + if req_id in seen_ids: + duplicates.append(req_id) + seen_ids.add(req_id) + + if duplicates: + results["valid"] = False + results["errors"].append(f"Duplicate requirement IDs: {duplicates}") + + # 8. Check for missing checks + if existing_checks: + missing_checks = all_checks - existing_checks + if missing_checks: + results["valid"] = False + results["errors"].append(f"Missing checks in codebase ({len(missing_checks)}): {sorted(missing_checks)}") + + # 9. Warn about empty automated + if empty_automated: + results["warnings"].append(f"Automated requirements with no checks: {empty_automated}") + + # 10. Compile statistics + results["stats"] = { + "framework": data.get("Framework"), + "name": data.get("Name"), + "version": data.get("Version"), + "provider": data.get("Provider"), + "total_requirements": len(requirements), + "automated_requirements": automated_count, + "manual_requirements": manual_count, + "unique_checks_referenced": len(all_checks), + "checks_found_in_codebase": len(all_checks - (all_checks - existing_checks)) if existing_checks else "N/A", + "missing_checks": len(all_checks - existing_checks) if existing_checks else "N/A" + } + + return results + + +def print_report(results: dict): + """Print a formatted validation report.""" + print("\n" + "=" * 60) + print("PROWLER COMPLIANCE FRAMEWORK VALIDATION REPORT") + print("=" * 60) + + stats = results.get("stats", {}) + if stats: + print(f"\nFramework: {stats.get('name', 'N/A')}") + print(f"Provider: {stats.get('provider', 'N/A')}") + print(f"Version: {stats.get('version', 'N/A')}") + print("-" * 40) + print(f"Total Requirements: {stats.get('total_requirements', 0)}") + print(f" - Automated: {stats.get('automated_requirements', 0)}") + print(f" - Manual: {stats.get('manual_requirements', 0)}") + print(f"Unique Checks: {stats.get('unique_checks_referenced', 0)}") + print(f"Checks in Codebase: {stats.get('checks_found_in_codebase', 'N/A')}") + print(f"Missing Checks: {stats.get('missing_checks', 'N/A')}") + + print("\n" + "-" * 40) + + if results["errors"]: + print("\nERRORS:") + for error in results["errors"]: + print(f" [X] {error}") + + if results["warnings"]: + print("\nWARNINGS:") + for warning in results["warnings"]: + print(f" [!] {warning}") + + print("\n" + "-" * 40) + if results["valid"]: + print("RESULT: PASS - Framework is valid") + else: + print("RESULT: FAIL - Framework has errors") + print("=" * 60 + "\n") + + +def main(): + if len(sys.argv) < 2: + print("Usage: python validate_compliance.py ") + print("Example: python validate_compliance.py prowler/compliance/azure/cis_5.0_azure.json") + sys.exit(1) + + json_path = sys.argv[1] + results = validate_compliance_framework(json_path) + print_report(results) + + sys.exit(0 if results["valid"] else 1) + + +if __name__ == "__main__": + main() diff --git a/skills/prowler-compliance-review/references/review-checklist.md b/skills/prowler-compliance-review/references/review-checklist.md new file mode 100644 index 0000000000..d8673d8c03 --- /dev/null +++ b/skills/prowler-compliance-review/references/review-checklist.md @@ -0,0 +1,57 @@ +# Compliance PR Review References + +## Related Skills + +- [prowler-compliance](../../prowler-compliance/SKILL.md) - Creating compliance frameworks +- [prowler-pr](../../prowler-pr/SKILL.md) - PR conventions and checklist + +## Documentation + +- [Prowler Developer Guide](https://docs.prowler.com/developer-guide/introduction) +- [Compliance Framework Structure](https://docs.prowler.com/developer-guide/compliance) + +## File Locations + +| File Type | Location | +|-----------|----------| +| Compliance JSON | `prowler/compliance/{provider}/{framework}.json` | +| Dashboard | `dashboard/compliance/{framework}_{provider}.py` | +| CHANGELOG | `prowler/CHANGELOG.md` | +| Checks | `prowler/providers/{provider}/services/{service}/{check}/` | + +## Validation Script + +Run the validation script from the project root: + +```bash +python3 skills/prowler-compliance-review/assets/validate_compliance.py \ + prowler/compliance/{provider}/{framework}.json +``` + +## PR Review Summary Template + +When completing a compliance framework review, use this summary format: + +```markdown +## Compliance Framework Review Summary + +| Check | Result | +|-------|--------| +| JSON Valid | PASS/FAIL | +| All Checks Exist | PASS/FAIL (N missing) | +| No Duplicate IDs | PASS/FAIL | +| CHANGELOG Entry | PASS/FAIL | +| Dashboard File | PASS/FAIL | + +### Statistics +- Total Requirements: N +- Automated: N +- Manual: N +- Unique Checks: N + +### Recommendation +APPROVE / REQUEST CHANGES / FAIL + +### Issues Found +1. ... +```