feat: Scan Results Import Feature #9704
Conversation
Implement Phase 1 of the scan results import feature (Tasks 1.1-1.3): - Add IMPORTED trigger type to Scan.TriggerChoices for imported scans - Create parsers module with OCSF JSON and CSV format support - OCSF parser handles Prowler CLI JSON output (OCSF schema) - CSV parser handles semicolon/comma-delimited Prowler CSV output - Both parsers include validation, error handling, and provider extraction - Add comprehensive docstrings to models.py for better documentation - Include unit tests for both parsers New files: - api/src/backend/api/parsers/ (module with OCSF and CSV parsers) - api/src/backend/api/tests/test_ocsf_parser.py - api/src/backend/api/tests/test_csv_parser.py - api/src/backend/api/migrations/0066_scan_imported_trigger.py - api/docs/models.md
Implement a new service for importing scan results from various formats (OCSF JSON, CSV, JSON-OCSF) into the Prowler API. The service handles: - Format detection and validation - Content parsing with error handling - Resource UID extraction and deduplication - Check metadata building from scan data - Raw result storage Includes comprehensive unit tests covering: - Format detection edge cases - Content parsing and error handling - Resource UID extraction - Check metadata building - Import validation - Default value handling
Add POST /api/v1/scans/import endpoint for importing Prowler CLI scan results in JSON/OCSF or CSV format. Changes: - ScanImportView with multipart file upload and inline JSON support - ScanImportSerializer with file/data validation and 50MB size limit - URL route registration at /scans/import - Comprehensive test suite covering JSON, CSV, provider handling, validation errors, authentication, permissions, and tenant isolation - Enhanced logging with timing metrics in ScanImportService Relates to: prowler-cloud#8972
Implement Phase 3 of the scan results import feature (Tasks 3.1-3.8): - Add ScanImportSection component with collapsible UI and state machine - Add ScanImportDropzone for drag-and-drop file upload (.json, .csv) - Add ScanImportForm with provider selection and validation - Add ScanImportProgress for upload/processing status display - Add importScan server action with Zod validation and API integration - Integrate ScanImportSection into scans page with permission check - Add router.refresh() on successful import to update scan list - Add new icons: Upload, ChevronUp, File, AlertCircle, CheckCircle, etc. New files: - ui/actions/scans/import-scan.ts - ui/components/scans/scan-import/ (types, dropzone, form, progress, section) Relates to: prowler-cloud#8972
Add comprehensive documentation for the scan import feature including: - User guide for importing JSON/OCSF and CSV scan results - Detailed field mappings for both formats - API usage examples with curl commands - Extensive troubleshooting guide covering: - Format detection and validation errors - Provider resolution issues - Authentication and permission errors - File size and performance guidance - Common error codes reference Also includes: - OpenAPI schema examples for request/response in views.py - API changelog entry for the new endpoint - Playwright e2e test configuration for scan import - Navigation entry in docs.json Relates to: prowler-cloud#8972
- Replace empty string with sentinel value for auto-detect provider option to work with Radix UI Select constraints - Update scan import form to properly convert sentinel value back to undefined for API calls - Change "View Imported Scan" link to navigate to findings page with scan filter instead of direct scan view - Update link text to "View Scan Findings" for better clarity on destination - Add comprehensive JSDoc documentation to next.config.js with feature descriptions and environment variable reference - Enhance CSP header configuration with detailed comments explaining each directive and security implications - Add getSentryReportEndpoint function documentation with usage examples
- Add detailed API configuration documentation covering Django settings, environment variables, and file upload limits - Add services README documenting scan import service architecture and usage - Update scan import service with improved error handling and validation - Enhance scan import tests with additional edge cases and validation scenarios - Update serializers and views to support improved scan import workflow - Increase file upload limits to 1GB for handling large enterprise scan imports - Add scan import API route handler for Next.js frontend integration - Update scan import UI components with improved error handling and user feedback - Update gunicorn configuration for optimal performance with large file uploads - Add API documentation for scan import endpoint and configuration - Improve finding detail component display and scan import section UX - Update project README files with scan import feature documentation
- Add changelog entry for scan import UI components - Create comprehensive PR description following template - Document all changes for issue prowler-cloud#8972 prowler-cloud#8972
…cumentation - Reorganize test files from api/src/backend/api/tests to api/tests/manual directory - Add comprehensive README.md documentation for manual test suite - Reformat test data generation functions with improved code readability and line length compliance - Update CSV and JSON test data structures to match actual Prowler CLI output format - Enhance docstrings with clearer descriptions and parameter documentation - Improve code formatting for better maintainability and consistency across test files - Consolidate test scenarios for error handling, large file processing, and real-world data imports
github-actions
bot
added
documentation
component/ui
component/api
review-django-migrations
|
✅ Conflict Markers Resolved All conflict markers have been successfully resolved in this pull request. |
removing from PR as unneeded
github-actions
bot
removed
the
has-conflicts
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| return <SettingsIcon className={iconClass} />; | ||
| default: | ||
| return ( | ||
| <Loader2Icon className={cn(iconClass, isActive && "animate-spin")} /> |
Check warning
Code scanning / CodeQL
Useless conditional Warning
sonofagl1tch
changed the title
| if time_val: | ||
| try: | ||
| timestamp = datetime.fromtimestamp(float(time_val)) | ||
| except (ValueError, TypeError): |
Check notice
Code scanning / CodeQL
Empty except Note
|
all testing was completed on a macbook pro m2 locally in docker containers. I am going to do additional testing to deploy on aws ecs but until then, I think some additional attention should be paid to reviewing my api code. this is new to me and I wasnt able to test it as much as I think I could. |
|
Hi @sonofagl1tch thanks for this contribution! We'll review it as soon as we get the chance to. |
|
Hello @sonofagl1tch, we've been reviewing and discussing the content of this PR internally and there are several things we want to discuss with you. We can jump into a call if you prefer but I'm going to leave a summary of the action points:
As we don't want to leave the whole RFC in a comment, I'm leaving a summary of the key changes we'd want to discuss with you:
None of the above is written in stone but that's something we're still defining to make a decision. The following are also some alternatives we considered, one is the one you implemented, again to give you a summary of them: Redis / Celery Only (no local storage):Push the full OCSF payload directly into Redis as a Celery message. Why rejected:
No Celery, No S3 (Process in the API request)
Why rejected
Regarding the UI work you did that's something is out of our scope but we can keep it after developing the Findings Ingestion API. One of our main points is to keep you as contributor of this feature. We'd be pleased to continue working with you if you have the bandwidth enough to work on what we've shared. Take into account that the above is just a summary, there are more aspects to take into account to continue the development. Next time, as you did in other contributions we prefer to have a conversation first using the issue/feature-request. |
Hey folks! I fully understand the feedback. This was a large feature submission, and I'm still learning your preferred best practices. I'm happy to continue collaborating with you on this feature. Suggested path forward:
How does this plan sound? I'll take it as a yes if you complete step 1 and post it to this PR, and I'll then start on step 2. cheers, |
That's great Ryan! Let's talk next week about this. Have a great weekend! |
|
Hi @sonofagl1tch we're going to require more time until we send you a plan to continue the development. We need to finish the RFC and prepare the work items. By the way, we've incorporate some AI skills that will help a lot during development. You can found them in the
|
2 participants
Pull Request: Scan Results Import Feature
Context
This PR implements the scan results import feature requested in issue #8972. It enables users to import Prowler CLI scan results (JSON/OCSF and CSV formats) into the Prowler API, allowing distributed resources protected by network boundaries to send results to a central Prowler instance for visualization in the UI.
Closes #8972
Description
This feature adds a complete end-to-end solution for importing Prowler CLI scan results:
Backend (API)
IMPORTEDtoScan.TriggerChoicesfor tracking imported scansapi/src/backend/api/parsers/module with:ocsf_parser.py: Parses Prowler JSON/OCSF output formatcsv_parser.py: Parses Prowler CSV output (semicolon/comma-delimited)api/src/backend/api/services/scan_import.pywith:POST /api/v1/scans/importendpoint supporting:Frontend (UI)
ui/components/scans/scan-import/:ScanImportSection: Main container with collapsible UIScanImportDropzone: Drag-and-drop file upload (.json, .csv)ScanImportForm: Provider selection with validationScanImportProgress: Upload/processing status displayui/actions/scans/import-scan.tsfor API integrationui/app/api/scans/import/route.tsfor large file handlingDocumentation
docs/user-guide/tutorials/prowler-app-scan-import.mdxdocs/docs.jsonTesting
Implementation Status
All implementation tasks are complete. The feature has been fully implemented and tested:
Remaining items for PR submission:
Steps to Review
Backend Review:
api/src/backend/api/parsers/api/src/backend/api/services/scan_import.pyapi/src/backend/api/v1/views.py(ScanImportView)poetry run pytest api/src/backend/api/tests/test_ocsf_parser.py api/src/backend/api/tests/test_csv_parser.py api/src/backend/api/tests/test_scan_import_service.py api/src/backend/api/tests/test_scan_import_view.py -vFrontend Review:
ui/components/scans/scan-import/ui/actions/scans/import-scan.tsui/app/api/scans/import/route.tsui/app/(prowler)/scans/page.tsxManual Testing:
docker-compose -f docker-compose-dev.yml upDocumentation Review:
docs/user-guide/tutorials/prowler-app-scan-import.mdxChecklist
UI
API
Files Changed
New Files
api/src/backend/api/parsers/__init__.pyapi/src/backend/api/parsers/ocsf_parser.pyapi/src/backend/api/parsers/csv_parser.pyapi/src/backend/api/parsers/README.mdapi/src/backend/api/services/__init__.pyapi/src/backend/api/services/scan_import.pyapi/src/backend/api/services/README.mdapi/src/backend/api/tests/test_ocsf_parser.pyapi/src/backend/api/tests/test_csv_parser.pyapi/src/backend/api/tests/test_scan_import_service.pyapi/src/backend/api/tests/test_scan_import_view.pyapi/src/backend/api/tests/test_scan_import_real_csv.pyapi/src/backend/api/tests/test_scan_import_real_json.pyapi/src/backend/api/migrations/0066_scan_imported_trigger.pyapi/tests/manual/- Manual test fixtures and scriptsui/actions/scans/import-scan.tsui/app/api/scans/import/route.tsui/app/api/scans/import/README.mdui/components/scans/scan-import/index.tsui/components/scans/scan-import/types.tsui/components/scans/scan-import/scan-import-dropzone.tsxui/components/scans/scan-import/scan-import-form.tsxui/components/scans/scan-import/scan-import-progress.tsxui/components/scans/scan-import/scan-import-section.tsxui/tests/scan-import.spec.tsdocs/user-guide/tutorials/prowler-app-scan-import.mdxModified Files
api/src/backend/api/models.py- Added IMPORTED trigger typeapi/src/backend/api/v1/serializers.py- Added import serializersapi/src/backend/api/v1/views.py- Added ScanImportViewapi/src/backend/api/v1/urls.py- Added import routeapi/src/backend/api/rls.py- RLS policy updatesapi/src/backend/config/django/base.py- Config updatesapi/src/backend/config/guniconf.py- Gunicorn config updatesapi/CHANGELOG.md- Added changelog entryui/app/(prowler)/scans/page.tsx- Integrated import sectionui/actions/scans/index.ts- Export updatesui/components/scans/index.ts- Export updatesui/components/icons/Icons.tsx- Added upload iconui/next.config.js- Increased server actions body size limit to 1GBui/package.json- Dependenciesui/playwright.config.ts- Test configui/CHANGELOG.md- Added changelog entrydocs/docs.json- Added navigation entryContributions Review
Code Quality ✅
123456789012)Test Coverage ✅
test_ocsf_parser.py,test_csv_parser.py,test_scan_import_service.pytest_scan_import_view.py(auth, permissions, tenant isolation, validation)ui/tests/scan-import.spec.ts(Playwright)api/tests/manual/(real JSON/CSV, large files, error scenarios)Documentation ✅
docs/user-guide/tutorials/prowler-app-scan-import.mdxapi/src/backend/api/services/README.mdapi/src/backend/api/parsers/README.mdChangelog Entries ✅
api/CHANGELOG.md- New endpoint entry addedui/CHANGELOG.md- Scan import UI entry addedSecurity Review
123456789012)License
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.