-
Notifications
You must be signed in to change notification settings - Fork 1.9k
feat(api): add resource group overview endpoint and filtering #9694
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(api): add resource group overview endpoint and filtering #9694
Conversation
|
✅ All necessary |
|
✅ Conflict Markers Resolved All conflict markers have been successfully resolved in this pull request. |
🔒 Container Security ScanImage: 📊 Vulnerability Summary
10 package(s) affected
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #9694 +/- ##
===========================================
+ Coverage 3.94% 92.57% +88.62%
===========================================
Files 830 165 -665
Lines 23425 23608 +183
===========================================
+ Hits 925 21854 +20929
+ Misses 22500 1754 -20746
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
jfagoagas
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's great @AdriiiPRodri I did a first pass without a functional test and left some comments/suggestions. Thanks!
api/src/backend/api/migrations/0066_finding_resource_group_scanresourcegroupsummary.py
Outdated
Show resolved
Hide resolved
api/src/backend/api/migrations/0066_finding_resource_group_scanresourcegroupsummary.py
Outdated
Show resolved
Hide resolved
- Add resource_group field to Resource model (0067 migration) - Add backfill migration to populate from findings (0068 migration) - Add resource_group filter (exact, in) to ResourceFilter - Add resource_group to ResourceSerializer - Update API spec with resource_group field and filter params - Populate resource_group from check metadata in scan job - Add tests for resource_group filter and field presence in response
Remove migration 0068 that backfills resource_group from findings. This operation is too expensive for production databases. The resource_group field will only be populated for new resources during scans. Existing resources will have NULL until re-scanned.
- Add resource_groups field to ResourceMetadataSerializer - Add resource_groups query to /resources/metadata endpoint - Add resource_groups query to /resources/metadata/latest endpoint - Add resource_group filter to LatestResourceFilter
jfagoagas
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After our conversation:
- Rename
resource_grouptogroup - Mark
resource_typeas deprecated in the API spec
- Rename resource_group field to group in Resource model - Update migration 0067 with new field name - Update ResourceSerializer and ResourceMetadataSerializer - Update filters (ResourceFilter, LatestResourceFilter) - Update views metadata queries - Update scan.py Resource creation/update logic - Update fixtures and tests - Update API spec with new field names and filters - Mark resource type field as deprecated in API spec - Add groups assertions to resources metadata tests
AdriiiPRodri
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think there's a bit of confusion between group and groups, at least for now. That said, I don't think this should change: resources should belong to a single group only
That's correct. The relationship is 1:1. |
But I have a question: The singular/plural pattern already exists throughout the system:
The logic is:
This is consistent with Django/DRF conventions and how the rest of the API already works. Maybe I don't have enough context (sorry in advance) |
I mean that if it is a 1:1 relationship, the API output must use the singular form. If it is already like that, there is nothing to change. While testing the changes, I see several things: in findings the change from Findings: Resources: |
…ary models Rename the field from 'resource_group' to 'group' for consistency with the API naming conventions (singular for model fields, plural for metadata). Changes: - Models: Finding.group, ScanResourceGroupSummary.group (migration updated) - Filters: filter[group], filter[group__in] - Serializers: 'group' field in responses - Views: /api/v1/overviews/groups endpoint (was /resource_groups) - Metadata: 'groups' (plural) in metadata responses - Jobs: scan.py and backfill.py updated to use 'group' - Tests: All fixtures and assertions updated Manual testing performed: - GET /api/v1/findings -> returns 'group' field ✓ - GET /api/v1/resources -> returns 'group' field ✓ - GET /api/v1/findings?filter[group]=monitoring -> works ✓ - GET /api/v1/findings?filter[group__in]=monitoring,security -> works ✓ - GET /api/v1/overviews/groups -> returns group aggregations ✓ - GET /api/v1/findings/metadata -> returns 'groups' (plural) ✓ All 1426 API tests pass. Note for existing deployments: If migration 0066 was already applied with the old column name 'resource_group', run: ALTER TABLE findings RENAME COLUMN resource_group TO "group"; ALTER TABLE scan_resource_group_summaries RENAME COLUMN resource_group TO "group";
Changes made (rename resource_group → group) Renamed field from resource_group to group for consistency with API naming conventions (singular for model fields, plural for metadata). Files changed:
Bug fixed: Manual testing:
|
jfagoagas
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the final changes @vicferpoy, as always:
Co-authored-by: Alan Buscaglia <[email protected]> Co-authored-by: Víctor Fernández Poyatos <[email protected]>

Context
This PR adds a new Resource Group API feature that enables grouping and filtering findings by logical resource categories (storage, compute, security, database, etc.). This provides users with a high-level view of their security posture organized by resource type, complementing the existing categories functionality.
Related to the Resource Inventory component requirements.
Description
Add support for resource group aggregations to enable grouping findings by logical resource categories. This feature mirrors the existing categories implementation pattern.
Changes:
API:
ScanResourceGroupSummarymodel for pre-aggregated metrics per scangroupfield toFinding,Resource, andScanResourceGroupSummarymodels/api/v1/overviews/groupsendpoint with provider filteringgroupandgroup__infilters to findings and resources endpointsgroupsin metadata responsesaggregate_resource_group_countsfor real-time scan aggregationbackfill_scan_resource_group_summariestask for historical dataResourceGroupOverviewFilterandResourceGroupOverviewSerializerThe endpoint returns aggregated counts (
total_findings,failed_findings,new_failed_findings,resources_count) grouped bygroupandseverity, following the same pattern as the categories endpoint.Field naming convention (consistent with existing API patterns):
groupgroupsgroup,group__inManual Testing Performed
All endpoints verified working:
GET /api/v1/findings→ returnsgroupfieldGET /api/v1/resources→ returnsgroupfieldGET /api/v1/findings?filter[group]=monitoring→ filter worksGET /api/v1/findings?filter[group__in]=monitoring,security→ multi-filter worksGET /api/v1/overviews/groups→ returns group aggregationsGET /api/v1/findings/metadata→ returnsgroups(plural)All 1426 API tests pass.
Steps to review
0066_finding_resource_group_scanresourcegroupsummary.pyGET /api/v1/overviews/groups- returns all resource groupsGET /api/v1/overviews/groups?filter[group]=storage- filter by specific groupGET /api/v1/overviews/groups?filter[provider_type]=aws- filter by providerGET /api/v1/findings?filter[group]=storageGET /api/v1/findings?filter[group__in]=storage,securitygroups:GET /api/v1/findings/metadataGET /api/v1/resources/metadataScanResourceGroupSummaryrecords are createdresources_countcorrectly counts unique resources (not total findings)Checklist
UI
API
License
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.