Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update corpus tests in 3.3.x #8

Merged
merged 1 commit into from
Aug 29, 2024
Merged

update corpus tests in 3.3.x #8

merged 1 commit into from
Aug 29, 2024

Conversation

khieta
Copy link
Contributor

@khieta khieta commented Aug 26, 2024

Issue #, if available:

Description of changes:

Same as #7, but for the 3.3.x branch. Also includes a freshly-generated corpus, which is distinct from the one in that PR.

@khieta khieta marked this pull request as ready for review August 26, 2024 15:54
@cdisselkoen
Copy link

Should we be concerned that the new corpus tests are 85% smaller than the ones checked in?

@khieta
Copy link
Contributor Author

khieta commented Aug 26, 2024

Hm, I'll gather some stats about the generated tests & post results here before I merge.

@khieta
Copy link
Contributor Author

khieta commented Aug 29, 2024

Your question prompted me to write a script to check this in the future, which I put up in a separate PR to main (#9). According to that script, here is the diff between the old and new corpus:

Original New Difference
Total test cases 6065 5324 -741
Unique policies 3486 ( 57.5% ) 2826 ( 53.1% ) -660
Unique schemas 2298 ( 37.9% ) 2163 ( 40.6% ) -135
Unique entity stores 2826 ( 46.6% ) 2455 ( 46.1% ) -371
Entities per entity store mean: 1 / median: 1 / p90: 4 mean: 2 / median: 2 / p90: 4 n/a
Schema entity types mean: 1 / median: 1 / p90: 2 mean: 1 / median: 1 / p90: 2 n/a
Schema actions mean: 1 / median: 1 / p90: 2 mean: 1 / median: 1 / p90: 1 n/a
Trivial policies 2191 ( 36.1% ) 2259 ( 42.4% ) 68
Pure RBAC (non-trivial) policies 1148 ( 18.9% ) 910 ( 17.1% ) -238
Pure ABAC (non-trivial) policies 156 ( 2.6% ) 111 ( 2.1% ) -45

So the corpus has indeed become smaller compared to the old data (which was from several months ago). This could be due to performance regressions in Cedar leading to fewer tests being executed, or just changes to the fuzzing infrastructure in cedar-spec. Hopefully using the new script in #9 will allow us to track this better.

@khieta khieta merged commit de66a6d into release/3.3.x Aug 29, 2024
4 checks passed
@khieta khieta deleted the khieta/3.3.x branch August 29, 2024 19:30
@khieta khieta mentioned this pull request Aug 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants