Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix up nightly corpus tests #22

Merged
merged 5 commits into from
Jul 3, 2024
Merged

Fix up nightly corpus tests #22

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
99631a5
Set default shell to bash
patjakdev Jul 3, 2024
b8fbdc9
Only create a GitHub issue if the corpus tests fail on their schedule…
patjakdev Jul 3, 2024
0068b7e
Reduce the threshold for assuming that the corpus test population fai…
patjakdev Jul 3, 2024
8147a91
Fix the casing on the comparison of q.Decision with "allow"
patjakdev Jul 3, 2024
6b6cc00
Merge additional fixes made in PR from AWS team (https://github.com/c…
patjakdev Jul 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions .github/workflows/corpus.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,11 @@ on:

jobs:
build:
defaults:
run:
shell: bash
Comment on lines +10 to +12
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting bash here invokes the default -o pipefail behavior that correctly makes it so that the go test -count=1 -v -tags corpus ./integration_tests/... | tail fails when the tests fail.

runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4

Expand All @@ -26,7 +30,7 @@ jobs:
run: go test -count=1 -v -tags corpus ./integration_tests/... | tail

- name: Notify on Failure
if: failure()
if: failure() && github.event_name == 'schedule'
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We only want to be notified when this fails in the nightly run. Manual invocations shouldn't file an issue.

uses: actions/github-script@v6
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
Expand All @@ -36,6 +40,6 @@ jobs:
repo: context.repo.repo,
title: 'Nightly Test Corpus Failed',
body: 'The Nightly Test Corpus workflow failed. Please investigate.',
assignees: ['jmccarthy', 'philhassey'],
assignees: ['jmccarthy', 'philhassey', 'patjakdev'],
labels: ['bug']
})
18 changes: 6 additions & 12 deletions integration_tests/corpus_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,10 @@ var integrationFS embed.FS
type corpusTest struct {
Schema string `json:"schema"`
Policies string `json:"policies"`
ShouldValidate bool `json:"should_validate"`
ShouldValidate bool `json:"shouldValidate"`
Entities string `json:"entities"`
Requests []struct {
Desc string `json:"desc"`
Desc string `json:"description"`
Principal jsonEntity `json:"principal"`
Action jsonEntity `json:"action"`
Resource jsonEntity `json:"resource"`
Expand All @@ -45,13 +45,7 @@ func TestCorpus(t *testing.T) {
t.Fatal("err loading tests", p, err)
}
for _, tn := range more {
if strings.Contains(tn, "schema_") {
continue
}
if strings.Contains(tn, ".cedarschema.json") {
continue
}
if strings.Contains(tn, ".entities.json") {
if strings.Contains(tn, "entities.json") {
continue
}
tests = append(tests, tn)
Expand All @@ -74,8 +68,8 @@ func TestCorpus(t *testing.T) {
return "tmp/" + v
}

// detect possible corpus data pipeline failure
if len(tests) < 4_000 {
// detect possible corpus data pipeline failure. As of 2024/07/02, there were 1982 tests.
if len(tests) < 100 {
t.Fatalf("corpus test count too low: %v", len(tests))
}

Expand Down Expand Up @@ -115,7 +109,7 @@ func TestCorpus(t *testing.T) {
Resource: cedar.EntityUID(q.Resource),
Context: q.Context,
})
if ok != (q.Decision == "Allow") {
if ok != (q.Decision == "allow") {
t.Fatalf("got %v want %v", ok, q.Decision)
}
var errors []string
Expand Down
Loading