Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] feat(conformance): adds example conformance workflows #4

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

[WIP] feat(conformance): adds example conformance workflows #4

wants to merge 2 commits into from

Conversation

jpower432
Copy link
Member

@jpower432 jpower432 commented Oct 4, 2024 •
edited
Loading

Summary

This is a draft PR is meant to open discussion on how testing the SDK core capabilities across languages could be accomplished.

What is proposed here

Taking a hybrid approach to SDK testing with only the common capabilities being tested centrally (essentially for testing common workflows and verifying outputs) and any unique functionalities being tested in the their respective SDK codebases.

What could be defined here

  • An agreed upon spec or set of capabilities
  • OSCAL test data
  • Common tests for valid OSCAL outputs including testing around constraints

What tools were used for testing

  • python, pytest, and GitHub Action (This would be consumed as a composite action in SDK repositories)
  • trestle is used as an example for integration testing

Rationale

  • This can reduce duplication of the same tests
  • Ensure the OSCAL documents processed and produced are uniform for common capabilities
  • Enhance collaboration between maintainer teams around the spec and criteria

Caveats

  • SDKs would need CLI wrappers to use this solution
  • Centralizing the test suite could hinder API development as it initially evolves (perhaps a phased approach where we can the tests after the initial spec is created?)

Additional Information

Triggered by oscal-compass/community#83

@jpower432 jpower432 linked an issue Oct 8, 2024 that may be closed by this pull request
Add workflows for SDK conformance testing #2
Open
2 tasks
@jpower432
Copy link
Member Author

Some specific decisions/questions where I think feedback would be helpful:

  1. We agreed that we should specific a common set of functionality for the SDKs here (discussed in the community meeting Oct 8th). We need to choose what format we want to use to describe this, but for now I have used Markdown as an simple example.
  2. Should we have common test data here?
  3. Should supported OSCAL schemas for the projects be moved here?
  4. Should we have a common set of tests for inputs and outputs or should tests be maintained in each repository that contains an SDK?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@butler54 butler54 Awaiting requested review from butler54

@vikas-agarwal76 vikas-agarwal76 Awaiting requested review from vikas-agarwal76

@degenaro degenaro Awaiting requested review from degenaro

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add workflows for SDK conformance testing
1 participant
@jpower432