Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support a non-exiting run mode #39

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

itowlson
Copy link

@itowlson itowlson commented Oct 1, 2024

The Spin runtime tests use conformance-tests as a library (https://github.com/fermyon/spin/blob/967fdf368612478cef176bccc491faffad680050/tests/runtime.rs#L31). However, run_tests entry point calls libmimic::Conclusion::exit(), which causes the test process to exit at the end of the tests, with the result of the libmimic tests only. This means that if a Spin runtime test fails but all conformance tests succeed, the overall test suite still passes, disguising the test failure unless the avid reader pores over the logs.

This PR proposes splitting the run_tests entry point into run_tests_and_exit, which retains the current behaviour, and run_tests_to_conclusion, which instead returns the Conclusion object. libmimic still prints all progress and results as normal.

With this in place, the Spin runtime test that checks conformance will be able to examine Conclusion::has_failed and propagate failure (e.g. by returning an error such as "One or more conformance tests failed") or pass on success, without interfering with other tests.

Of course the names and behaviour are totally up for grabs (some names are deliberate to make consumers aware of changed behaviour, but that may be misguided), and if you feel there's a better way of managing this then please feel free to circular-file this - I'm not at all familiar with the code base or with other use cases I'm afraid.

Signed-off-by: itowlson <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant