Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor Regression Tests to Remove External Deps & Other Failure Cases #2

Open
WilfredTA opened this issue May 4, 2022 · 0 comments
Labels

Comments

@WilfredTA
Copy link
Member

What is wrong
Currently, regression tests rely on both Minisat and SPASS. The regression test suite thus reports 8 total failures when run on a freshly cloned repository:
2 reported failures are from book code examples that are meant to raise errors.
1 reported failure is due to proof search failing when using a build generated by SML/NJ. This failure is not due to problems with Athena, rather, the non-deterministic nature of proof search procedures, possible timeouts, etc.
1 reported failure is due to missing the minisat_out.txt file in the ATHENA_HOME directory.
The other four failures are due to external calls to either Minisat or SPASS, which fail when they have not been installed.

How to fix
First, we will remove the intentionally incorrect code examples since they are meant for pedagogical purposes. Later, we might add to the python regression script an ability to expect an error from the Athena proof-checking.

Next, we will add to the regression testing script(s) an initial procedure that checks for Minisat and SPASS in the user's path. If Minisat is present, the original tests will be used, otherwise, it will trigger equivalent tests that use an Athena-native sat procedure. If SPASS is absent, tests that rely on SPASS will simply be skipped.

@WilfredTA WilfredTA added the tests label May 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant