Testing roadmap

[MichaelClerx]

This ticket is an attempt to summarise the cross-repo stuff that's been happening with testing in PINTS.

See also this project

There are different aspects to testing PINTS.

• Unit tests are implemented for each bit of the code. For simple methods, these test the full functionality (is the result returned the correct one). For complex ones --- i.e. the central algorithms in samplers and optimisers --- these mostly test the API (does the result have the right shape, are errors thrown for incorrect input). Simpler parts of sampler & optimiser classes should still be tested, e.g. does get_x() return the value passed in with set_x().
  • PINTS has 100% unit test coverage and PRs can only be merged if they maintain this ratio.
  • It may be good to add value-based unit tests for the optimisers/samplers: these would seed, run, and simply check that the result is the same as the last time it was run (whether or not the result is "correct"): Add value-based (numerical) tests for all samplers / optimisers pints#1260
• Method merge tests are notebooks, stored in a separate repository, that show how well new notebooks perform on toy problems.
  • This ticket shows who is (or was!) assigned to each method. Feel free to add yourself and add another test!
  • Once you've picked a test, please also open an issue here, or assign yourself to an existing issue for this test
• A PR Template will be created that reminds people to have a merge test notebook ready when they introduce a new method: Create a PR template for introduction of new sampling methods into PINTS pints#1276
• Based on the method merge tests, we will create functional tests for each method: Create a functional testing module with tests for all methods pints#1294
• The results of functional tests will be monitored with changepoint detection algorithms: https://github.com/pints-team/functional-testing-2 and Overarching design for new functional testing change-point-testing#38 and Functional testing structure #13

Old stuff, probably best deleted

• First attempt at functional and change-point testing https://github.com/pints-team/change-point-testing
• Single page draft https://github.com/pints-team/functional-testing-paper