Formalising Testing Framework and Approach

[al-fisher]

Introducing a greater breakdown and categorisation of our various testing scripts and procedures to enable further scaling and automation beyond our current CI and compliance.

Below is a first list of, I think, the key categories that we need to focus on. Not all will be fully automated in the first instance and priorities will vary naturally.

Listed below for comments/input and then more granular plans to be formulated around each as needed.
Focus on Interoperability Test for 3.1 Milestone as discussed and planned.

I. Compliance

• Development Process Compliance Tests
  • e.g. branch naming, issue and PR template checks, label checking etc.
• Code Compliance Tests
  • Review BHoM_Test repo and Compliance_Test , Coverage_Test etc.
  • Creation of Wiki - explaining BHoM Compliance which can hyperlink to each encoded Compliance Test for clarity and transparency for all developers.

II. Functionality

• Data-driven Unit Tests
  • datasets required of serialised method base, input & expected output as per our BH.oM.Testing.UnitTest
  • Generation of dataset as per previous gh approach to be revisited and polished
• Interoperability Tests
  • In the first instance - scripts testing each of the implemented BHoM_Adapter functionality
• Issue Tests
  • As currently created to enable review of each PR.
    Issue_Tests guidelines and best practice to be developed further to enable associated Regression Tests where appropriate
• Regression Tests
  • Ensuring fixed bugs do not reappear. Ensuring previous working features do not break for subsequent Betas
  • Regression Tests are not necessarily a separate type of test - but keen to include as a discrete category to be be included in our process of ongoing development of tests and integrated into process of raising and successfully closing PR.
• Macro Tests
  • any combination of the above
• Backwards Compatibility Tests
  • Tests targeting deserialisation of previous versions of BHoM.
  • Potential to include creation as part of deprecation
• Performance tests
• Define metric of speed and checking that subsequent PRs are not X times slower to run tests

III. System

• Installer Testing
  • Works on end user machine
  • Metrics and licencing etc. information is correct.
  • Validation that installer accurately recreates BHoM environment on master say
• Environment Testing
  • Both code developer and end user (see above)

IV. User Testing

• Beta Testing
  • Formalised and focused user testing as part of Beta Release process
  • Comms around best practice raising of issues/ideas from wider user community beyond release, and with Beta in the wild. To all encourage engagement
• Sample Testing
  • Checking key samples are all still working for each release

[FraserGreenroyd]

I am changing the milestone for this because there isn't enough time left in 3.1 to achieve everything listed. There are some workshops scheduled to discuss this but most work will be completed (hopefully) in 3.2.

[FraserGreenroyd]

To update this issue and work out the final plans for close out.

Items under 1 - Compliance:

These have been completed and rolled out for some time now, with @BHoMBot check compliance being a staple of many pull requests handling our compliance guidelines for BHoM contribution, outlined both in the documentation and this repository for public record and examination.

Items under 2 - Functionality:

• Data Driven Unit-Tests have been rolled out and are slowly being implemented, [BHoM_Engine)(https://github.com/BHoM/BHoM_Engine/tree/develop/.ci/Datasets) boasting the current lead in having these implemented. They also run nightly via BHoMBot, aiding in the pick up of issues when they occur.
• Interoperability Tests - these have been started, but need final roll out onto Bot infrastructure to be fully complete. However, thanks to @alelom's work on Test Driven Development, some adapters are now producing Unit Tests via the NUnit framework to support and assist in these tests. Likewise, @IsakNaslundBh and @peterjamesnugent have completed sterling work on the framework within Test_Toolkit for providing functionality to aid building and running interoperability tests locally.
• Issue Tests - @alelom has provided a framework for Test Driven Development which aids with issue tests. We have also been better at providing test data on issues I think, though more improvements could be made, it is down to community engagement to provide and further this.
• Regression Tests - ongoing and roll out onto Bot infrastructure required to make complete.
• Macro Tests - we have a very well formalised end-of-milestone procedure, including a final testing sprint (sprint 6) which has been working well for the last few years and improving reliability of the code-base from beta-to-beta. More work could be done to automate these, but @jamesramsden-bh and @kprusicka have both worked to achieve this within the Environments discipline in the last 12 months.
• Backwards compatibility tests - we have the versioning check, which has again become a staple and requirement for any repository included within the beta. More work is desired to automate some of this, and speed it up, but those are now DevOps tasks and are extra to this particular issue and have been captured elsewhere as appropriate.
• Performance tests - not really started, this needs some work to close out.

Items under 3 - System:

Test procedures exist for testing the installer which are run every beta and periodically if issues arise. There are some issues with installing BHoM over a previous install by another user which are being tracked separately. Validation against the BHoM main branches hasn't been implemented as the main (or develop) change so frequently that it is difficult to have a static test for these. Instead, installer checks have become a staple of Pull Requests, also handled by BHoMBot to ensure pull requests don't merge if they introduce issues with the installer creation process.

Items under 4 - User Testing

As mentioned, the final sprint of each milestone is a testing sprint and this has been going well in recent milestones. This has formalised our testing process for beta delivery massively, though some documentation is still outstanding.

Sample testing is also now conducted as part of Toolbox testing, led by @kayleighhoude @michaelhoehn and @enarhi in particular.

As such, although it has been a long time since this issue was updated, I think it's fair to say we have massively improved and formalised the testing frameworks and approaches undertaken within the development teams for BHoM contribution, and I don't think anyone would disagree to say we've come on along way since January 2020 when this issue was first raised.

To summarise the issues outstanding which prevent this issue being marked as complete currently, are to the best of my knowledge, as follows:

• Automation of Interoperability Tests
• Introduction of Performance Tests
• Public documentation of end of milestone testing
• Documentation of full testing processes, including link to this issue

I think once we have tackled these items, this issue could be safely closed as complete, though should also be linked in documentation for future reference.