diff --git a/ci-and-testing/intro.md b/ci-and-testing/intro.md deleted file mode 100644 index 00d5867f..00000000 --- a/ci-and-testing/intro.md +++ /dev/null @@ -1,7 +0,0 @@ -# CI and Testing - Coming Soon! - - -This section is evolving and should be published by the end of Spring 2023 - - -coming soon diff --git a/ci-tests-data/ci.md b/ci-tests-data/ci.md new file mode 100644 index 00000000..c59194ff --- /dev/null +++ b/ci-tests-data/ci.md @@ -0,0 +1 @@ +# CI for packaging diff --git a/ci-tests-data/code-cov.md b/ci-tests-data/code-cov.md new file mode 100644 index 00000000..e69de29b diff --git a/ci-tests-data/data.md b/ci-tests-data/data.md new file mode 100644 index 00000000..17f5215b --- /dev/null +++ b/ci-tests-data/data.md @@ -0,0 +1 @@ +# Data for packaging diff --git a/ci-tests-data/index.md b/ci-tests-data/index.md new file mode 100644 index 00000000..1926c8c2 --- /dev/null +++ b/ci-tests-data/index.md @@ -0,0 +1,17 @@ +# Tests and data for your Python package + +Tests and data are important! + +But why + +```{toctree} +:hidden: +:maxdepth: 2 +:caption: Tests and data + +Intro +Writing tests +Package data +CI + +``` diff --git a/ci-tests-data/tests.md b/ci-tests-data/tests.md new file mode 100644 index 00000000..f5d75cd3 --- /dev/null +++ b/ci-tests-data/tests.md @@ -0,0 +1,248 @@ +# Writing tests for your code + +Writing sets of test functions and methods, also known as test suites, for your package is important for both you as a maintainer, your users and any potential new contributors. Test suites consist of sets of functions, methods and classes that are written with the intention of making sure a specific part of your code works as you expected it to. + +## Why create tests for your package + +Tests serve as a safety net for changes to your code. Tests help you identify and fix bugs before they reach users. They also provide confidence that code changes submitted by contributors won't break existing functionality. Finally, a comprehensive test suite can empower you to refactor code with less fear of unintentionally breaking things. Finally tests allows you to account for various edge cases that may break + +your package's functionality. For instance what happens if a function expects a pandas ` dataframe` as an input but a user provides a numpy `array`? Does your code fail gracefully and let the user know what happened? Or does your code break leaving a user frustrated as they +don't know what broke things? + +:::{note} + +For a good introduction to testing, see [this Software Carpentry lecture](https://swcarpentry.github.io/python-novice-inflammation/10-defensive.html) + +::: + +Imagine you're working on a puzzle where each puzzle piece represents a function, method, class or attribute in your Python package that you want other people to be able to use. Would you want to give someone a puzzle that has missing pieces or pieces that don't fit together? Providing people with the right puzzle pieces that work together can be compared to writing tests for your Python package. + + + +### Why write tests? + +Here's why writing tests for your Python package is important, especially if you're just starting out: + +- Catching Mistakes: Tests act like a safety net. When you make changes or add new features to your package, tests can quickly tell you if you accidentally broke something that was working fine before. +- Saving Time: Imagine you have a magic button that can automatically check if your package is still working properly. Tests are like that magic button! They can run all those checks for you in an instant, saving you lots of time. +- Teamwork Made Easy: If you're working with others, tests help everyone stay on the same page. Your tests explain how your package is supposed to work, making it easier for others to understand and contribute to your project. +- Fearless Refactoring: Refactoring means making improvements to your code without changing its behavior. Tests give you the courage to do this. You can confidently make changes because tests will quickly show you if something broke. +- Documentation: Tests serve as examples of how to use your package. When others want to use your tools, they can look at your tests to see how everything is supposed to fit together. +- Long-Term ease of maintenance: As your package grows, it becomes harder to keep track of everything. Tests ensure that your code behaves as expected, even as you make changes over time. This helps prevent future headaches! + +By running these tests, each time you or a contributor makes a change to your code-base, you can catch issues and things that may have changed in your code base. This ensures that your software behaves the way you expect it to. + +````{note} + +### BREAKOUT - Test example + +Let’s say you have a python function that adds two numbers a and b together. + +```python +def add_numbers(a, b): + + return a + b +``` + +A test to ensure that function runs as you might expect when provided with different numbers might look like this: + +```python +def test_add_numbers(): + + result = add_numbers(2, 3) + + assert result == 5, f"Expected 5, but got {result}" + + result = add_numbers(-1, 4) + + assert result == 3, f"Expected 3, but got {result}" + + result = add_numbers(0, 0) + + assert result == 0, f"Expected 0, but got {result}" + +test_add_numbers() + +``` +```` + +🧩🐍 + +### How do I know what type of tests to write? + +

>>>>> gd2md-html alert: Definition ↓↓ outside of definition list. Missing preceding term(s)?
(Back to top)(Next alert)
>>>>>

+ +:::{note} + +This section has been adapted from a presentation by Nick Murphy - https://zenodo.org/record/8185113 + +

>>>>> gd2md-html alert: Definition ↓↓ outside of definition list. Missing preceding term(s)?
(Back to top)(Next alert)
>>>>>

+ +::: + +At this point, you may be wondering - what should you be testing in your package? Below are a few examples: + +**test some typical cases: **Test that the package functions as you expect it to when users use it. For instance, if your package is supposed to add two numbers, test that the outcome value of adding those two numbers is correct. + +**Test special cases: **Sometimes there are special or outlier cases. For instance, if a function performs a specific calculation that may become problematic closer to the value = 0, test it with the input of both 0 and + +**Test at and near the expected boundaries : **If a function requires a value that is greater than or equal to 1, make sure that the function still works with both the values 1 and less than one and 1.001 as well (something close to the constraint value).. + +**Test that code fails correctly: **If a function requires a value greater than or equal to 1 + +test at 0.999 - Make sure that the function fails gracefully when given unexpected values and help and that the user can easily understand why if failed (provides a useful error message) + +## Different types of tests: Unit, Integration & Functional Tests + +There are different types of tests that you want to consider when creating your test suite: + +1. Unit tests +2. Integration +3. Functional (also known as end-to-end) tests + +### Unit Tests + +A unit test in Python is a type of software testing where individual components or units of code are tested in isolation to ensure that they work correctly. The goal of unit testing is to verify that each part of the software, typically at the function or method level, performs its intended task correctly. Unit tests can be compared to examining each piece of your puzzle to ensure parts of it are not broken. If all of the pieces of your puzzle don’t fit together, you will never complete it. Similarly, when working with code, tests ensure that each function, attribute, class, method works properly when isolated. + +**Unit test example: **Pretend that you have a function that converts a temperature value from Celsius to Fahrenheit. A test for that function might ensure that when provided with a value in Celsius, the function returns the correct value in degrees Fahrenheit. That function is a unit test. It checks a single unit (function) in your code. + +

>>>>> gd2md-html alert: inline image link here (to images/image1.png). Store image on your image server and adjust path/filename/extension if necessary.
(Back to top)(Next alert)
>>>>>

+ +![alt_text](images/image1.png "image_tooltip") + +Caption If puzzle pieces have missing ends, they can’t work together with other elements in the puzzle. The same is true with individual functions, methods and classes in your software. The code needs to work both individually and together to perform certain sets of tasks. + +_Alt: image of two puzzle pieces with some missing parts. The puzzle pieces are purple teal yellow and blue. The shapes of each piece don’t fit together. _ + +### Integration tests + +Integration tests involve testing how parts of your package work together or integrate. Integration tests can be compared to connecting a bunch of puzzle pieces together to form a whole picture. Integration tests focus on how different pieces of your code fit and work together. For example, if you had a series of steps that collected temperature data in a spreadsheet, converted it from degrees celsius to Fahrenheit and then provided an average temperature for a particular time period. An integration test would ensure that all parts of that workflow behaved as expected. + +​​ + +

>>>>> gd2md-html alert: inline image link here (to images/image2.png). Store image on your image server and adjust path/filename/extension if necessary.
(Back to top)(Next alert)
>>>>>

+ +![alt_text](images/image2.png "image_tooltip") + +### end to end (functional) tests + +End-to-end tests (also referred to as functional tests by some) in Python are like comprehensive checklists for your software. They simulate real user end-to-end workflows to make sure the code base supports real life applications and use-cases from start to finish. These tests help catch issues that might not show up in smaller tests and ensure your entire application or program behaves correctly. Think of them as a way to give your software a final check before it's put into action, making sure it's ready to deliver a smooth experience to its users. + +**Breakout: ** For scientific packages, creating short tutorials that highlight core workflows that your package supports, that are run when your documentation is built could also serve as end-to-end tests. + +### Comparing unit, integration and end-to-end tests + +Unit tests, integration tests, and end-to-end tests have complementary advantages and disadvantages. The fine-grained nature of unit tests make them well-suited for isolating where errors are occurring, but not very suitable for verifying that different sections of code work together. Integration and end-to-end tests verify that the different portions of the program work together, but are less well-suited for isolating where errors are occurring. A thorough test suite should have a mixture of unit tests, integration tests, and functional tests. + +## Code coverage + +Code coverage is the amount of your package's codebase that is run as a part of running your project's tests. A good rule of thumb is to ensure that \*\*every line of + +your code is run at least once during testing\**. However, note that good code coverage does not *guarantee\* that your package is well-tested. For example, you may run all of your lines of code, but not account for many edge-cases that users may have. Ultimately, you should think carefully about the way your package will be used, and decide whether your tests adequately cover all of that usage. + +A common service for analyzing code coverage is [codecov.io](https://codecov.io/). This project is free for open source tools, and will provide dashboards that tell you how much of your codebase is covered during your tests. We recommend setting up an account, and using codecov to keep track of your code coverage. + +

>>>>> gd2md-html alert: inline image link here (to images/image3.png). Store image on your image server and adjust path/filename/extension if necessary.
(Back to top)(Next alert)
>>>>>

+ +![alt_text](images/image3.png "image_tooltip") + +### What test suite tools should I use to run tests? + +We recommend using Pytest to set up testing infrastructure for your package as it is the most common testing tool used in the Python ecosystem is + +[the pytest package](https://docs.pytest.org/en/latest/). Pytest also has + +a number of extensions that can be used in addition to pytest's core functionality. + +Such as [pytest-cov](https://pytest-cov.readthedocs.io/en/latest/) allows you to analyze the code coverage of your package during your tests, and generates a report that you can [upload to codecov](https://codecov.io/). + +_Breakout: probably useful to mention the issue with running tests in vscode, breakpoints and –no-cov flag. Then link to tutorial that explains how to deal with this._ + +### Running tests?? + +- [Nox:](https://nox.thea.codes/en/stable/index.html) Nox is a great Python focused tool that allows you to not only run tests in isolated environments, but also create workflows to run your documentation. + +**Breakout: **Other tools that you may see in the wild to run tests + +[Tox](https://tox.wiki/en/latest/index.html#useful-links): tox is an automation tool that supports common steps such as building documentation, running tests across various versions of python and more. You can find a nice overview of tox in the plasmaPy documentation. <TODO: ADD LINK TO GUIDE> + +**Make : **- Make is a more generic… (doesn’t handle envts) + +**Hatch:** - hatch is a modern end-to-end packaging tool that also has a nice backend called hatchling. Hatch offers a tox like setup where you can run tests locally using different python versions. If you are using hatch to support your packaging workflow, you may want to also use its testing capabilities rather than using nox! + +## Run your test suite + +Your package will be used by a diverse set of users who will be running various Python versions and using various operating systems. Thus you will want to run your test suite in all of these environments to identify issues that users may have before they run into them “in the wild”. + +There are two primary ways that you can run tests: + +### Locally on your computer + +In this guide we recommend Nox for running your tests locally. However you will also see packages using , Tox, or Hatch to create local environments with different versions of python for you. Some packages use the more traditional Make approach. Note that Make does not manage environments for you. + +### CI Environment + +CI Environment: When you’re ready to publish your code online, you can setup Continuous Integration (CI). A CI platform will allow you to not only run your tests on various python versions but also different operating systems like Mac, Linux and Windows. Tools like GitHub Actions and GitLab CI/CD make it easy for you to run tests on various Python versions, and even on Windows, Mac, and Linux. CI can finally be configured to ensure that tests run on every push and pull request to your repository. This ensures that any changes made to your package are tested across operations systems and python versions before they are merged into the main branch of your codebase. <> + +By embracing these testing practices, you can confidently ensure that your code runs as you expect it to across the diverse landscapes of user environments. + +pr checks. + +## Section 2: Running tests locally + +As discussed above, it’s ideal for you to make sure that you run your tests on the various combinations of operating systems and python versions that your users may be using. There are some great tools out there that you can use to do this including: + +- **Make: **Make is a build automation tool that you can use to create workflows such as running tests, building documentation and building your package. When using Make, you will create a Makefile that defines each workflow and the associated workflow “call” that you want to run . +- **Tox: **Tox is a command-line tool used in Python development for testing and packaging software projects. It automates the process of setting up virtual environments, running test suites across different Python versions, and ensuring the project is compatible and consistent across various environments. +- **Hatch:** Hatch has some built in functionality for running local tests across python versions that would replace the need for a tool like TOX. . If you are using Hatch, you may not need to use Nox. +- **Nox : **Nox is a Python-based automation tool that builds upon the features of both Make and Tox. Nox is designed to simplify and streamline testing and development workflows. Everything that you do with nox can be implemented using a Python-based interface. + +In this guide you will learn about Nox. Nox simplifies the process of creating and managing different testing environments, allowing you to check how your code behaves across various Python versions and configurations. With Nox, you can define specific test scenarios, set up virtual environments, and run tests across operating systems with a single command. Running tests locally using tools like Nox help you create controlled environment(s) to run your tests, making sure your code works correctly on your own computer before sharing it with others. + +We recommend Nox for this purpose because you can also use it to setup other types of development builds including building your documentation, package distributions and more. Nox also supports working with conda, venv and other environments. + +### Working with Nox + +More here ….. + +Nox uses venv by default - so to run using your venv envrts you can use + +Demo example below + +```python +import nox + +# for this to run you will need to have python3.9, python3.10 and python3.11 installed on your computer. Otherwise nox will skip running tests for whatever versions are missing + +@nox.session(python=["3.9", "3.10", "3.11"]) +def test(session): + +# install +session.install(".[all]") +# install dev requirements +session.install("-r", "requirements-dev.txt") +# Run tests +session.run("pytest") + +# the syntax below allows you to use mamba / conda as your environment manager, if you use this approach you don’t have to worry about installing different versions of python + +@nox.session(venv_backend='mamba', python=["3.9", "3.10", "3.11"]) +def test(session): + """Nox function that installs dev requirements and runs + tests on python 3.9 through 3.11 + """ + + # install dev requirements + session.install(".[all]") + #install dev requirements + session.install("-r", "requirements-dev.txt") + # Run tests using any parameters that you need + session.run("pytest") + +``` + +### Running tests on CI + +Running your test suite locally is useful as you develop code and also test new features or changes to the code base. However, you also will want to setup Continuous Integration (CI) to run your tests online. CI allows you to run all of your tests in the cloud. While you may only be able to run tests locally on a specific operating system that you run, in CI you can specify tests to run both on various versions of Python and across different operating systems. + +CI can also be triggered for pull requests and pushes to your repository. This means that every pull request that you, your maintainer team or a contributor submit, can be tested. In the end CI testing ensures your code continues to run as expected even as changes are made to the code base. [Read more about CI here. ](https://docs.google.com/document/d/1jmo2l5u02c_F1zZi0bAIYXeJ6HiIryJbXzsNbMQQX6o/edit#heading=h.3mx2na93o7bf) diff --git a/index.md b/index.md index 164a3bec..a1440ae1 100644 --- a/index.md +++ b/index.md @@ -20,7 +20,8 @@ Packaging :hidden: :caption: CI and Testing -CI & Tests +CI & Tests + ```