Skip to content

Commit

Permalink
ENH: fixes from Jonny's review
Browse files Browse the repository at this point in the history
  • Loading branch information
lwasser committed Sep 30, 2023
1 parent 532d87a commit a22558d
Show file tree
Hide file tree
Showing 7 changed files with 228 additions and 66 deletions.
17 changes: 16 additions & 1 deletion ci-tests-data/ci.md
Original file line number Diff line number Diff line change
@@ -1 +1,16 @@
# CI for packaging
# What is continuous integration?

Some works needs to happen here...

on this page i think we can show various CI builds and talk about how they work.

we can then in the future link to our example package where those builds run.
and to tutorials

below is duplicate text...

## CI Environment

CI Environment: When you’re ready to publish your code online, you can setup Continuous Integration (CI). A CI platform will allow you to not only run your tests on various Python versions but also different operating systems like Mac, Linux and Windows. Tools like GitHub Actions and GitLab CI/CD make it easy for you to run tests on various Python versions, and even on Windows, Mac, and Linux. CI can finally be configured to ensure that tests run on every push and pull request to your repository. This ensures that any changes made to your package are tested across operations systems and Python versions before they are merged into the main branch of your codebase. &lt;<tests in ci link here>>

By embracing these testing practices, you can ensure that your code runs as you expect it to across the diverse landscapes of user environments.
15 changes: 15 additions & 0 deletions ci-tests-data/code-cov.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Code coverage

Code coverage is the amount of your package's codebase that is run as a part of running your project's tests. A good rule of thumb is to ensure that \*\*every line of

your code is run at least once during testing\**. However, note that good code coverage does not *guarantee\* that your package is well-tested. For example, you may run all of your lines of code, but not account for many edge-cases that users may have. Ultimately, you should think carefully about the way your package will be used, and decide whether your tests adequately cover all of that usage.

A common service for analyzing code coverage is [codecov.io](https://codecov.io/). This project is free for open source tools, and will provide dashboards that tell you how much of your codebase is covered during your tests. We recommend setting up an account, and using codecov to keep track of your code coverage.

```{figure} ../images/code-cov-stravalib.png
:height: 450px
:alt: Screenshot of the code cov service - showing test coverage for the stravalib package. in this image you can see a list of package modules and the associated number of lines and % lines covered by tests. at the top of the image you can see what branch is being evaluated and the path to the repository being shown.
the Code cov platform is a useful tool if you wish to visually track code coverage. Using it you can not only get the same summary information that you can get with pytest-cov extension. You can also get a visual representation of what lines are covered by your tests and what lines are not covered. Code cove is mostly useful for evaluating unit tests and/or how much of your package code is "covered. It however will not evaluate things like integration tests and end-to-end workflows. b
```
71 changes: 65 additions & 6 deletions ci-tests-data/index.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,78 @@
# Tests and data for your Python package

Tests and data are important!
In this section you will learn more about the importance of writing
tests for you Python package.

But why
::::{grid} 1 1 2 2
:class-container: text-center
:gutter: 3

:::{grid-item-card}
:link: write-tests
:link-type: doc

✨ Why write tests ✨
^^^

Learn more about the art of writing tests for your Python package.
Learn about why you should write tests and how they can help you and
potential contributors to your project.
:::

:::{grid-item-card}
:link: test-types
:link-type: doc

✨ Types of tests ✨
^^^
There are three general types of tests that you can write for your Python
package: unit tests, integration tests and end-to-end (or functional) tests. Learn about all three.
:::

:::{grid-item-card}
:link: run-tests
:link-type: doc

✨ How to Run Your Tests ✨
^^^

Learn more about what tools you can use to run tests. And how to run your
tests on different Python versions and different operating systems both on
your computer and using continuous integration on GitHub (or in any other CI).
:::

:::{grid-item-card}
:link: data
:link-type: doc

✨ Package data ✨
^^^
This section is current in progress... link coming soon
:::

:::{grid-item-card}
:link: ci
:link-type: doc

✨ Continuous Integration ✨
^^^
More on this coming soon... What is Continuous Integration? How do I set
it up for my project?
:::
::::

```{toctree}
:hidden:
:maxdepth: 2
:caption: Tests and data
Intro <self>
Writing tests <tests>
Test types: unit, integration, functional <test-types>
Run tests in CI & locally <run-tests>
Writing tests <write-tests>
Test types <test-types>
Run tests locally <run-tests>
Run tests in CI <tests-ci>
Code coverage <code-cov>
Package data <data>
CI <ci>
Intro to CI <ci>
```
155 changes: 114 additions & 41 deletions ci-tests-data/run-tests.md
Original file line number Diff line number Diff line change
@@ -1,69 +1,127 @@
# Running your tests

### What test suite tools should I use to run tests?
There are many different ways to setup and run tests for your Python package.
There are two types of tools that you should know about when setting up your
test suites:

We recommend using Pytest to set up testing infrastructure for your package as it is the most common testing tool used in the Python ecosystem.
1. A **testing framework**, as discussed here in this guide, is a package that provides a particular syntax and set of tools for both writing your tests and running them. Some testing frameworks also have plugins that add additional features such as evaluating how much of your code the tests cover or run. Below you will learn about the **pytest** framework.
2. A **runner tool** refers to a tool that allows you to easily automate running things such as tests in specific ways using user-defined commands. For instance it's useful to be able to run tests across different Python versions. Tools such as nox and tox allow you to easily run tests across Python versions. You can also use CI to run your tests not only across Python versions but also on various operating systems (Windows, Mac and Linux).

[Pytest package](https://docs.pytest.org/en/latest/) also has a number of extensions that can be used to add functionality such as:
You will learn about both test frameworks and runners on this page.

## What testing framework / package should I use to run tests?

We recommend using `Pytest` to build and run your package tests. Pytest is the most common testing tool used in the Python ecosystem.

[The Pytest package](https://docs.pytest.org/en/latest/) also has a number of
extensions that can be used to add functionality such as:

- [pytest-cov](https://pytest-cov.readthedocs.io/en/latest/) allows you to analyze the code coverage of your package during your tests, and generates a report that you can [upload to codecov](https://codecov.io/).

[Learn more about code coverage here.](code-cov)

```{note}
Reference the issue with running tests in vscode, breakpoints and –no-cov flag. Then link to tutorial that explains how to deal with this.
TODO: add note about running tests in vscode, breakpoints and –no-cov flag. Then link to tutorial that explains how to deal with this.
```

### Running tests
## Run tests using pytest

If you are using **pytest**, you can run your tests locally by
calling:

**Tox**
`pytest`

- [Tox](https://tox.wiki/en/latest/index.html#useful-links) is an automation tool that supports common steps such as building documentation, running tests across various versions of Python, and more. You can find [a nice overview of tox in the plasmaPy documentation](https://docs.plasmapy.org/en/stable/contributing/testing_guide.html#using-tox).
Or if you want to run a specific test file - let's call such file - filename.py - you can run:

**Make**
`pytest filename.py`

- Some developers opt to use Make for writing tests due to its versatility; it's not tied to a specific language and can be employed to orchestrate various build processes. However, Make's unique syntax and approach can make it more challenging to learn, particularly if you're not already familiar with it. Make also won't manage environments for you like nox will do.
Learn more from the [get started docs here](https://docs.pytest.org/en/7.1.x/getting-started.html).

**Hatch**
Running pytest on your computer is going to run your tests in whatever
Python environment you current have activated. This means that tests will be
run on a single version of Python and only on the operating system you
are running locally.

- [Hatch](https://github.com/ofek/hatch) is a modern end-to-end packaging tool that also has a nice build backend called hatchling. Hatch offers a tox-like setup where you can run tests locally using different Python versions. If you are using hatch to support your packaging workflow, you may want to also use its testing capabilities rather than using nox.
This is a great start to making your Python package more robust! However, your users may be using your package on different
versions of Python. Or they also may use other operating systems.

**[Nox](https://nox.thea.codes/):** is a Python-based automation tool that builds upon the features of both Make and Tox. Nox is designed to simplify and streamline testing and development workflows. Everything that you do with nox can be implemented using a Python-based interface.
Runners can begin to help with the issue of running tests
in various environments using single commands.

## Run your test suite
```{warning}
TODO: text to move / integrate / delete here...
TODO: add a note about the importance of using isolated environments...
Make sure your tests run across Python versions
Your package will be used by a diverse set of users who will be running various Python versions and using various operating systems. Thus you will want to run your test suite in all of these environments to identify issues that users may have before they run into them “in the wild”.
There are two primary ways that you can run tests - locally on your computer and in your CI build. We discuss both below.
There are two primary ways that you can run tests across environments:
## Section 2: Run tests on your computer
1. Locally on your computer and
2. in your CI build
In this guide we recommend Nox for running your tests locally. However you will also see packages using Tox or Hatch to create local environments with different versions of Python for you. Some packages use the more traditional Make approach. Note that Make does not manage environments for you.
add: JONNY Feedback - you need to isolate the environment so your tests don't spuriously pass because of some special thing about your computer (eg. some system package installed that you don't have specified in the dependencies)
As discussed above, it’s ideal for you to make sure that you run your tests on the various combinations of operating systems and Python versions that your users may be using.
```

### Why we like nox
### Tools to automate running your tests - Test runners

To run tests on various Python versions or in various specific environments with a single command, you can use a **test
runner**. A runner such as nox or tox will both create a virtual environment for you and run your tests in multiple environments. This allows you to test your package across Python versions.

Nox simplifies the process of creating and managing different testing environments, allowing you to check how your code behaves across various Python versions and configurations. With Nox, you can define specific test scenarios, set up virtual environments, and run tests across operating systems with a single command. Running tests locally using tools like Nox help you create controlled environment(s) to run your tests, making sure your code works correctly on your own computer before sharing it with others.
We will focus on [Nox](https://nox.thea.codes/) in this guide given it's Python-based. nox is a Python-based automation tool that builds upon the features of both Make and Tox. Nox is designed to simplify and streamline testing and development workflows. Everything that you do with nox can be implemented using a Python-based interface.

We recommend Nox for this purpose because you can also use it to setup other types of development builds including building your documentation, package distributions and more. Nox also supports working with conda, venv and other environment managers.
```{admonition} Other runner tools you'll see in the wild
:class: note
- **[Tox](https://tox.wiki/en/latest/index.html#useful-links)** is an automation tool that supports common steps such as building documentation, running tests across various versions of Python, and more. You can find [a nice overview of tox in the plasmaPy documentation](https://docs.plasmapy.org/en/stable/contributing/testing_guide.html#using-tox).
- **[Hatch](https://github.com/ofek/hatch)** is a modern end-to-end packaging tool that also has a nice build backend called hatchling. Hatch offers a tox-like setup where you can run tests locally using different Python versions. If you are using hatch to support your packaging workflow, you may want to also use its testing capabilities rather than using nox.
```

### Where does Make fit in?

Some developers use Make, which is a build automation tool, for running tests
due to its versatility; it's not tied to a specific language and can be used
to run various build processes. However, Make's unique syntax and approach can
make it more challenging to learn, particularly if you're not already familiar
with it. Make also won't manage environments for you like **nox** will do.

## Run tests across Python versions with nox

We recommend Nox for running tests locally across Python versions. Since **Nox**
is Python-based, it's more accessible for scientific Python users.

### Why we like nox

### Working with Nox
Nox simplifies creating and managing testing environments. With Nox, you can
set up virtual environments, and run tests across operating systems with a
single command.

Nox can also be used for other development tasks such as building
documentation, creating your package distribution, and testing across various
environment managers such as conda and pip.

## Environments

By default, Nox uses the Python built in ` venv` environment manager. A virtual environment (`venv`) is a self-contained Python environment that allows you to isolate and manage dependencies for different Python projects. It helps ensure that project-specific libraries and packages do not interfere with each other, promoting a clean and organized development environment.
By default, Nox uses the Python built in `venv` environment manager. A virtual environment (`venv`) is a self-contained Python environment that allows you to isolate and manage dependencies for different Python projects. It helps ensure that project-specific libraries and packages do not interfere with each other, promoting a clean and organized development environment.

Nox uses `venv`` by default. An example of using nox to run tests in venv environments for python versions 3.9, 3.10 and 3.11 is below.
An example of using nox to run tests in `venv` environments for Python versions 3.9, 3.10 and 3.11 is below.

```{warning}
Note that for the code below to work, you need to have all 3 versions of python installed on your computer for venv to find.
Note that for the code below to work, you need to have all 3 versions of Python installed on your computer for `venv` to find.
```

Below is an example of setting up nox to run tests using `venv`.
### Nox with venv environments

Below is an example of setting up nox to run tests using `venv` which is the built in environment manager that comes with base Python.

```python
import nox

# for this to run you will need to have python3.9, python3.10 and python3.11 installed on your computer. Otherwise nox will skip running tests for whatever versions are missing
# For this to run you will need to have python3.9, python3.10 and python3.11 installed on your computer. Otherwise nox will skip running tests for whatever versions are missing

@nox.session(python=["3.9", "3.10", "3.11"])
def test(session):
Expand All @@ -77,17 +135,40 @@ session.run("pytest")

```

Below is an example for setting up nox to use mamba (or conda).
Note that when you are using conda, it can automagically install
Above you create a nox session in the form of a function
with a @nox.session decorator. Notice that within the decorator you declare the versions of python that you
wish to run.

To run the above you'd use the command where `-s` stands for
session. Your function above is called test there for
the session name is test.

```
nox -s test
```

### Nox with conda / mamba

Below is an example for setting up nox to use mamba (or conda) for your
environment manager.
Note that when you are using conda, it can automatically install
the various versions of Python that you need. You won't need to install all three Python versions if you use conda/mamba, like you do with `venv`.

```{note}
For conda to work with nox, you will need to
install a conda-friendly version of Python. We suggest
the mamba-forge installation.
More on that here...<link to tutorial??>
```

```python
import nox

# The syntax below allows you to use mamba / conda as your environment manager, if you use this approach you don’t have to worry about installing different versions of Python

@nox.session(venv_backend='mamba', python=["3.9", "3.10", "3.11"])
def test(session):
def test_mamba(session):
"""Nox function that installs dev requirements and runs
tests on Python 3.9 through 3.11
"""
Expand All @@ -100,16 +181,8 @@ def test(session):
session.run("pytest")
```

### Running tests on CI

Running your test suite locally is useful as you develop code and also test new features or changes to the code base. However, you also will want to setup Continuous Integration (CI) to run your tests online. CI allows you to run all of your tests in the cloud. While you may only be able to run tests locally on a specific operating system that you run, in CI you can specify tests to run both on various versions of Python and across different operating systems.

CI can also be triggered for pull requests and pushes to your repository. This means that every pull request that you, your maintainer team or a contributor submit, can be tested. In the end CI testing ensures your code continues to run as expected even as changes are made to the code base. [Read more about CI here. ](https://docs.google.com/document/d/1jmo2l5u02c_F1zZi0bAIYXeJ6HiIryJbXzsNbMQQX6o/edit#heading=h.3mx2na93o7bf)
To run the above session you'd use:

### CI Environment

CI Environment: When you’re ready to publish your code online, you can setup Continuous Integration (CI). A CI platform will allow you to not only run your tests on various Python versions but also different operating systems like Mac, Linux and Windows. Tools like GitHub Actions and GitLab CI/CD make it easy for you to run tests on various Python versions, and even on Windows, Mac, and Linux. CI can finally be configured to ensure that tests run on every push and pull request to your repository. This ensures that any changes made to your package are tested across operations systems and Python versions before they are merged into the main branch of your codebase. &lt;<tests in ci link here>>

By embracing these testing practices, you can ensure that your code runs as you expect it to across the diverse landscapes of user environments.

pr checks.
```bash
nox -s test_mamba
```
Loading

0 comments on commit a22558d

Please sign in to comment.