[TESTING.md] Enhance TESTING docs (#2611)

* Update TESTS.md * Update TESTS.md * Update TESTS.md * Start of TOOLS.md * Finished writing VS Code part * Added Pycharm docs & relative img links * Update TOOLS.md * Added Spyder Docs * Added VENV docs * Added conda environments docs * Wrote Windows tutorial for adding to path * Improved shared test file and added disclaimer to TOOLS.md * Added docs for JupyterLab and Sublime Text * Added Virtualenvwrapper piece. * Cleaned up. * Updated image paths * Fixed Images x2 * Images have absolute path * Update docs/TESTS.md Co-authored-by: Isaac Good <[email protected]> * Update docs/TESTS.md Co-authored-by: Isaac Good <[email protected]> * Update docs/TESTS.md Co-authored-by: Isaac Good <[email protected]> * Update docs/TESTS.md Co-authored-by: Isaac Good <[email protected]> * Update docs/TESTS.md Co-authored-by: Isaac Good <[email protected]> * Update docs/TESTS.md Co-authored-by: Isaac Good <[email protected]> * Update docs/TOOLS.md Co-authored-by: Isaac Good <[email protected]> * Update docs/TOOLS.md Co-authored-by: Isaac Good <[email protected]> * Update docs/TESTS.md Co-authored-by: Isaac Good <[email protected]> * Update docs/TOOLS.md Co-authored-by: Isaac Good <[email protected]> * Update docs/TOOLS.md Co-authored-by: Isaac Good <[email protected]> * Update exercises/shared/.docs/tests.md Co-authored-by: Isaac Good <[email protected]> * Update exercises/shared/.docs/tests.md Co-authored-by: Isaac Good <[email protected]> * Fixed capitalization * Update docs/TOOLS.md Co-authored-by: Isaac Good <[email protected]> * Fixed language Co-authored-by: Isaac Good <[email protected]> * Update docs/TOOLS.md Co-authored-by: Isaac Good <[email protected]> * Update docs/TOOLS.md * Update docs/TOOLS.md * Update docs/TESTS.md * Update docs/TOOLS.md * Update docs/TOOLS.md * Update docs/TESTS.md Co-authored-by: Mukesh Gurpude <[email protected]> * Update docs/TESTS.md Co-authored-by: Mukesh Gurpude <[email protected]> * Update docs/TESTS.md Co-authored-by: Mukesh Gurpude <[email protected]> * Update docs/TOOLS.md Co-authored-by: Mukesh Gurpude <[email protected]> * Update docs/TESTS.md Co-authored-by: ynfle <[email protected]> * Added pytest.ini docs * Update docs/TESTS.md Co-authored-by: BethanyG <[email protected]> * Update docs/TESTS.md Co-authored-by: BethanyG <[email protected]> * Update docs/TESTS.md Co-authored-by: BethanyG <[email protected]> * Update docs/TOOLS.md Co-authored-by: BethanyG <[email protected]> * Apply suggestions from code review Co-authored-by: BethanyG <[email protected]> * Update docs/TOOLS.md Co-authored-by: BethanyG <[email protected]> * Update docs/TOOLS.md Co-authored-by: BethanyG <[email protected]> * Update docs/TOOLS.md Co-authored-by: BethanyG <[email protected]> * Update docs/TOOLS.md Co-authored-by: BethanyG <[email protected]> * Update headers and links Co-authored-by: Isaac Good <[email protected]> Co-authored-by: Mukesh Gurpude <[email protected]> Co-authored-by: ynfle <[email protected]> Co-authored-by: BethanyG <[email protected]>
exercism · Oct 21, 2021 · 69bb569 · 69bb569
1 parent 5a88f9d
commit 69bb569
Show file tree

Hide file tree

Showing 15 changed files with 354 additions and 103 deletions.
diff --git a/docs/TESTS.md b/docs/TESTS.md
@@ -1,115 +1,211 @@
 # Tests
 
-## Tools to install
+- [Tests](#tests)
+  - [Pytest](#pytest)
+    - [Installing pytest Globally](#installing-pytest-globally)
+      - [Windows](#windows)
+      - [Linux / MacOS](#linux--macos)
+      - [Installing pytest within a virtual environment](#installing-pytest-within-a-virtual-environment)
+    - [Running the tests](#running-the-tests)
+      - [Failures](#failures)
+    - [Extra arguments](#extra-arguments)
+      - [Stop After First Failure [`-x`]](#stop-after-first-failure--x)
+      - [Failed Tests First [`--ff`]](#failed-tests-first---ff)
+      - [Recommended Workflow](#recommended-workflow)
+      - [Using PDB, the Python Debugger, with pytest](#using-pdb-the-python-debugger-with-pytest)
+  - [Extending your IDE](#extending-your-ide)
+  - [Additional information](#additional-information)
+    - [Adding pytest to your PATH](#adding-pytest-to-your-path)
+      - [Windows](#windows-1)
+    - [Fixing warnings](#fixing-warnings)
+
+---
+
+## Pytest
+
+_Official pytest documentation can be found on the [pytest Wiki](https://pytest.org/en/latest/) page._
+
+Pytest lets you test your solutions using our provided tests, and is what we use to validate your solutions on the website.
+
+### Installing pytest Globally
+
+Pytest can be installed and updated using the built-in Python utility `pip`.
+
+#### Windows
+
+```powershell
+PS C:\Users\foobar> python3 -m pip install pytest pytest-cache pytest-subtests pytest-pylint
+Successfully installed pytest-6.2.5 ...
+```
 
-We recommend you install [pytest](http://pytest.org/en/latest/) with the following plugins:
+#### Linux / MacOS
 
--  [pytest-cache](http://pythonhosted.org/pytest-cache/)
--  [pytest-subtests](https://github.com/pytest-dev/pytest-subtests)
--  [pytest-pylint](https://github.com/carsongee/pytest-pylint)
+```bash
+$ python3 -m pip install pytest pytest-cache pytest-subtests pytest-pylint
+Successfully installed pytest-6.2.5 ...
 
-The PyTest [Getting Started Guide](https://docs.pytest.org/en/latest/getting-started.html) has quick general instructions, although they do not cover installing the plugins.
-Continue reading below for plugin installation.
+```
 
-We also recommend [pylint](https://pylint.pycqa.org/en/latest/user_guide/), as it is part of our automated feedback on the website, and can be a very useful (if noisy!) code analysis tool.
+To check if the installation was succesful:
 
-Pylint can be a bit much, so this [tutorial](https://pylint.pycqa.org/en/latest/tutorial.html) can be helpful for getting started, as can this overview of [Code Quality: Tools and Best Practices](https://realpython.com/python-code-quality/) from Real Python.
+```bash
+$ python3 -m pytest --version
+pytest 6.2.5
+```
 
-Finally, [this site](https://pycodequ.al/docs/pylint-messages.html) is a great place to look up more human-readable descriptions of Pylint linting messages.
+If you do not want to precede every command with `python3 -m` please refer to [adding to PATH](#adding-to-path) at the end of this document.
 
+#### Installing pytest within a virtual environment
 
-### Installing `pytest`
+*For more information about virtual environments please refer to the [TOOLS](./TOOLS.md) file.*
 
-To install `pytest`, run the following command in the environment (_active `venv`, `conda env`, `docker container`, `vm` or other project space with Python 3.8 installed_):
+When installing pytest or any other module(s), make sure that you have [activated your environment](.\TOOLS.md#activating-your-virtual-environment). After which you can run:
 
 ```bash
-pip3 install pytest pytest-cache pytest-subtests pytest-pylint
+$ pip install pytest pytest-cache pytest-subtests pytest-pylint
+Successfully installed pytest-6.2.5 ...
 ```
 
-If you get a `command not found` response from your system, you can find a
-tutorial on how to install `pip`
-[here](https://pip.pypa.io/en/stable/installing/).
+### Running the tests
 
-Once the installation has completed, you can check what the default version of `pytest` is by running the following:
+To run the tests, go to the folder where the exercise is stored using `cd` in your terminal (_replace `{exercise-folder-location}` below with the path_).
 
 ```bash
-pytest --version
+$ cd {exercise-folder-location}
 ```
 
-## Testing
+The file you'll want always ends with `_test.py`.
+This file contains the tests for your solution, and are the same tests which run on the website.
+Now run the following command in your terminal, replacing `{exercise_test.py}` with the location/name of the test file:
 
-### Run All Tests for an Exercise
+```bash
+$ python3 -m pytest {exercise_test.py}
+==================== 7 passed in 0.08s ====================
+```
 
-To run all tests for a specific exercise (_we will take the `bob.py` exercise as
-an example here_), navigate to the directory where that exercise has been
-downloaded and run the following in the terminal:
+#### Failures
+
+When your code returns an incorrect or unexpected value, pytest returns all the failed tests and the returned and expected values of each. Look at the following failed test file:
 
 ```bash
-pytest bob_test.py
+$ python3 -m pytest {exercise_test.py}
+=================== FAILURES ====================
+______________ name_of_failed_test ______________
+# Test code inside of {exercise_test.py} that failed.
+...
+E   TypeOfError: ReturnedValue != ExpectedValue
+
+exercise_test.py:{line_of_failed_test}: TypeOfError
+============ short test summary info ============
+FAILED exercise_test.py::ExerciseTest::name_of_failed_test
+========== 1 failed, 2 passed in 0.13s ==========
 ```
 
-**Note:** To run the tests you need to pass the name of the testsuite file to
-`pytest` (_generally, this will be the file ending with `_test.py`_), **NOT** the file that was
-created to solve the exercsim problem (which is the _solution implementation_).
-`PyTest` needs the test definitions in the `_test.py` file in order to know how to call, run, and exercise the _solution implementation_ code.
-Since there are no test cases defined in the _solution implementation_, `pytest` will just return a positive result, specifying that it found and ran zero tests. Like this:
+### Extra arguments
+
+If you really want to be specific about what pytest returns on your screen, here are some handy arguments that allows you to configure its behavior.
 
+#### Stop After First Failure [`-x`]
 
+Running the `pytest -x {exercise_test.py}` command, will run the tests like normal, but will stop the tests after the first failed test. This will help when you want to debug a single failure at a time.
+
+```bash
+$ python -m pytest -x example_test.py
+=================== FAILURES ====================
+_______________ example_test_foo ________________
+...
+...
+============ short test summary info ============
+FAILED example_test.py::ExampleTest::example_test_foo
+!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!
+========== 1 failed, 5 passed in 0.28s ==========
 ```
-=============================  bob.py  ==============================
 
----------------------------------------------------------------------
+#### Failed Tests First [`--ff`]
 
-Ran 0 tests in 0.000s
+`pytest-cache` remembers which tests failed last time you ran `pytest`, running `pytest --ff {exercise_test.py}` will run those previously failed tests first, then it will continue with the rest of the tests. This might speed up your testing if you are making a lot of smaller fixes.
 
-OK
+```bash
+$ python -m pytest --ff bob_test.py
+==================== 7 passed in 503s ====================
 ```
 
+#### Recommended Workflow
 
-### More `pytest` Examples
+We recommend using the following commands to make your debugging easier and (possibly) faster:
 
-Below are some additional examples and details for getting up and running quickly with Pytest.
-[How to invoke pytest](https://docs.pytest.org/en/latest/how-to/usage.html#usage) and [pytest command-line flags](https://docs.pytest.org/en/latest/reference/reference.html#command-line-flags) offer full details on all of pytests run options.
+First change your working directory to the directory of the exercise you want to test:
 
+```bash
+cd path/to/exercise
+```
 
-#### Stop After First Failure
-The above will run all the tests, whether they fail or not. If you'd rather stop
-the process and exit on the first failure, run:
+Then, run the tests together with the previously explained arguments `-x` and`--ff`:
 
 ```bash
-pytest -x bob_test.py
+pytest -x -ff bob_test.py
 ```
 
-#### Failed Tests First
+This will test your solution. When `pytest` encounters a failed test, the program will stop and tell you which test failed. When you run the test again, `pytest` will first test that failed test, then continue with the rest.
+
+#### Using PDB, the Python Debugger, with pytest
 
-`pytest-cache` remembers which tests failed, and can run those tests first.
+If you want to truly debug like a pro, use the `--pdb` argument after the `pytest` command. 
 
 ```bash
-pytest --ff bob_test.py
+$ python3 -m pytest --pdb bob_test.py
+=============== 4 passed in 0.15s ===============
 ```
 
-#### Running All Tests for All Exercises
+When a test fails, `PDB` allows you to look at variables and how your code responds. If you want to learn how to use the `PDB` module, have a look at the [Python Docs](https://docs.python.org/3/library/pdb.html#module-pdb) or [this](https://realpython.com/python-debugging-pdb/) Real Python article.
 
-```bash
-cd exercism/python/
-pytest
-```
+## Extending your IDE
 
-## Recommended Workflow
+If you'd like to extend your IDE with some tools that will help you with testing and improving your code, check the [TOOLS](./TOOLS.md) page. We go into multiple IDEs, editors and some useful extensions.
 
-Try this command while working on exercises:
+## Additional information
+
+### Adding pytest to your PATH
+
+**Note:** If you are running a [virtual environment](.\TOOLS.md) you do not need to *add to path* as it should work fine.
+
+Typing `python3 -m` every time you want to run a module can get a little annoying. You can add the `Scripts` folder of your Python installation to your path. If you do not know where you have installed Python, run the following command in your terminal:
 
 ```bash
-cd exercism/python/bob
-pytest -x --ff bob_test.py
+$ python3 -c "import os, sys; print(os.path.dirname(sys.executable))"
+{python_directory}
 ```
 
-## PDB
+The *returned* directory is where your Python version is installed, in this tutorial it is referred to as `{python_directory}`.
 
-Typing pdb on the command line will drop you into the python debugger when a test fails.
-To learn how to usepdb, check out the
-[documentation](https://docs.python.org/3/library/pdb.html#debugger-commands).
+#### Windows
 
-```bash
-pytest --pdb bob_test.py
+Click the `Windows Start` button and lookup *Edit the system environment variables* and press enter. Next press, `Environment Variables...`:
+
+![Press the blue button, lol](https://raw.githubusercontent.com/exercism/python/main/docs/img/Windows-SystemProperties.png)
+
+Then find the `Path` variable in your *User variables*, select it, and click `Edit...`:
+
+![Selecting the path variable](https://raw.githubusercontent.com/exercism/python/main/docs/img/Windows-EnvironmentVariables.png)
+
+Then add a new line, as shown in the picture, replacing `{python_directory}` with your Python installation's directory:
+
+![Add python to path](https://raw.githubusercontent.com/exercism/python/main/docs/img/Windows-AddPythonPath.png)
+
+### Fixing warnings
+
+It is possible that you will get `warnings` about "unknown markers" when running a test that uses our _new_ syntax.
+
+To resolve this issue, we use a `pytest.ini` file, which can be downloaded from the top level of the Python track directory: [pytest.ini](https://github.com/exercism/python/blob/main/pytest.ini).
+
+You can also create your own file with the following content:
+
+```ini
+[pytest]
+markers = 
+    task: A concept exercise task.
 ```
+
+Whenever you run your tests, make sure that this file is in your _root_ or _working_ directory.
+
+_More information on customizing pytest can be found in the [PyTest docs](https://docs.pytest.org/en/6.2.x/customize.html#pytest-ini)_