-
Notifications
You must be signed in to change notification settings - Fork 5
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Extend test guidelines with implementation notes for unit/regression …
…tests (#186) Signed-off-by: Martin Morgenstern <[email protected]>
- Loading branch information
Showing
3 changed files
with
156 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
# Developer documentation | ||
|
||
Welcome to the developer section of the contributor docs. Here you will find | ||
guidelines for the implementation of tooling such as the SCS conformance tests. |
File renamed without changes.
152 changes: 152 additions & 0 deletions
152
contributor-docs/development/tests/test-implementation-guide.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,152 @@ | ||
--- | ||
title: SCS Conformance Test Implementation Guide | ||
type: | ||
status: Draft | ||
track: Global | ||
--- | ||
|
||
SovereignCloudStack (SCS) uses [conformance tests][gh-scripts] to certify | ||
compliance of a given cloud offering with respect to a given [certificate | ||
scope][cert-scopes] such as *SCS Compatible IaaS v4*. | ||
Our aim is that these tests are reliable, consistent and comprehensible for | ||
the users. | ||
This document is a guideline for conformance test authors and summarizes the | ||
current best practices to achieve this goal. | ||
|
||
## 1. Mapping of RFC2119 Keywords | ||
|
||
Test authors working on new and existing conformance tests for an SCS standard | ||
must implement the keywords such as MUST and SHOULD according to the [SCS | ||
RFC2119 Keyword Test Guide][scs-rfc2119-guide]. | ||
|
||
## 2. Unit and Regression Tests | ||
|
||
Test authors are *strongly* encouraged to include unit and regression tests for | ||
the conformance test's logic. | ||
Conformance tests will inevitably contain some non-trivial algorithms, be it for | ||
parsing flavor names or evaluating CVE vulnerability reports. | ||
Testing them automatically and regularly against valid and invalid inputs helps | ||
test authors to find programming mistakes early on. | ||
As a bonus, a well written unit test also makes it easier for reviewers to cross | ||
check a given pull request with new, enhanced or refactored conformance tests. | ||
|
||
In general, unit and regression tests for the conformance tests are located in | ||
the same location as the conformance tests, that is, the `Tests/` directory of | ||
the [SCS standards repository][scs-standards]. | ||
Setup and development of the unit tests is described in the [SCS conformance | ||
tests README][tests-readme]. | ||
|
||
### Naming Conventions | ||
|
||
Any module that ends in `_test` will be picked up as a unit test module by | ||
pytest in our CI pipeline (in other words, all files matched by the pattern | ||
`Tests/**/*_test.py`). | ||
Pytest will execute any function prefixed with `test_` as a test function, or, | ||
alternatively, tests based on the `unittest` module from the Python standard | ||
library. | ||
|
||
Follow these naming rules: | ||
|
||
* The name of a unit test module is constructed from the module to be unit | ||
tested and the `_test` suffix. For example, unit tests for `flavor_names.py` | ||
should reside in `flavor_names_test.py`. | ||
* The name of a Python module in general should be a valid [Python | ||
identifier][python-identifiers], to allow imports via the `import` statement. | ||
For example, use `flavor_names.py` instead of `flavor-names.py`. | ||
|
||
### Write Testable Conformance Tests | ||
|
||
Software design fills complete book shelves and this section can only cover some | ||
basic best practices. | ||
As a general rule of thumb, the earlier you begin writing unit tests, the better | ||
– it will force you to write modules that are easy to test. | ||
|
||
Divide conformance tests scripts into smaller, loosely coupled units, i.e., | ||
functions that serve one purpose each. | ||
Use the [*dependency injection*][wiki-di] technique, i.e., pass externally | ||
retrieved data as function arguments instead of hardcoding the calls to the | ||
retrieval functions inside a function. | ||
Avoid logging calls deep down in the call hierarchy and use explicit return | ||
values or raise exceptions, instead. | ||
Here is an abstract example of a conformance test which does exactly that: | ||
|
||
```python | ||
# property_compliance.py | ||
|
||
def retrieve(location): | ||
"""Retrieve raw data via network.""" | ||
... | ||
|
||
def parse(data): | ||
"""Parse raw data and return a dict.""" | ||
... | ||
|
||
def handle(...): | ||
"""Evaluate parsed data.""" | ||
... | ||
|
||
def main(args): | ||
... | ||
data = retrieve(location) | ||
parsed = parse(data) | ||
result = handle(parsed) | ||
... | ||
if result.some_prop != expected_value: | ||
logging.warn("some_prop is not as expected: %s (vs. %s)", result.some_prop, expected_value) | ||
... | ||
return result.success | ||
|
||
if __name__ == "__main__": | ||
# using sys.exit(…) only here makes it possible to unit test main(…) | ||
sys.exit(main(sys.argv)) | ||
``` | ||
|
||
Adhering to this style makes it easier to test an algorithm in isolation, | ||
without actually making a call to some external service. | ||
|
||
### Pytest Test Example | ||
|
||
Assuming we want to unit test some members of the module `property_compliance` | ||
from the previous section, we would create a file `property_compliance_test.py` | ||
with the following content as a starting point: | ||
|
||
```python | ||
"""Unit tests for property_compliance. | ||
(c) Your Name <[email protected]>, 4/2024 | ||
SPDX-License-Identifier: CC-BY-SA-4.0 | ||
""" | ||
|
||
import pytest | ||
|
||
from property_compliance import parse | ||
|
||
|
||
def test_success(): | ||
assert parse("some valid input") == "expected output" | ||
|
||
|
||
def test_failure(): | ||
with pytest.raises(ValueError): | ||
parse("invalid input") | ||
``` | ||
|
||
## 3. Conformance Tests Shouldn't Require Admin Privileges | ||
|
||
Conformance tests are expected to be executable without admin privileges (see §2 of | ||
[Regulations for achieving SCS-compatible certification][scs-0004-v1]). | ||
In particular, this means: | ||
|
||
* The cloud credentials (e.g., kubeconfig and OpenStack `clouds.yaml`) passed to | ||
the scripts are non-admin credentials. | ||
* Conformance tests scripts should not require root privileges, except for the | ||
installation of operating system prerequisites (e.g., a Python interpreter). | ||
|
||
[scs-standards]: https://github.com/SovereignCloudStack/standards/ | ||
[tests-readme]: https://github.com/SovereignCloudStack/standards/blob/main/Tests/README.md | ||
[python-identifiers]: https://docs.python.org/3/reference/lexical_analysis.html#identifiers | ||
[gh-scripts]: https://github.com/SovereignCloudStack/standards/tree/main/Tests | ||
[cert-scopes]: https://docs.scs.community/standards/certification/scopes-versions | ||
[scs-rfc2119-guide]: https://docs.scs.community/contributor-docs/operations/tests/rfc2119-keyword-test-guide | ||
[scs-0004-v1]: https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0004-v1-achieving-certification.md | ||
[wiki-di]: https://en.wikipedia.org/wiki/Dependency_injection |