Adding schema in order to support stage customization

.github/workflows/linting.yml

@@ -9,7 +9,7 @@ on:
 jobs:
   linting:
     strategy:
-      fail-fast: false
+      fail-fast: true
       matrix:
         python: ["3.8", "3.9", "3.10", "3.11"]
     runs-on: ubuntu-latest
@@ -29,11 +29,15 @@ jobs:
       - uses: isort/isort-action@master
         with:
           configuration: "--check-only --diff"
+      - name: Analysing the code style
+        run: |
+          tox -e codestyle
+      - name: Analysing the docstyle
+        run: |
+          tox -e docstyle
       - name: Analysing the code with mypy
-        continue-on-error: true
         run: |
           tox -e types
       - name: Analysing the code with pylint/flake
-        continue-on-error: true
         run: |
           tox -e errors

README.md

@@ -22,7 +22,8 @@ pip install pytest-fluent
 
 ## Usage
 
-pytest-fluent-logging forwards meta data from pytest to Fluentd for further processing. The meta data are 
+pytest-fluent-logging forwards meta data from pytest to Fluentd for further processing. The meta data are
+
 * unique session ID
 * unique test ID
 * status of the session respectively test case
@@ -31,7 +32,7 @@ pytest-fluent-logging forwards meta data from pytest to Fluentd for further proc
 * `record_property` entries
 * custom testcase information
 * custom session information
-  
+
 Furthermore, the Python logging instance can be extended in order to forward test case runtime logging.
 
 ```python
@@ -43,7 +44,7 @@ def test_my_runtime_log():
     assert value == 1
 ```
 
-or 
+or
 
 ```python
 from logging import getLogger
@@ -104,22 +105,22 @@ def provide_more_test_information() -> dict:
 
 The pytest CLI can be called with the following arguments in order to configure fluent-logging.
 
-| argument            | description                                                                        | default  |
-|---------------------|------------------------------------------------------------------------------------|----------|
-| --session-uuid      | Use a custom externally created UUID, e.g. link a CI job with the pytest session.  |          |
-| --fluentd-host      | Fluentd host address. If not provided, a local Fluentd instance will be called.    |          |
-| --fluentd-port      | Fluent host port                                                                   | 24224    |
-| --fluentd-tag       | Set a custom Fluentd tag                                                           | 'test'   |
-| --fluentd-label     | Set a custom Fluentd label                                                         | 'pytest' |
-| --fluentd-timestamp | Specify a Fluentd timestamp                                                        | None     |
-| --extend-logging    | Extend the Python logging with a Fluent handler                                    | False    |
-| --add-docstrings    | Add test docstrings to testcase call messages                                      |          |
+| argument            | description                                                                       | default  |
+| ------------------- | --------------------------------------------------------------------------------- | -------- |
+| --session-uuid      | Use a custom externally created UUID, e.g. link a CI job with the pytest session. |          |
+| --fluentd-host      | Fluentd host address. If not provided, a local Fluentd instance will be called.   |          |
+| --fluentd-port      | Fluent host port                                                                  | 24224    |
+| --fluentd-tag       | Set a custom Fluentd tag                                                          | 'test'   |
+| --fluentd-label     | Set a custom Fluentd label                                                        | 'pytest' |
+| --fluentd-timestamp | Specify a Fluentd timestamp                                                       | None     |
+| --extend-logging    | Extend the Python logging with a Fluent handler                                   | False    |
+| --add-docstrings    | Add test docstrings to testcase call messages                                     |          |
 
-### Ini Configuration Support 
+### Ini Configuration Support
 
 Default values of the CLI arguments for a project could also be defined in one of the following ini configuration files:
 
-1. pytest.ini: Arguments are defined under pytest section in the file. This file takes precedence over all other configuration files even if empty. 
+1. pytest.ini: Arguments are defined under pytest section in the file. This file takes precedence over all other configuration files even if empty.
 
 ```python
 [pytest]
@@ -224,7 +225,7 @@ Timestamps are added to the information if the ``--fluentd-timestamp`` option is
 addopts= --session-uuid="ac2f7600-a079-46cf-a7e0-6408b166364c" --fluentd-port=24224  --fluentd-host=localhost --fluentd-tag='dummytest' --fluentd-label='pytest' --fluentd-timestamp='@timestamp' --extend-logging
 ```
 
-The timestamp is added to each message. The value is in ISO 8601 format. A sample 
+The timestamp is added to each message. The value is in ISO 8601 format. A sample
 of the data collection from `test_addoptions.py` (as above) would look as below:
 
 ```json
@@ -253,4 +254,3 @@ The changelog.
 ## Contributing
 
 We welcome any contributions, enhancements, and bug-fixes. Open an [issue](https://github.com/Rohde-Schwarz/pytest-fluent/issues) on [Github](https://github.com) and [submit a pull request](https://github.com/Rohde-Schwarz/pytest-fluent/pulls).
-

docs/conf.py

@@ -19,4 +19,4 @@
 ]
 suppress_warnings = ["myst.header"]
 html_theme = "sphinx_rtd_theme"
-html_static_path = ["images"]
+# html_static_path = ["image"]

docs/usage.md

@@ -4,3 +4,6 @@
 :start-after: "## Usage"
 :end-before: "## Changelog"
 ```
+
+```{include} ./usage/stage_settings.md
+```

docs/usage/stage_settings.md

@@ -0,0 +1,239 @@
+### Custom stage settings
+
+Sometimes, the default settings are not enough in order to forward the test information as needed. Thus, you can set custom stage settings
+in order to fit your needs.
+
+You can set specific values for `all` stages or specific values for any used stage. In order to do so, call your test run with the `--stage-settings=YourFileName.json` parameter. The following example stage settings JSON file content
+
+```json
+{
+    "all": {
+        "tag": "run",
+        "label": "pytest",
+        "replace": {"keys": {"status": "state", "sessionId": "id"}},
+    },
+    "pytest_sessionstart": {
+        "tag": "run",
+        "label": "test",
+        "add": {"start_info": "Pytest started"},
+    },
+    "pytest_sessionfinish": {
+        "tag": "result",
+        "label": "test",
+        "add": {"stop_info": "Pytest finished"},
+    },
+    "pytest_runtest_logstart": {
+        "tag": "run",
+        "label": "testcase",
+        "add": {"start_info": "Testcase started"},
+    },
+    "pytest_runtest_logreport": {
+        "tag": "result",
+        "label": "testcase",
+        "replace": {
+            "values": {"passed": "pass", "failed": "fail"},
+        },
+        "add": {"stop_info": "Testcase finished"},
+    },
+    "logging": {
+        "replace": {"keys": {"message": "msg", "sessionId": "id"}},
+    },
+}
+```
+
+will result in the following output
+
+```json
+[
+  {
+    "stage": "session",
+    "tag": "test",
+    "label": "pytest",
+    "state": "start",
+    "id": "3d82b514-60e2-4580-96ab-3daf5a5446c8"
+  },
+  {
+    "stage": "testcase",
+    "testId": "6b5092ad-c905-4879-a70c-cb5b2a7df90d",
+    "name": "test_data_reporter_with_patched_values.py::test_base",
+    "tag": "test",
+    "label": "pytest",
+    "state": "start",
+    "id": "3d82b514-60e2-4580-96ab-3daf5a5446c8"
+  },
+  {
+    "type": "logging",
+    "host": "hostname",
+    "where": "test_data_reporter_with_patched_values.test_base",
+    "level": "INFO",
+    "stack_trace": "None",
+    "message": "Test running",
+    "testId": "6b5092ad-c905-4879-a70c-cb5b2a7df90d",
+    "stage": "testcase",
+    "id": "3d82b514-60e2-4580-96ab-3daf5a5446c8"
+  },
+  {
+    "name": "test_data_reporter_with_patched_values.py::test_base",
+    "outcome": "pass",
+    "duration": 0.0034263000000001043,
+    "markers": {
+      "test_base": 1,
+      "test_data_reporter_with_patched_values.py": 1,
+      "test_data_reporter_with_patched_values0": 1
+    },
+    "stage": "testcase",
+    "when": "call",
+    "testId": "6b5092ad-c905-4879-a70c-cb5b2a7df90d",
+    "tag": "test",
+    "label": "pytest",
+    "id": "3d82b514-60e2-4580-96ab-3daf5a5446c8",
+    "stop_info": "Testcase finished"
+  },
+  {
+    "stage": "testcase",
+    "testId": "6b5092ad-c905-4879-a70c-cb5b2a7df90d",
+    "name": "test_data_reporter_with_patched_values.py::test_base",
+    "tag": "test",
+    "label": "pytest",
+    "state": "finish",
+    "id": "3d82b514-60e2-4580-96ab-3daf5a5446c8"
+  },
+  {
+    "duration": 1.3674933910369873,
+    "stage": "session",
+    "tag": "test",
+    "label": "pytest",
+    "state": "finish",
+    "id": "3d82b514-60e2-4580-96ab-3daf5a5446c8"
+  }
+]
+```
+
+for this test case
+
+```python
+import logging
+
+def test_base():
+    logger = logging.getLogger()
+    logger.info("Test running")
+    assert True
+```
+
+#### Stage setting file
+
+Custom settings for each supported stage can be easily setup. You have to create a file with
+a `.json` or `.yaml` extension and call pytest with this additional parameter `--stage-settings`.
+
+The file will be validated against a schema of supported values and in case of an error, a `jsonschema.ValidationError`
+will be thrown.
+
+#### Stage settings
+
+##### Number of supported stage
+
+The following stages can be modified.
+
+* `pytest_sessionstart`
+* `pytest_runtest_logstart`
+* `pytest_runtest_logreport`
+* `pytest_runtest_logfinish`
+* `pytest_sessionfinish`
+* `logging`
+
+These values are the keys for the dictionary object. Additionally, you can set also
+a `all` key for convenience reasons to patch all keys at once.
+
+#### Patch events
+
+Probably, your stage setting would look like
+
+```json
+{
+    "pytest_sessionstart": {
+        "tag": "run",
+        "label": "pytest",
+        "replace": {
+            "keys": {
+                "status": "state",
+                "sessionId": "id"
+            },
+            "values": {
+                "passed": "pass"
+            }
+        },
+        "add": {
+          "start_info": "Pytest started"
+        },
+    }
+}
+```
+
+The following values are supported
+
+| Key name  | action                                                                                 | type   |
+| --------- | -------------------------------------------------------------------------------------- | ------ |
+| `tag`     | Set a specifc Fluent tag for this stage                                                | `str`  |
+| `label`   | Set a specifc Fluent label for this stage                                              | `str`  |
+| `replace` | Replace key values from a dictionary and also replace some preset pytest result values | `dict` |
+| `add`     | Add new values to the result dictionary                                                | `dict` |
+| `drop`    | Drop specific values from the result dictionary                                        | `dict` |
+
+##### Replace dictionary
+
+The `replace` patching action has two keys `keys` and `values` in order to replace either a key value or a result value.
+See the following default values in order to get an idea about the content.
+
+At the moment, the following values can be changed
+
+* `passed`
+* `failed`
+* `skipped`
+* `error`
+* `start`
+* `finish`
+* `session`
+* `testcase`
+
+##### Use values from ARGV and ENV
+
+If you want to use data provided by the command line arguments or directly from environment variables,
+use the following syntax for value strings.
+
+| Type | Syntax                         |
+| ---- | ------------------------------ |
+| ARGV | `"<fluentd-tag>"`              |
+| ENV  | `"${USE_ENV}"` or `"$USE_ENV"` |
+
+Here is a simple example using both variants
+
+```json
+{
+    "pytest_sessionstart": {
+        "tag": "run",
+        "label": "pytest",
+        "replace": {
+            "keys": {
+                "tag": "<fluentd-tag>",
+                "sessionId": "${ID}"
+            },
+            "values": {
+                "passed": "$OUTCOME_PASSED"
+            }
+        }
+    }
+}
+```
+
+The data will be mapped after starting the pytest session.
+
+#### Default values
+
+| stage                      | value                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
+| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `pytest_sessionstart`      | <pre lang="json">{<br>    "status": "start",<br>    "stage": "session",<br>    "sessionId": "8d0d165d-5581-478c-ba0f-f7ec7d5bcbcf",<br>    "tag": "test",<br>    "label": "pytest"<br>}</pre>                                                                                                                                                                                                                                                                                                                                                                           |
+| `pytest_runtest_logstart`  | <pre lang="json">{<br>    "status": "start",<br>    "stage": "testcase",<br>    "sessionId": "8d0d165d-5581-478c-ba0f-f7ec7d5bcbcf",<br>    "testId": "9f0363fa-ef99-49c7-8a2d-6261e90acb00",<br>    "name": "test_data_reporter_with_patched_values.py::test_base",<br>    "tag": "test",<br>    "label": "pytest"<br>}</pre>                                                                                                                                                                                                                                          |
+| `pytest_runtest_logreport` | <pre lang="json">{<br>    "name": "test_data_reporter_with_patched_values.py::test_base",<br>    "outcome": "passed",<br>    "duration": 0.0035069000000005346,<br>    "markers": {<br>      "test_base": 1,<br>      "test_data_reporter_with_patched_values.py": 1,<br>      "test_data_reporter_with_patched_values0": 1<br>    },<br>    "stage": "testcase",<br>    "when": "call",<br>    "sessionId": "8d0d165d-5581-478c-ba0f-f7ec7d5bcbcf",<br>    "testId": "9f0363fa-ef99-49c7-8a2d-6261e90acb00",<br>    "tag": "test",<br>    "label": "pytest"<br>}</pre> |
+| `pytest_runtest_logfinish` | <pre lang="json">{<br>    "status": "finish",<br>    "stage": "testcase",<br>    "sessionId": "8d0d165d-5581-478c-ba0f-f7ec7d5bcbcf",<br>    "testId": "9f0363fa-ef99-49c7-8a2d-6261e90acb00",<br>    "name": "test_data_reporter_with_patched_values.py::test_base",<br>    "tag": "test",<br>    "label": "pytest"<br>}</pre>                                                                                                                                                                                                                                         |
+| `pytest_sessionfinish`     | <pre lang="json">{<br>    "status": "finish",<br>    "duration": 1.5651893615722656,<br>    "stage": "session",<br>    "sessionId": "8d0d165d-5581-478c-ba0f-f7ec7d5bcbcf",<br>    "tag": "test",<br>    "label": "pytest"<br>}</pre>                                                                                                                                                                                                                                                                                                                                   |
+| `logging`                  | <pre lang="json">{<br>    "type": "logging",<br>    "host": "hostname",<br>    "where": "test_data_reporter_with_patched_values.test_base",<br>    "level": "INFO",<br>    "stack_trace": "None",<br>    "message": "Test running",<br>    "sessionId": "8d0d165d-5581-478c-ba0f-f7ec7d5bcbcf",<br>    "testId": "9f0363fa-ef99-49c7-8a2d-6261e90acb00",<br>    "stage": "testcase"<br>}</pre>                                                                                                                                                                          |

pyproject.toml

@@ -32,7 +32,9 @@ dependencies = [
     "pytest>=7.0.0",
     "msgpack",
     "six",
-    "fluent-logger"
+    "fluent-logger",
+    "jsonschema",
+    "ruamel.yaml",
 ]
 dynamic = ["version"]
 

setup.cfg

@@ -1,6 +1,6 @@
 [metadata]
 copyright = Copyright © Rohde & Schwarz GmbH & Co. KG 2022
-platform = 
+platform =
     Unix
     Linux
     Windows
@@ -9,10 +9,15 @@ platform =
 package_dir=
     =src
 packages=find:
+include_package_data = True
 
 [options.entry_points]
 pytest11 =
     fluent-logging = pytest_fluent.plugin
 
 [options.packages.find]
-where = src
+where = src
+
+[options.package_data]
+pytest_fluent.data =
+    *.json