Feature: WP4 General API compliance validation #45
Conversation
|
A personal note for reviewers considerations, this code is not cleaned up. It does work and I did my best to integrate it with the existing infrastructure, but keep in mind that big refactors or cleanups might take some additional time. For now I suggest focusing on the functional aspects and any missing endpoints/scenarios and if time allows I will look into cleaning up in the future. |
| @@ -0,0 +1,532 @@ | |||
| from typing import Union | |||
There was a problem hiding this comment.
I guess this module and maybe the whole of general are utilities, without actual tests for the test suite.
This should be moved under src/openeo_test_suite/lib to have better separation between reusable components and actual tests
There was a problem hiding this comment.
I've moved the compliance_util.py, but I'm unsure what you mean when you say the whole of general. test_compliance.py does contain tests, just to make sure, you don't want those moved too, correct?
| @@ -0,0 +1,340 @@ | |||
| { | |||
There was a problem hiding this comment.
This file is invalid, e.g. schemas, examples etc. Make sure only properties are provided when needed.
| @@ -0,0 +1,103 @@ | |||
| { | |||
There was a problem hiding this comment.
Does the test suite somehow makes sure that the processes used here are actually available?
There was a problem hiding this comment.
Not currently, how would you suggest that we separate testing for the existence of processes and this specific process. The included process includes "subtract", "multiply", "sum" and "divide". I was hoping we can just assume the existence of those very basic processes.
If we cannot assume the existence of a single process then I am confused how I am supposed to construct a valid example for the validation endpoint at all.
There was a problem hiding this comment.
I'd just do the same as in process testing: skip if not available with a corresponding message.
...peneo_test_suite/tests/general/payload_examples/put_process_graphs/put_process_graphs_1.json
Outdated
Show resolved
Hide resolved
src/openeo_test_suite/tests/general/payload_examples/patch_jobs/patch_jobs_1.json
Outdated
Show resolved
Hide resolved
| @@ -0,0 +1,41 @@ | |||
| { | |||
There was a problem hiding this comment.
How does this ensure the data and file formats are available?
There was a problem hiding this comment.
It does not, is there some base dataset/file format that we can expect to always be here?
Is there a way we can include a baseline/test process that can be guaranteed to never fail?
There was a problem hiding this comment.
I'd just do the same as in process testing: skip if not available with a corresponding message.
|
Conducted a very high-level review, didn't look into details yet. But it should already help to avoid some potential issues. For a more detailed review I'm waiting until the changes from #44 are not part of this PR changelist any longer. |
|
quick note: can you move the submodule to https://github.com/Open-EO/openeo-test-suite/tree/main/assets (which already has the openeo-processes submodule)? |
Yes of course! |
@soxofaan |
…on and test_endpoint function, replace getting filenames with glob, fix typehints, introduce validate_uri function, more robust get_spec_path and get_examples_path functions
NOTE: because I kept working on my own fork of the openeo-test-suite repo, this also includes the original changes from WP3, these can be ignored for the purpose of review (and should probably be merged seperately and not rely on this PR alone).
Add: general API compliance tests to the openeo-test-suite as requested in #20
The tests cover a range of different scenarios and endpoints and the openapi-core library is used to check the validity of responses from the API against the openapi specification. (The one included is the newest at version 1.2.0).
The files included are basic payloads, api spec (openapi.json), a utility python file and the test file containing the full range of tests.
Also note that actual requests are sent to the API, I have put skip marks on any tests that need a process graph to run through so that runtimes are not too long. Any created batch jobs, UDPs or other objects are cleaned up after every test.
Usage:
pytest src/openeo_test_suite/tests/general --html=reports/general-compliance.html --openeo-backend-url=openeo-dev.eodc.eu