diff --git a/.github/actions/build-technical-doc/action.yml b/.github/actions/build-technical-doc/action.yml index 7953a920d..55c91b1b4 100644 --- a/.github/actions/build-technical-doc/action.yml +++ b/.github/actions/build-technical-doc/action.yml @@ -17,6 +17,6 @@ runs: - name: Generate technical documentation run: | - poetry run asciidoxy --base-dir doc --build-dir ../dist/doc/ --multipage doc/index.adoc + poetry run asciidoxy --warnings-are-errors --base-dir doc --image-dir images --build-dir ../dist/doc/ --multipage doc/index.adoc working-directory: ./docs shell: bash diff --git a/docs/doc/dev/background/workflow.adoc b/docs/doc/dev/background/workflow.adoc index ed84386bb..7896c190e 100644 --- a/docs/doc/dev/background/workflow.adoc +++ b/docs/doc/dev/background/workflow.adoc @@ -2,7 +2,7 @@ == Git -The project is using the https://git-flow.readthedocs.io/fr/latest/presentation.html[gitflow]. +The project is using the https://git-flow.readthedocs.io/en/latest/presentation.html[gitflow]. The feature branches are named following the pattern "feat-/" For example : "feat-rspy31/init-tech-doc" diff --git a/docs/doc/dev/design/services/design.adoc b/docs/doc/dev/design/services/design.adoc index 549608143..72be59ced 100644 --- a/docs/doc/dev/design/services/design.adoc +++ b/docs/doc/dev/design/services/design.adoc @@ -33,8 +33,8 @@ It is handled as a python project as if it is a rs-server service. == Services -:leveloffset: +3 +:leveloffset: +2 include::cadip.adoc[] -:leveloffset: -3 +:leveloffset: -2 diff --git a/docs/doc/dev/doc-generation/how-to.adoc b/docs/doc/dev/doc-generation/how-to.adoc index c5f3c8ee6..01a503623 100644 --- a/docs/doc/dev/doc-generation/how-to.adoc +++ b/docs/doc/dev/doc-generation/how-to.adoc @@ -1,28 +1,29 @@ = Generate rs-server technical documentation +:sectnums: This procedure is run automatically in the CI by the 'generate documentation' workflow. It can be executed locally to verify the generated documentation (before publishing it for example). -See ${include("description.adoc")} for more details on the process and technical stuff. +See ${cross_document_ref("description.adoc")} for more details on the process and technical stuff. -. Pre-requises +== Prerequisites The local developer environment has been setup. The global python project is setup. The python sub-project for doc generation is also setup. -. Generate the technical documentation +== Generate the technical documentation From the rs-server/docs folder, execute the command: [source, bash, indent=0] ---- -poetry run asciidoxy --base-dir doc --build-dir ../dist/doc/ --multipage doc/index.adoc +poetry run asciidoxy --warnings-are-errors --base-dir doc --image-dir images --build-dir ../dist/doc/ --multipage doc/index.adoc ---- This command generates the html pages of all the technical documentation and write them in the rs-server/dist/doc folder. -. Generate the python api documentation +== Generate the python api documentation From the root folder, run the following command : [source, bash, indent=0] @@ -43,7 +44,7 @@ poetry run sphinx-build -M html docs/doc/api/python/ dist/doc/output/api/python/ It generates the html pages for the python api of the rs-server. The generated api is accessible locally at 'dist/doc/api/html/index.html'. -. Generate the REST API documentation +== Generate the REST API documentation From the root folder, run the following command : [source, bash, indent=0] @@ -69,7 +70,7 @@ The output folder is the one expected for integration with the main document of NOTE: This is a POC waiting for the frontend to be implemented. The documented REST API will be the one of the frontend at the end. -. Verify the generated documentation +== Verify the generated documentation The technical documentation is generated in the folder "dist/doc/output/". The main page is the file "index.html" is this folder. diff --git a/docs/doc/dev/environment/description.adoc b/docs/doc/dev/environment/description.adoc index c03c38d49..7c1ce6ff0 100644 --- a/docs/doc/dev/environment/description.adoc +++ b/docs/doc/dev/environment/description.adoc @@ -46,7 +46,7 @@ and explains briefly the choices that have been made. |commonly used by the team |security check -|trivi +|trivy |used in the previous phase |technical documentation diff --git a/docs/doc/dev/environment/installation.adoc b/docs/doc/dev/environment/installation.adoc index b054e6fad..86f783fca 100644 --- a/docs/doc/dev/environment/installation.adoc +++ b/docs/doc/dev/environment/installation.adoc @@ -1,10 +1,10 @@ = Install your environment -== Pre requises +== Prerequisites This tutorial assumes the developer has already basic knowledge and is using a ubuntu working station. -This tutorial lets the user uses its preferred IDE. +This tutorial lets the user uses their preferred IDE. You could extend your environment with specific IDE integration. == Checkout the project @@ -72,7 +72,7 @@ You should first initialize the project locally cd $RSPY_ROOT/src/rs-server poetry install --with dev ---- -This command initialize and activate a virtual environment for your project. +This command initializes and activates a virtual environment for your project. It installs in this environment the project dependencies and the develop dependencies (--with dev). @@ -93,15 +93,15 @@ pytest tests -m "unit" NOTE: Your IDE can provide integration to run your unittests. -== Install trivi +== Install trivy -link:https://aquasecurity.github.io/trivy/v0.47/[trivy] is used to do some security and license checks on the repository, the generated wheel packages and the docker images. +link:https://aquasecurity.github.io/trivy/latest/[trivy] is used to perform security and license checks on the repository, the generated wheel packages and the docker images. -trivi is run on the pre-commit and the CI +trivy is run on the pre-commit and the CI to verify the repository compliance with common vulnerabilities, secrets and licenses. To install trivy on your local environment, -follow link:https://aquasecurity.github.io/trivy/v0.47/getting-started/installation/[the official procedure]. +follow link:https://aquasecurity.github.io/trivy/latest/getting-started/installation/[the official procedure]. == Install pre-commit hooks @@ -115,7 +115,8 @@ poetry run pre-commit install ---- Once installed, checks will be performed before each commit -to ensure your code is compliant with project best practice : +to ensure your code is compliant with project best practices : + * assert no unresolved merge conflicts * fix end of file * fix trailing whitespace @@ -137,6 +138,6 @@ to keep your code compliant with the coding style at any moment. == The next steps -The ${include("description.adoc")} can provide you with more details about the environment. -The ${include("../background/workflow.adoc")} describes the workflow followed by developers to implement stories. -You may also want to read the ${include("../code-style.adoc")}. +The ${cross_document_ref("description.adoc")} can provide you with more details about the environment. +The ${cross_document_ref("../background/workflow.adoc")} describes the workflow followed by developers to implement stories. +You may also want to read the ${cross_document_ref("../code-style.adoc")}. diff --git a/docs/doc/index.adoc b/docs/doc/index.adoc index aa5dea042..e125bd4ca 100644 --- a/docs/doc/index.adoc +++ b/docs/doc/index.adoc @@ -2,7 +2,10 @@ == User manual -The main functionalities are explained here : +The overall architecture is described here: +${include("users/architecture.adoc")} + +The main functionalities are explained here: ${include("users/functionalities.adoc")} @@ -10,15 +13,15 @@ ${include("users/functionalities.adoc")} All you want to know when you are a rs-server developer. -The documentation is organized in 4 sections : +The documentation is organized in 4 sections: -* Tutorials to help you start coding on rs-server : +* Tutorials to help you start coding on rs-server: ** ${include("dev/environment/installation.adoc")} * How to guides to provide practical common procedures ** ${include("dev/doc-generation/how-to.adoc")} -* Reference guides give you technical information about the code and software API : +* Reference guides give you technical information about the code and software API: ** ${include("dev/code-style.adoc")} ** ${include("api/python/html/index.adoc")} ** ${include("api/rest/index.adoc")} diff --git a/docs/doc/users/architecture.adoc b/docs/doc/users/architecture.adoc new file mode 100644 index 000000000..c9ede9176 --- /dev/null +++ b/docs/doc/users/architecture.adoc @@ -0,0 +1,37 @@ += Architecture + +== Static View + +We can split the system into following Components: + +* *RS-Server*: The RS-Server controls user access to all sensitive interfaces: Catalog, LTA, ADGS, PRIP, CADIP. As a consequence, we can distinguish following components: + +** *RS-Server Frontend* +** *RS-Server Backend / Catalog* +** *RS-Server Backend / PRIP* +** *RS-Server Backend / CADIP* +** *RS-Server Backend / LTA* +** *RS-Server Backend / AUXIP* + +* *RS-Server libraries*: this is a set of python functions that provide processing task and flows. Flows and tasks can be orchestrated by Prefect server or executed from any Python environment. On RS-Server side there will be also function to compute performance indicator. + +* *RS-Virtual environment*: Component that gives users access to computational environments and resources for executing processing chains. + +* *Processing*: this group hosts all components to process Sentinel products. The CFI are not part of the group. Only wrappers, preparation worker and Dask cluster to provide processing power. Here are the components: + +** *Dask cluster* +** *DPR libraries* + +* *Infrastructure*: The infrastructure is a portable, extensible and open-source platform orchestrated by Kubernetes. We can isolate two specific layout which are monitoring and security. As a consequence, we can distinguish following components: + +** *Infrastructure core* +** *Infrastructure monitoring* +** *Infrastructure security* + +image::staticview.png[] + +== Dynamic View + +The following schema highlights main interactions between the components. + +image::dynamicview.png[] diff --git a/docs/doc/users/functionalities.adoc b/docs/doc/users/functionalities.adoc index abfab21ae..8a440a787 100644 --- a/docs/doc/users/functionalities.adoc +++ b/docs/doc/users/functionalities.adoc @@ -7,7 +7,7 @@ The CADU Interface delivery Point (CADIP) is a pick-up point for Sentinel CADU d clients to straightforwardly discover and retrieve available data files through a standard OData RESTful API. The following endpoints have been implemented in RS-Server to interact with CADIP RESTful API [[cadip-search]] -== [.green]#Search Endpoint# +=== [.green]#Search Endpoint# This endpoint retrieves a list of products from the CADU system for a specified station within a given time range and return a STAC compatible FeatureCollection response. ==== API Reference @@ -53,7 +53,7 @@ GET /cadip/station123/cadu/search?datetime=2023-01-01T00:00:00Z/2023-01-02T23:59 } [[cadip-download]] -== [.green]#Download Endpoint# +=== [.green]#Download Endpoint# This endpoint initiates an asynchronous download process for a CADU product using EODAG. If specific parameters are provided, endpoint also upload the file to an S3 bucket. @@ -93,7 +93,7 @@ GET /cadip/station123/cadu?name=DCS_04_S1A_20231121072204051312_ch1_DSDB_00001.r ---- [[cadip-status]] -== [.green]#Status Endpoint# +=== [.green]#Status Endpoint# This endpoint is used to query the download status of an CADU file. ==== API Reference @@ -127,7 +127,7 @@ GET /cadip/{station}/cadu/status?name=DCS_04_S1A_20231121072204051312_ch1_DSDB_0 The Auxiliary Data Gathering Service (ADGS) is a pick-up point for Sentinel auxiliary files. This service allows clients to discover and retrieve available auxiliary data files through a standard OData RESTful API. The following endpoints have been implemented in RS-Server to interact with ADGS RESTful API. [[adgs-search]] -== [.green]#Search Endpoint# +=== [.green]#Search Endpoint# This endpoint handles the search for products in the AUX station within a specified time interval and return a STAC compatible FeatureCollection response. @@ -169,7 +169,7 @@ GET /adgs/aux/search?datetime=2018-01-01T00:00:00Z/2023-01-02T23:59:59Z&limit=10 } [[adgs-download]] -== [.green]#Download Endpoint# +=== [.green]#Download Endpoint# This endpoint initiates an asynchronous download process for an AUX product using EODAG. If specific parameters are provided, endpoint also upload the file to an S3 bucket. @@ -208,7 +208,7 @@ GET /adgs/aux?name=S2__OPER_AUX_ECMWFD_PDMC_20190216T120000_V20190217T090000_201 ---- [[adgs-status]] -== [.green]#Status Endpoint# +=== [.green]#Status Endpoint# This endpoint is used to query the download status of an AUX file. ==== Endpoint diff --git a/docs/doc/users/installation.adoc b/docs/doc/users/installation.adoc index 2c50d31ae..7a83d045e 100644 --- a/docs/doc/users/installation.adoc +++ b/docs/doc/users/installation.adoc @@ -2,7 +2,7 @@ NOTE: The rs-server installation will be described here -// TODO pre requises to install rs-server +// TODO prerequisites to install rs-server // TODO the install procedure // TODO what is expected as a result // TODO how to check it is nominally installed diff --git a/docs/images/dynamicview.png b/docs/images/dynamicview.png new file mode 100644 index 000000000..6f2223535 Binary files /dev/null and b/docs/images/dynamicview.png differ diff --git a/docs/images/staticview.png b/docs/images/staticview.png new file mode 100644 index 000000000..c6fb12d31 Binary files /dev/null and b/docs/images/staticview.png differ