diff --git a/docs/source/development-guide/doc-overview.rst b/docs/source/development/doc-overview.rst similarity index 100% rename from docs/source/development-guide/doc-overview.rst rename to docs/source/development/doc-overview.rst diff --git a/docs/source/development-guide/docker.rst b/docs/source/development/docker.rst similarity index 100% rename from docs/source/development-guide/docker.rst rename to docs/source/development/docker.rst diff --git a/docs/source/development-guide/index.rst b/docs/source/development/index.rst similarity index 95% rename from docs/source/development-guide/index.rst rename to docs/source/development/index.rst index 9a085c76b..273d93ccc 100644 --- a/docs/source/development-guide/index.rst +++ b/docs/source/development/index.rst @@ -38,4 +38,5 @@ A typical development workflow might look like the following: doc-overview docker release-workflow - style-guide/style-guide \ No newline at end of file + style-guide/style-guide + technology-stack \ No newline at end of file diff --git a/docs/source/development-guide/release-workflow.rst b/docs/source/development/release-workflow.rst similarity index 100% rename from docs/source/development-guide/release-workflow.rst rename to docs/source/development/release-workflow.rst diff --git a/docs/source/development-guide/style-guide/checklist-for-pull-requests.rst b/docs/source/development/style-guide/checklist-for-pull-requests.rst similarity index 100% rename from docs/source/development-guide/style-guide/checklist-for-pull-requests.rst rename to docs/source/development/style-guide/checklist-for-pull-requests.rst diff --git a/docs/source/development-guide/style-guide/git-and-github-workflow.rst b/docs/source/development/style-guide/git-and-github-workflow.rst similarity index 100% rename from docs/source/development-guide/style-guide/git-and-github-workflow.rst rename to docs/source/development/style-guide/git-and-github-workflow.rst diff --git a/docs/source/development-guide/style-guide/naming-conventions.rst b/docs/source/development/style-guide/naming-conventions.rst similarity index 100% rename from docs/source/development-guide/style-guide/naming-conventions.rst rename to docs/source/development/style-guide/naming-conventions.rst diff --git a/docs/source/development-guide/style-guide/poetry-environment.rst b/docs/source/development/style-guide/poetry-environment.rst similarity index 100% rename from docs/source/development-guide/style-guide/poetry-environment.rst rename to docs/source/development/style-guide/poetry-environment.rst diff --git a/docs/source/development-guide/style-guide/python-coding.rst b/docs/source/development/style-guide/python-coding.rst similarity index 100% rename from docs/source/development-guide/style-guide/python-coding.rst rename to docs/source/development/style-guide/python-coding.rst diff --git a/docs/source/development-guide/style-guide/python-docstrings.rst b/docs/source/development/style-guide/python-docstrings.rst similarity index 100% rename from docs/source/development-guide/style-guide/python-docstrings.rst rename to docs/source/development/style-guide/python-docstrings.rst diff --git a/docs/source/development-guide/style-guide/review-standards.rst b/docs/source/development/style-guide/review-standards.rst similarity index 100% rename from docs/source/development-guide/style-guide/review-standards.rst rename to docs/source/development/style-guide/review-standards.rst diff --git a/docs/source/development-guide/style-guide/security.rst b/docs/source/development/style-guide/security.rst similarity index 100% rename from docs/source/development-guide/style-guide/security.rst rename to docs/source/development/style-guide/security.rst diff --git a/docs/source/development-guide/style-guide/style-guide.rst b/docs/source/development/style-guide/style-guide.rst similarity index 100% rename from docs/source/development-guide/style-guide/style-guide.rst rename to docs/source/development/style-guide/style-guide.rst diff --git a/docs/source/development-guide/style-guide/tools-and-library-recommendations.rst b/docs/source/development/style-guide/tools-and-library-recommendations.rst similarity index 100% rename from docs/source/development-guide/style-guide/tools-and-library-recommendations.rst rename to docs/source/development/style-guide/tools-and-library-recommendations.rst diff --git a/docs/source/development-guide/style-guide/versioning.rst b/docs/source/development/style-guide/versioning.rst similarity index 100% rename from docs/source/development-guide/style-guide/versioning.rst rename to docs/source/development/style-guide/versioning.rst diff --git a/docs/source/development/technology-stack.rst b/docs/source/development/technology-stack.rst new file mode 100644 index 000000000..76c5a39cd --- /dev/null +++ b/docs/source/development/technology-stack.rst @@ -0,0 +1,149 @@ +Technology Stack +---------------- + +This page lists the various technologies and libraries that the IMAP SDC +utilizes along with a few notes on what they are used for, why they were chosen, +and what other options were considered. It is worth noting that the development +of the SDC began in ~2022-2023, and so many of the technology decisions reflect +the technological ecosystem of that time. + +AWS +~~~ + +The SDC uses Amazon Web Services (AWS) as the solution for `cloud +infrastructure `_ +used for data processing. Amongst cloud vendors, AWS was chosen mostly because +of its popularity within scientific software applications. Other cloud vendors +such as Microsoft Azure and Google Cloud were not really considered. + +AWS was chosen over an on-prem architecture due to the general movement within +the community to move data processing and storage to the cloud. + +The following sections describe the particular cloud services used in the SDC: + +CDK +""" + + +API Gateway +""""""""""" + +Lambda +"""""" + +Batch +""""" + +EventBridge +""""""""""" + +RDS +""" + +`RDS `_ was +famously chosen after an extremely lengthy discussion going back-and-forth on +pros and cons of various database technologies (OpenSearch and DynamoDB were the +other major considerations). RDS was ultimately chosen for being the "old +reliable" option, as it has been extensively used and is extremely well +documented. + +S3 +"" + +cdflib + Xarray +~~~~~~~~~~~~~~~ + +The Python `xarray `_ and `cdflib +`_ libraries are used for creating data +structures for IMAP data and reading/writing those data structures to CDF files, +respectively. ``cdflib`` was chosen for CDF file I/O because of its convenient +``xarray_to_cdf()`` and ``cdf_to_xarray()`` functions. Additionally, the main +developer for ``cdflib`` (Bryan Harter) is also a developer for the IMAP SDC, +and so there is a lot of in-house knowledge of the library. ``xarray`` was +chosen for its support of data structures that closely match the format of CDF +files (i.e. use of data variables, data attributes, time coordinates, etc.). + + +Common Data Format +~~~~~~~~~~~~~~~~~~ + +The Common Data Format (CDF) was selected as the file format for IMAP data from +requirements. CDF is a widely used data format within the Heliophsyics +community. This decision was based purely on `requirements +`_. +As such, no other data formats were considered. + +Docker +~~~~~~ + +GitHub +~~~~~~ + +The SDC uses `GitHub `_ for +version controlling its software, as well as keeping track of development tasks +(i.e. GitHub Issues) and progress (i.e. GitHub Projects), and performing code +reviews. GitHub was chosen over other solutions like GitLab and Bitbucket mainly +for its collaborative features and unlimited free public repositories. As the +IMAP SDC strives to comply with the `NASA SMD SPD-41a policies +`_, +(adopted in 2022) this open-source collaborative GitHub solution made the most +sense. + +Poetry +~~~~~~ + + +Pytest +~~~~~~ + +The `pytest `_ python library was chosen for writing +and executing unit tests. This library was chosen because it has a large and +active community of users, is well documented, and has several features that +allow for a robust testing framework (e.g. fixtures, parametrized testing, +object mocking, etc.). There are other tools/libraries for unit testing in +Python (e.g. ``unittest``, ``nose``, etc.), but those were not really +considered. + + +Python +~~~~~~ + +The SDC uses Python as the primary programming language for the implementation +of the vast majority of the system and its ancillary tools. A few other +languages are supported for Level 3 `algorithm development containers +`_, +and a lot of the SDC cloud infrastructure is built with AWS CDK, but otherwise +everything is written in Python. Python was chosen mostly because at the time of +development, it was the most widely used and supported language for scientific +software development, and the main programming language used within the +Data Systems group at LASP. + + +Space Packet Parser +~~~~~~~~~~~~~~~~~~~ + +The SDC uses the `space_packet_parser +`_ library for the +decommutation of CCSDS packets and processing L0-level data. This library was +chosen for its support of XTCE format for telemetry definitions. Another benefit +to using this library is that it was developed and is actively maintained +in-house here in Data Systems at LASP (Gavin Medley), and so the library can be +updated to help meet the needs of the IMAP SDC. The other option for packet +decommutation would have been `CCSDSPy `_. + +Sphinx + ReadTheDocs +~~~~~~~~~~~~~~~~~~~~ + +The SDC uses the Python `sphinx `_ +library for generating project documentation as well as reference documentation +that gets automatically rendered from code docstrings. `ReadTheDocs +`_ is used to host the +documentation via the HTML files that ``sphinx`` generates. Together, these +tools product the 'official' `IMAP SDC documentation +`_. + +These tools were chosen because they are widely used and have great integration +with Poetry and GitHub. + +Sqlalchemy +~~~~~~~~~~ diff --git a/docs/source/index.rst b/docs/source/index.rst index df621d721..0b7f4de8e 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -20,7 +20,7 @@ The explicit code interfaces and structure are described in the :ref:`code-docum :maxdepth: 1 code-documentation/index - development-guide/index + development/index project-management/index external-tools/index data-access-api/index