Small stylistic fixes and add pre-commit config

.dockerignore

@@ -3,4 +3,4 @@
 *.iml
 .idea
 docker_host_volumes
-/venv
+/venv

.github/ISSUE_TEMPLATE.md

@@ -12,6 +12,3 @@ Short description explaining the high-level reason for the new issue.
 
 
 ## Screenshots
-
-
-

.github/workflows/integration_tests.yml

@@ -71,7 +71,7 @@ jobs:
           python3 -m venv dmod_venv
           . dmod_venv/bin/activate
           pip install --upgrade pip
-          deactivate 
+          deactivate
           export PROJECT_ROOT=$(pwd)
           echo ${PROJECT_ROOT}
           ./scripts/update_package.sh --venv dmod_venv --no-service-packages --dependencies
@@ -146,4 +146,3 @@ jobs:
           . dmod_venv/bin/activate
           python3 -m pip install pytest
           python3 -m pytest -o python_files="it_*.py" -o env_vars='IT_REDIS_CONTAINER_PASS="DPXzqRqjhsXokOVQcPUqOJuzKePMsfUc" IT_REDIS_CONTAINER_HOST_PORT=19639 MODEL_EXEC_ACCESS_KEY=dmod MODEL_EXEC_SECRET_KEY=password'
-

.github/workflows/testing_and_validation.yml

@@ -43,7 +43,7 @@ jobs:
           python3 -m venv dmod_venv
           . dmod_venv/bin/activate
           pip install --upgrade pip
-          deactivate 
+          deactivate
           ./scripts/update_package.sh --venv dmod_venv --dependencies
 
       - name: Cache Package Checksums

.pre-commit-config.yaml

@@ -0,0 +1,8 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.6.0
+    hooks:
+      - id: check-added-large-files
+      - id: check-merge-conflict
+      - id: end-of-file-fixer
+      - id: trailing-whitespace

INSTALL.md

@@ -20,7 +20,7 @@ Detailed instructions on how to install, configure, and get the project running.
 
 ## Local Environment Configuration
 
-Each local deployment and/or development environment requires its own configuration be set up in order to function properly.  This is currently done via a local environment file.  
+Each local deployment and/or development environment requires its own configuration be set up in order to function properly.  This is currently done via a local environment file.
 
 Because of its custom nature and potentially sensitive contents, local environment configs should never be committed to the Git repo.  As such, one must be custom built for a given environment.
 
@@ -40,12 +40,12 @@ The environment config can also be created by copying the [example.env](example.
 
 ## Creating Required Docker Networks
 
-A Docker-Swarm-based DMOD deployment currently requires [several Docker networks](#about-the-required-docker-networks) be set up for different communication with and between the services and job workers.  These networks are set up automatically by the `.scripts/control_stack.sh` tool, based on values in the [local environment config](#local-environment-configuration).  
+A Docker-Swarm-based DMOD deployment currently requires [several Docker networks](#about-the-required-docker-networks) be set up for different communication with and between the services and job workers.  These networks are set up automatically by the `.scripts/control_stack.sh` tool, based on values in the [local environment config](#local-environment-configuration).
 
 Network config values (and possibly, a [manually constructed config-only network](#special-options-for-mpi-worker-network)) should be set up before running the control script.  However, the networks can be manually removed and re-created, either manually or automatically on subsequent script execution.
 
 ### Special Options for MPI Worker Network
-Special options are available for the MPI worker network to support higher performance via the MACVLAN network driver.  These are `DOCKER_MPI_NET_USE_CONFIG` and `DOCKER_MPI_NET_VXLAN_ID`.  
+Special options are available for the MPI worker network to support higher performance via the MACVLAN network driver.  These are `DOCKER_MPI_NET_USE_CONFIG` and `DOCKER_MPI_NET_VXLAN_ID`.
 
 To utilize this, a user must first *manually* create a Docker config-only network (essentially, pre-stage the config).  This is done by adding the `--config-only` flag to `docker network create`, along with the other required args.  The name of this config must also match the network name in `DOCKER_MPI_NET_NAME`, with the suffix of `-config` appended.
 
@@ -55,7 +55,7 @@ If `DOCKER_MPI_NET_USE_CONFIG=true`, the MPI worker network will be created usin
 
 The networks are:
 
-- (MPI) Job Worker Network 
+- (MPI) Job Worker Network
   - Purpose: job worker communication, with other workers or something external (e.g., object store data source)
   - Default name: `mpi-net`
 - Requests Network
@@ -96,9 +96,9 @@ Note that the `local/` directory is primarily for tests, rather than a service.
 
 As illustrated, each service sub-directory must then contain two files:
 
-* _certificate.pem_ 
-  * the actual SSL certificate 
-* _privkey.pem_ 
+* _certificate.pem_
+  * the actual SSL certificate
+* _privkey.pem_
   * the private key used to create the certificate
 
 ### Setting Up Directories and Certificates
@@ -117,7 +117,7 @@ See the `./scripts/gen_certs.sh --help` for more details on the scripts options.
 > Users are always free to manually create the directory structure and obtain (or individually create with `gen_certs.sh`) the required certificate files.
 
 > [!IMPORTANT]
-> Users must configure the deployment environment properly, such that `DMOD_SSL_DIR` in the config points to the aforementioned top-level directory. 
+> Users must configure the deployment environment properly, such that `DMOD_SSL_DIR` in the config points to the aforementioned top-level directory.
 
 ## Create Job Worker SSH Keys
 The job worker containers require SSH keys in order to communicate with each other during an MPI job.  The location of these is configurable in the local environment, specifically under `DOCKER_BASE_IMAGE_SSH_HOST_DIR` and `DOCKER_NGEN_IMAGE_SSH_HOST_DIR`.
@@ -128,17 +128,17 @@ The `./scripts/control_stack.sh` script will automatically create directories an
 
 ## Installing Python and Path Verification
 
-As noted in the [dependencies](doc/DEPENDENCIES.md), Python, the Python development package, and Pip should be installed in the appropriate manner for your environment.  
+As noted in the [dependencies](doc/DEPENDENCIES.md), Python, the Python development package, Pip, and pre-commit should be installed in the appropriate manner for your environment.
 
 ### Different Python Executable Names
 It is quite common for the Python executables to be named `python3`, `python311`, `pip3`, etc. in the *global* Python environment, instead of just `python` and `pip`.  This is accounted for when [creating a virtual environment](#using-a-virtual-environment), provided the right Python command was used to create it.  Users not doing this should consider setting up shell aliases, symlinks in their path, or some other mechanism to ensure `python` and `pip` execute the desired versions appropriately.
 
 > [!NOTE]
-> DMOD components and documentation may assume use of `python` or `pip`. 
+> DMOD components and documentation may assume use of `python` or `pip`.
 
 ## Using a Virtual Environment
 
-It is recommended that contributors use a Python virtual environment if they intend to do development on the Python source code used by the project.  This is not an absolute requirement, but typically is appropriate. 
+It is recommended that contributors use a Python virtual environment if they intend to do development on the Python source code used by the project.  This is not an absolute requirement, but typically is appropriate.
 
 ```bash
 # Typically it's best to run this from within the local DMOD repo directory
@@ -152,15 +152,15 @@ The project is tested to work with environments created by the `venv` module.  A
 
 ## Python Packages and Dependencies
 
-The internal project packages have several external dependencies.  Some are "standard" dependencies, while others are required for building package distribution files.  
+The internal project packages have several external dependencies.  Some are "standard" dependencies, while others are required for building package distribution files.
 
-Additionally, there are dependency relationships between several of the internal project packages.  As such, to develop (and especially to test) reliably, internal packages also need to be installed in the local Python environment.  
+Additionally, there are dependency relationships between several of the internal project packages.  As such, to develop (and especially to test) reliably, internal packages also need to be installed in the local Python environment.
 
 ### Using `update_package.sh` for Dependencies and Internal Packages
 
-For simplicity, the [scripts/update_package.sh](scripts/update_package.sh) Bash script can help with all of this automatically.  If not passed a specific package source directory, it will build and install all internally developed Python packages.  
+For simplicity, the [scripts/update_package.sh](scripts/update_package.sh) Bash script can help with all of this automatically.  If not passed a specific package source directory, it will build and install all internally developed Python packages.
 
-This script also has options for first installing dependencies from the [requirements.txt](requirements.txt) file, either as an independent task or before proceeding to build and install the appropriate internal Python package(s).  
+This script also has options for first installing dependencies from the [requirements.txt](requirements.txt) file, either as an independent task or before proceeding to build and install the appropriate internal Python package(s).
 
 See its _help_/_usage_ message for more details.
 
@@ -244,4 +244,4 @@ This file is automatically generated by the `./scripts/control_stack.sh` script,
 
 <!--- TODO: re-add this once such a readme exists!
 See the [Docker usage README](./docker/README.md) for more information on Docker stacks, images, and building commands.
---->
+--->

NOTICE

@@ -1,4 +1,4 @@
 This project was originally inspired in part by [WALKOFF](https://github.com/nsacyber/WALKOFF).
 
 This project utilizes the Cloud Native Water Resource Modeling Hydrofabric dataset, developed by Lynker and available
-at https://www.lynker-spatial.com/.  See https://lynker-spatial.s3.amazonaws.com/copyright.html for its licensing.
+at https://www.lynker-spatial.com/.  See https://lynker-spatial.s3.amazonaws.com/copyright.html for its licensing.

README.md

@@ -102,7 +102,7 @@ end
 
 <!-- - Data Request service -->
 - Data service - Primary data and metadata store that houses model input, output, and evaluation data. This service is also responsible for creating Docker volumes to support model execution job environments (e.g. model input data).
-- Evaluation service - 
+- Evaluation service -
 - Monitor service - Monitors and updates the status of model execution job environments.
 - Partitioner service - Partitions NextGen hydrofabric domains to enable parallel execution (mpi) of NextGen model runs.
 - Request service - API gateway that coordinates communication with DMOD backend services.

SECURITY.md

@@ -1,6 +1,6 @@
 # Security Policy
 
-There MUST be no unpatched vulnerabilities of medium or higher severity that have been publicly known for more than 60 days. 
+There MUST be no unpatched vulnerabilities of medium or higher severity that have been publicly known for more than 60 days.
 
 The vulnerability must be patched and released by the project itself (patches may be developed elsewhere). A vulnerability becomes publicly known (for this purpose) once it has a CVE with publicly released non-paywalled information (reported, for example, in the <a href="https://nvd.nist.gov/">National Vulnerability Database</a>) or when the project has been informed and the information has been released to the public (possibly by the project). A vulnerability is considered medium or higher severity if its <a href="https://www.first.org/cvss/" >Common Vulnerability Scoring System (CVSS)</a> base qualitative score is medium or higher. In CVSS versions 2.0 through 3.1, this is equivalent to a CVSS score of 4.0 or higher. Projects may use the CVSS score as published in a widely-used vulnerability database (such as the <a href="https://nvd.nist.gov">National Vulnerability Database</a>) using the most-recent version of CVSS reported in that database. Projects may instead calculate the severity themselves using the latest version of <a href="https://www.first.org/cvss/">CVSS</a> at the time of the vulnerability disclosure, if the calculation inputs are publicly revealed once the vulnerability is publicly known. <strong>Note</strong>: this means that users might be left vulnerable to all attackers worldwide for up to 60 days. This criterion is often much easier to meet than what Google recommends in <a href="https://security.googleblog.com/2010/07/rebooting-responsible-disclosure-focus.html">Rebooting responsible disclosure</a>, because Google recommends that the 60-day period start when the project is notified _even_ if the report is not public. Also note that this badge criterion, like other criteria, applies to the individual project. Some projects are part of larger umbrella organizations or larger projects, possibly in multiple layers, and many projects feed their results to other organizations and projects as part of a potentially-complex supply chain. An individual project often cannot control the rest, but an individual project can work to release a vulnerability patch in a timely way. Therefore, we focus solely on the individual project's response time.  Once a patch is available from the individual project, others can determine how to deal with the patch (e.g., they can update to the newer version or they can apply just the patch as a cherry-picked solution).
 

TESTING.md

@@ -49,22 +49,22 @@ Additionally, it is possible to optionally create package-specific `.test_env` f
 ## Test Sources Directories
 
 For namespace packages, automated unit and integration test files live in a `test` sub-package/directory under the parent namespace directory (i.e., `dmod/`).  This will be a sibling to the main package source directory. Something like:
-    
+
     python/
         lib/
             access/
                 dmod/
                     access/
                     test/
-                    
+
 ## Automated Integration Testing Environment Services Setup/Teardown
 
 There can often be test environment management tasks for integration testing that don't need to (or even that expressly should not) follow the typical "setup, exec test method, teardown" convention used in unit testing. An example could be setup/teardown of a Redis instance in a testing environment.  It would likely be excessive to follow the typical unit test convention:
 
 * setup up _Redis instance 1_
 * run `integration_test_method_1()`
 * teardown _Redis instance 1_
-* setup up _Redis instance 2_ 
+* setup up _Redis instance 2_
 * run `integration_test_method_2()`
 * teardown _Redis instance 2_
 * etc.
@@ -91,12 +91,11 @@ The `test_package.sh` allows for easily running all the tests for a supplied nam
 E.g. to execute unit tests for the communication package, run (from the project root):
 
     ./scripts/test_package.sh lib/communication
-    
+
 ### `run_tests.sh`
 
-The `run_tests.sh` allows for running all unit tests or all integration tests for all of a configured group of supported packages, displaying either just a summary or more verbose output on test performance.  See the `-h` help option for more usage details.  
+The `run_tests.sh` allows for running all unit tests or all integration tests for all of a configured group of supported packages, displaying either just a summary or more verbose output on test performance.  See the `-h` help option for more usage details.
 
 E.g. to run all unit tests on supported packages and display a summary:
 
-    ./scripts/run_tests.sh 
-
+    ./scripts/run_tests.sh

conftest.py

@@ -173,4 +173,3 @@ def pytest_sessionfinish(session: Session, exitstatus: int):
 
         logging.info(f"tearing down: {module.parent.parent.name!r}")
         logging.debug(format_completed_process(out))
-

data/example_hydrofabric_1/crosswalk.json

@@ -46,4 +46,4 @@
     ],
     "outlet_COMID": 9731286
   }
-}
+}

data/example_hydrofabric_2/README.md

@@ -1,3 +1,3 @@
 # About
 
-This is a small Geopackage hydrofabric of the 11284400 basin, containing a handful of catchments.
+This is a small Geopackage hydrofabric of the 11284400 basin, containing a handful of catchments.

data/example_image_and_domain.yaml

@@ -28,4 +28,3 @@ ngen:
   output:
     local: 'docker_host_volumes/output'
     run: '/ngen/output'
-

data/example_realization_configs/ex_realization_config_01.json

@@ -94,4 +94,4 @@
         "end_time": "2015-12-30 23:00:00",
         "output_interval": 3600
     }
-}
+}

data/example_realization_configs/ex_realization_config_02.json

@@ -221,4 +221,4 @@
       ]
     }
   }
-}
+}

data/example_troute_configs/ngen_t-route.yaml

@@ -30,7 +30,7 @@ supernetwork_parameters:
     #Below will change with new catchment route link
     geo_file_path: "./Input/parameters/flowpath_parameters_mod.json"
     #CHANGE BELOW WITH NEW NGEN HYDRO FABRIC DATA
-    flowpath_edge_list: "./Input/parameters/flowpath_edge_list.json" 
+    flowpath_edge_list: "./Input/parameters/flowpath_edge_list.json"
     columns:
         #link????
         key: "id"
@@ -58,7 +58,7 @@ supernetwork_parameters:
 #waterbody_parameters:
 #    level_pool:
 #        #WRF-Hydro lake parm file
-#        level_pool_waterbody_parameter_file_path: "../../test/input/geo/NWM_2.1_Sample_Datasets/Croton_NY_TEST1/primary_domain/DOMAIN/LAKEPARM.nc" 
+#        level_pool_waterbody_parameter_file_path: "../../test/input/geo/NWM_2.1_Sample_Datasets/Croton_NY_TEST1/primary_domain/DOMAIN/LAKEPARM.nc"
 #        level_pool_waterbody_id: lake_id
 #        level_pool_waterbody_area: LkArea
 #        level_pool_weir_elevation: WeirE
@@ -93,7 +93,7 @@ supernetwork_parameters:
 #        # The default is 86400 (seconds per day). Set to 3600 for standard and extended AnA simulations.
 #        # Set to 1000000000 for short range and medium range forecasts.
 #        reservoir_observation_update_time_interval_seconds: 1000000000
-#        
+#
 #        # If using RFC forecast reservoirs, set to True. (default=.FALSE.)
 #        reservoir_rfc_forecasts: False
 #

data/serialized_dataset_examples/huc01-forcings-demo-1.json

@@ -14658,4 +14658,4 @@
   "manager_uuid" : "ba51a053-70fa-41c2-86fc-8f819bd20c6f",
   "name" : "huc01-forcings-demo-1",
   "uuid" : "9888e40f-6cac-4034-a698-6e9911b3ccbc"
-}
+}

data/serialized_dataset_examples/huc01-hydrofabric-2021-10-28-part-8.json

@@ -14659,4 +14659,4 @@
   "manager_uuid" : "ba51a053-70fa-41c2-86fc-8f819bd20c6f",
   "name" : "huc01-hydrofabric-2021-10-28-part-8",
   "uuid" : "9888e40f-6cac-4034-a698-6e9911b3ccbe"
-}
+}

data/serialized_dataset_examples/huc01-partition-8-2021-10-28.json

@@ -26,4 +26,4 @@
   "manager_uuid" : "ba51a053-70fa-41c2-86fc-8f819bd20c6f",
   "name" : "huc01-partition-8-2021-10-28",
   "uuid" : "9888e40f-6cac-4034-a698-6e9911b3ccbf"
-}
+}

data/serialized_dataset_examples/huc01-simple-config-1.json

@@ -14662,4 +14662,4 @@
   "manager_uuid" : "ba51a053-70fa-41c2-86fc-8f819bd20c6f",
   "name" : "huc01-simple-config-1",
   "uuid" : "9888e40f-6cac-4034-a698-6e9911b3ccca"
-}
+}