EPIC: CI/CD build Prod long chain scripts

[RobHanna-NOAA]

These are scripts that can run a series of production 'Build' steps. These include a simple QA script after each current build script. The pattern becomes one wrapper file, which executes a series of production tool build scripts with QA data validation scripts between each one. This creates a script that can be run in a long chain to take care of multiple steps to build production data.
For example: A new CatFIM "Build" script, runs CatFIM SB, then run a simple QA script to validate the results of it, then CatFIM FB, then a QA script validates it, then runs CatFIM Compare, etc.

Requirements:

• Each steps must pass the tests at the QA script level to validate for errors and valid data before progressing to the next step.
• Each Build script has its own config file which passes key variables into each of its child scripts.
• This system needs it own independent logging system
• Keep in mind the option to eventually add AWS tools to speed up any particular section.
• Add sections for deploying at key steps including sometimes deploying some files to HV and others to FIM.
• Any script can be daisy chained to other scripts
• Copying files to other locations, named Deploy scripts, are independent scripts and are independent steps.
• Create an code architecture that can eventually add long chain acquire and pre-process scripts. ie) pull osm bridges, make bridge dem diffs, etc.

Build Wrapper Scripts required: (more details to each step coming)

• Deploy HAND files for production releases
• CatFIM SB
  • Validate primary HAND dataset exists and any other key input data exist
  • Run CatFIM SB
  • Run CatFIM SB QA validation tool
  • Run CatFIM SB Compare
  • Deploy CatFIM SB Copy to FIM S3
  • Deploy CatFIM SB Copy files to HV
• CatFIM FB
  • Validate primary HAND dataset exists and any other key input data exist
  • Run CatFIM FB
  • Run CatFIM FB QA validation tool
  • Run CatFIM FB Compare
  • Deploy CatFIM FB Copy to FIM S3
  • Deploy CatFIM FB Copy files to HV
• FIM Performance (Catchments) - EFS
  • more details coming
• FIM Performance (Points / Polys) - UAT server
  • more details coming
• USGS Rating Curves
  • more details coming
• Sierra
  • more details coming