[RIPPLE] Isolated Step Function TEST to execute flows2fim using LAMBDA

[LorneLeonard-NOAA]

Objective: Using isolated step function to execute ONE Ripple example

• Access test data via S3 buckets
  • Control Inputs
  • FIM Inputs [RIPPLE] GDAL Virtual File Systems #1069
• Using HUC submitted to Lambda, search S3 library for correct model folder
• Execute flows2fim using Python process
  • Create Control File
  • Create FIM cog
    • GDAL environment required TBD [RIPPLE] GDAL environment with Lambda #1071
    • GDAL Virtual File Systems TBD [RIPPLE] GDAL Virtual File Systems #1069
• Convert Raster result to vector format
• Upload vector data to PostgresSQL db
• Code and Data cleanup
• Benchmark and evaluate performance

Needed to create a docker image to bypass 250 MB limit for lamda step function

• Create Dockerfile with following details:
  • Setup Python3 environment
  • Setup gdal environment needed to execute flows2fim
  • Setup conda environment to have more choices in vector conversion
  • Copy applications required.
  • Setup entry point and command settings
• Create Python file with following details (S3 bucket focused):
  • Boto functions to map details from user inputs
  • Argument settings to handle various buckets and options
  • Setup logging information for various debugging stages
  • Check user inputs
  • Check user output settings
  • Create Control file
  • Create FIM raster file (cog tif)
  • Add more error handling from flows2fim (both control and fim)
  • Convert FIM raster file to vector file (parquet)
  • Prepare vector data for database
  • Preprocessing of vector data for visualization pipeline.
  • Add more vector file output options
  • Upload vector file to database.
  • Upload log file to designated S3 bucket
• Code and Data cleanup
• Benchmark and evaluate performance
  • ble_12030106_EastForkTrinity
  • ble_11050003_LowerCimarron
  • ble_11140102_Blue
  • ble_12030108_Richland
  • ble_12040101_WestForkSanJacinto
  • ble_12040103_EastForkSanJacinto
  • ble_12090301_LowerColoradoCummins
  • ble_12100202_MiddleGuadalupe
  • mip_02050206
  • mip_03150110
  • mip_11140102
  • mip_12010004
  • mip_12010005
• Modify above Python file to work with local file access.
• Added docker image to Lambda environment
• Error checking logic for Lambda/Docker envs.

Note: It has already been detected that one HUC can have more than one model folder, ie) one for ble and one for mip. Figure out how we handle that. Might need a loop inside the lambda to run more than once if more than more than one model folder exists.