doc

_sphinx_config/pages/aws/automation.rst

@@ -43,9 +43,15 @@ name in the RIIS record to the GBIF 'species' API, to find the accepted name,
 
 * Create an AWS EventBridge Schedule
 
+  * Specify schedule detail: Name, Recurring, Cron-based schedule (allow enough time for
+    the previously scheduled task to complete).
+  * Select target: All APIs, AWS Lambda Invoke
+  * Settings:
+
+
 * Create a lambda function for execution when the trigger condition is activated, in
   this case, the time/date in the schedule.
-  aws/lambda/bison_s0_annotate_riis_lambda.py
+  aws/lambda/bison_s1_annotate_riis_lambda.py
 
   * The lambda function will make sure the data to be created does not already exist
     in S3, execute if needed, return if it does not.
@@ -62,10 +68,10 @@ schedule, roughly estimating completion of the previous step.  These steps with
 dependency on previous outputs will first check for the existence of required inputs,
 failing immediately if inputs are not present.
 
-Automatic execution (TODO):  The successful deposition of output of the first
+Triggered execution (TODO):  The successful deposition of output of the first
 (scheduled) and all following steps into S3 or Redshift triggers subsequent steps.
 
-Both automatic and scheduled execution will require examining the logs to ensure
+Both triggered and scheduled execution will require examining the logs to ensure
 successful completion.
 
 

_sphinx_config/pages/aws/lambda.rst

@@ -1,16 +1,28 @@
 Create lambda function to initiate processing
 --------------------------------------------
 
-Lambda functions can execute tasks in a workflow.  They can be initiated when a trigger
-condition occurs, set (separately) in the Amazon EventBridge as either Rules or
-Schedules.
+Lambda functions can execute tasks in a workflow.  They can be initiated manually, or
+automatically with the Amazon EventBridge by either a trigger condition (through Rules)
+or on a timetable (through Schedules).
 
-Lambda functions can only execute for up to 15 minutes at a time, but can launch longer
-running processes in another service, such as launch an EC2 instance to run a
-BISON workflow tasks.
+Lambda functions can only execute for up to 15 minutes at a time, but can initiate
+longer running processes in another service, such as launch an EC2 instance to run a
+BISON workflow task.
+
+Short tasks that require specific or in-development code, may be run in a Docker
+container on an EC2 instance.  The Docker image can be built with code pulled from a
+repository or the pre-built image pulled from a Docker Hub.
 
 Lambda functions log directly to AWS Cloudwatch - make sure the role executing
-the lambda function has Create/Write permissions to AWS Cloudwatch.
+the lambda function has Create/Write permissions to AWS Cloudwatch.  All print
+statements will go to the log.  More information is in the
+
+The BISON workflow has two steps that are run on an EC2 instance.  Both launch an EC2
+instance from a template which contains a startup script in the `userdata` for the
+EC2 instance.  The startup script builds a Docker image from the latest code in the
+BISON Github repository, then executes a task defined in that code.  The steps are
+the first and last of the workflow: **bison_s1_annotate_riis** and
+**bison_s8_calc_stats**, both in the bison/aws/lambda directory.
 
 Edit the Configuration for lambda function
 --------------------------------------------

_sphinx_config/pages/aws/logging.rst

@@ -0,0 +1,34 @@
+Logging to AWS Cloudwatch
+#####################################
+
+Workflow steps executed in Lambda Functions
+=====================================
+
+Lambda functions log directly to AWS Cloudwatch - make sure the role executing
+the lambda function has Create/Write permissions to AWS Cloudwatch.  All print
+statements will go to the log.
+
+Logs will be sent to a Log Group, prefixed by '/aws/lambda/' followed by the name of
+the lambda function, such as **/aws/lambda/bison_s4_intersect_bison**.  A new stream
+under that log group will be created for every run of the function.
+
+Workflow steps executed in EC2 instance
+========================================
+
+EC2 instances which execute steps will send Docker logs to a Log Group created for this
+purpose, **bison_task**.  A stream will be created for each task, with logging
+statements for all runs in the same stream.
+
+This logging is configured in the Docker compose file for each task.  Use the task name
+defined in the EC2 Launch Template version for the task.  For task calc_stats, this
+is the logging configuration of the compose file::
+
+    logging:
+      driver: awslogs
+      options:
+        awslogs-region: us-east-1
+        awslogs-group: bison_task_calc_stats
+        awslogs-stream: calc_stats
+
+TODO: Create a new log group for each lambda- initiated EC2 workflow task, with a new
+stream (named with timestamp) for each execution.