Update Context to be thread-safe

[krlberry]

Objective

Update the Context mechanism for the Prefect workflow to be thread-safe. The current design is not thread-safe, as indicated from testing during the previous sprint.

Some potential ideas for an approach are here: context

Requirements

• The context mechanism has been updated to a thread-safe implementation
• Testing to ensure that the context is thread-safe
• Update the wiki page to include the new design for the context and documentation of the thread-safety issue that triggered the design change

Key Decision Points

• Design for a thread-safe context.

Some possibilities include:

• Use the Prefect database
• Use a different database
• REDIS

[amcnicho]

Execution of pipeline.py had errors coming from the contents of the context being mismatched with the keys written by previous stages, so when a subsequent parameterized flow requested an expected key, that failed:

  File "RADPS/prefect_workflow/stage_image_cont_selfcal.py", line 183, in solve
    n_field = datashape[src]['n_field']
KeyError: 'J1752-2956'

This was seen when the pipeline was submitted to helm deployment of prefect-server, but not when running in a local process (prefect server start &).