[FEAT] Provide a Connection/Secret provider/handler to define generic configuration for Activities (ConnetionString, Storage Account...)

[jdevillard]

Feature Request

Problem Overview / Use Cases

Is your feature request related to a problem? Please describe.

As a developer, I would like to define some sort of Connection or Secret in a central repository.
For example, a SQL Connection String or a couple of property that define general settings for a Activity.

For a database connection, we could have activity like Select, Insert etc... which can be defined in differents activity but rely on the ConnectionString Property to configure the SQLClient.

For a Cloud Storage Connection, (eg: S3), we have the following properties :

• accessKey
• secretAccessKey
• region

which can be group in a Connection concept that identify a configuration to a specific S3 repository

Then this Connection is used in the client for different activities :

• Upload File
• List Objects
  etc...

In this case, If I want then to modify the S3 Storage Account, I can do it in one central place with no need to change the configuration in all the workflow that use this activity.

As a Operator, I want to be able to configure these connections in different environnement (Dev/UAT/Prod) and then just Export / Import Workflow without changing any configuration.

As an Administrator, I can be able to CREATE Connection configuration and the developer of the workflow can USE the connection but NOT READ it

Proposed Solution

Describe the solution you'd like

I would like to define something like a ConnectionProperty<T> in the Activity.
Where T is the model representing the properties use for the configuration.

In the Workflow Definition, only the connection Name should be appears in the JSON because Sensitive information can be present in the configuration. We could for example :

• use new [Attribute] to define the connection type, connection model
• use a IActivityExecutionMiddleware before the ActivityExecutionMiddleware which can be able to :
  • identify that activity has connection and get the Type
  • get the connection name
  • use a ConnectionStore to retrieve the properties regarding (Type, ConnectionName)
  • Set the T model of the Activity

In the Log Persistence, I would like to be able to obfuscate some property of the T model (enforced in code, maybe not unlockable by the strategy , or with some debug flag (to be sure that no leak can be done on a production environment)
We could for example :

• define a [NoLog] attribute
• use a ActivityStateFilterBase to obfuscate some part of the T model

(The [Nolog] attribute could be more generic and the StateFilter can for eg: obfuscale just a part of a sql connection string.

then we need a UI to CRUD the connection regarding activity Type available in the Registry.

Impact of Feature

Configuration centralization, easier continuous delivery and transport configuration between environment.

Visuals and Mockups

Provide any visuals

Drop down to select connection, we can define a custom UI-hint and add a ADD button to be able to create and use a new connection directly from this screen

obfuscation :