Updated output handling of asynchronously run workflows to be the same as when run synchronously #6228
Conversation
…e as when run synchronously
| { | ||
| elsa.UseWorkflowRuntime(workflowRuntime => { | ||
| workflowRuntime.ActivityExecutionLogStore = sp => activityExecutionStore; | ||
| workflowRuntime.WorkflowRuntime = sp => sp.GetRequiredService<DistributedWorkflowRuntime>(); |
There was a problem hiding this comment.
Not sure if this is a separate bug or not... but to be able to run multiple asynchronous activities in parallel (the only way it makes sense to me) the new DistributedWorkflowRuntime workflow runtime must be used. I don't believe that this was not the case with 3.2.
There was a problem hiding this comment.
This was indeed intentional to require the DistributedWorkflowRuntime or ProtoActorWorkflowRuntime to have support for parallel activity execution. To be honest, I'd love it to be able to still have support for parallel activity execution using the LocalWorkflowRuntime. Originally, the DistributedWorkflowRuntime was created for hosting Elsa on multiple nodes, which requires distributed locking. But it's fair to expect that when running on a single node, you should still be able to run activities in parallel. Perhaps this could be added easily by updating the LocalWorkflowRuntime with a semaphore to synchronise access to the workflow instance.
src/modules/Elsa.Workflows.Runtime/Middleware/Activities/BackgroundActivityInvokerMiddleware.cs
Show resolved
Hide resolved
|
|
||
| return outputValues; | ||
| { | ||
| var activityOutputRegister = activityExecutionContext.WorkflowExecutionContext.GetActivityOutputRegister(); |
There was a problem hiding this comment.
I fear that this "solution" may be too simplistic since the old code clearly went out of it's way to only return values that are backed by WorkflowInstanceStorageDriver. I'm not sure that this is a concern now considering that the parent workflow seems to be now resumed in the same thread. Anyway - I'd welcome discussion in this particular change (as well as any other change!).
There was a problem hiding this comment.
The concern is that when an activity produces a large object, e.g. a file or JSON string of 5 MB, the user might use a different variable storage driver, such as "Blob Storage Driver" that stores the data in Azure Storage.
For this reason, we want to avoid those large values to become part of the output register, given that it will be serialized as well.
Obviously, the current implementation is rather crude and intended to be a temporary hack.
The ultimate solution is to allow for better control of how output is recorded.
For example, for small values, it might be OK to be recorded. The assumption here is that if the storage driver is "Workflow Instance", which stores the value as part of the workflow instance, it's safe to also include it as part of the output register.
For large values, the assumption is that the user will have used the Memory storage driver or a custom one such as "Blob Storage Driver". In this case, we definitely do not want the value to be recorded in the output register.
Now, we actually do want to record something in the output register. In the case of large blobs, a download link could be appropriate.
So perhaps a cleaner approach would be to leverage the associated storage driver to have it produce a value suitable for storing in the output register. This would clean up this code because now we no longer need to test against the storage driver type.
Pending that solution, perhaps for now it might be best to keep the code to only include variables into the output if those variables are associated with a workflow instance driver.
|
|
||
| return outputValues; | ||
| { | ||
| var activityOutputRegister = activityExecutionContext.WorkflowExecutionContext.GetActivityOutputRegister(); |
There was a problem hiding this comment.
The concern is that when an activity produces a large object, e.g. a file or JSON string of 5 MB, the user might use a different variable storage driver, such as "Blob Storage Driver" that stores the data in Azure Storage.
For this reason, we want to avoid those large values to become part of the output register, given that it will be serialized as well.
Obviously, the current implementation is rather crude and intended to be a temporary hack.
The ultimate solution is to allow for better control of how output is recorded.
For example, for small values, it might be OK to be recorded. The assumption here is that if the storage driver is "Workflow Instance", which stores the value as part of the workflow instance, it's safe to also include it as part of the output register.
For large values, the assumption is that the user will have used the Memory storage driver or a custom one such as "Blob Storage Driver". In this case, we definitely do not want the value to be recorded in the output register.
Now, we actually do want to record something in the output register. In the case of large blobs, a download link could be appropriate.
So perhaps a cleaner approach would be to leverage the associated storage driver to have it produce a value suitable for storing in the output register. This would clean up this code because now we no longer need to test against the storage driver type.
Pending that solution, perhaps for now it might be best to keep the code to only include variables into the output if those variables are associated with a workflow instance driver.
| { | ||
| elsa.UseWorkflowRuntime(workflowRuntime => { | ||
| workflowRuntime.ActivityExecutionLogStore = sp => activityExecutionStore; | ||
| workflowRuntime.WorkflowRuntime = sp => sp.GetRequiredService<DistributedWorkflowRuntime>(); |
There was a problem hiding this comment.
This was indeed intentional to require the DistributedWorkflowRuntime or ProtoActorWorkflowRuntime to have support for parallel activity execution. To be honest, I'd love it to be able to still have support for parallel activity execution using the LocalWorkflowRuntime. Originally, the DistributedWorkflowRuntime was created for hosting Elsa on multiple nodes, which requires distributed locking. But it's fair to expect that when running on a single node, you should still be able to run activities in parallel. Perhaps this could be added easily by updating the LocalWorkflowRuntime with a semaphore to synchronise access to the workflow instance.
These changes result in outputs of activities run asynchronously to be treated the same as if the activity were run synchronously with regards to ActivityExecutionRecord and the activity output register.
See #6227 for more details.
This change is