Start lot of workflows from another one

[ZalinoSoft]

Hi, guys! I have a workflow and its goal is at a specific time to read a set of files and perform some operations with each of them sequentially. Everything works as expected except the fact that the entire amount of time spent on completing the workflow (compute files) is too much. So, I thought to compute the files in parallel in order to reduce the total amount of time spent. To achieve this, I set up a workflow (TRIG) that at a specific time just read the name of the files and for each name it starts a new computation workflow (COMP) sending the file name as parameter. With 10 files is OK since TRIG starts only 10 COMP workflows. However, when trying with 200 files after starting the last COMP workflow, the TRIG workflow throws "java.lang.RuntimeException: Failure processing decision task" and it re-starts itself and creates again 200 other COMP workflows... and so on... I don't know how to solve this and I thought to ask here: is there anything wrong in starting hundreds of workflows from another one? Should I configure something special in cadence or worflows (pooler, etcs)?Should I change my design to achieve the parallelism in other way? Any suggestion?

[longquanzheng]

Is it a distributed file system ? So every worker can access to the same files.

[longquanzheng]

Assuming the file system is not local..otherwise I don’t think multiple workflow can access to the same files.

General idea is good but you need to make it deterministic. Each file should only start exactly one workflow. Cadence provides ID uniqueness support so that any retry will not starting same workflows again.

More specifically, you can use a single activity to load all the file names and then use the file names as workflow ID to start COMP workflows.

[ZalinoSoft]

Your assumption is correct about the files. Regarding your idea, this is exactly what I tried to do. But starting 200 COMP workflows from the TRIG workflow seems too much for cadence. Is there any limitation in this scenario (starting hundreds of workflows from another one)?

[longquanzheng]

That’s not true. We start thousands of workflows concurrently without any problems.

If your cluster is too small for that, you can add some small sleep between each start request. If you are worried about activity retry would make it restart from beginning you can use activity heartbeat to checkpoint and restore.

But generally 200 is very small number. I don’t think it’s a problem

[meiliang86]

@ZalinoSoft Can you post the full exception for "java.lang.RuntimeException: Failure processing decision task"? The wrapped exception might have details.
Also, how do you start these child workflows? Depending on how you do it in your code, you might/might not consume Java thread, which is quite expensive.