Why run tasks in graph based on layers?

[sroussey]

I did this at one point for the purpose of doing certain kind of pipelines, but this is not the most efficient for the general case, so make this optional and do the easier thing in the general case.

[sroussey]

Ah yes, I remember now, and should doc this somewhere. When you have a DAG (and that is all we deal with at the moment), having a dependency graph organized in layers is a fast way to run the calculations assuming they are all about the same amount of work.

This is the case for the reactive stuff.

For the processing stuff... well, maybe not. Processing stuff that is long running async will likely run better if you get the first layer and run that in parallel, then notify dependents and when processing done. The dependent would then wait for all the parents to complete before running. This would let later layers start processing sooner rather than waiting for all parents siblings to be done.

To do the above will require some work though... you could use the fact that the DataFlow (edges) have a value and if that value is not the default, then it has been processed. Or the DataFlow can have things explicit rather than implicit and have a status. Explicit is usually easier to understand and debug, and it could help if we want to have partial data flow results and values.