What is the proper way to link a process in Livebook? (e.g. when running Flows/GenStage in separate processes)

[linusdm]

I’m playing with flow, which starts a bunch of processes in the background. All is good when the flow isn’t interrupted by any errors. But when an error happens, Livebook seems to be confused. The error is logged in the console, and a flash message appears in the upper right corner (“Evaluation process terminated - shutdown”). But then the underlying runtime seems to die (I'm running an Elixir standalone runtime). When I try to rerun the code-cell, any variables defined upstream are not defined anymore.

I think I have to do something with Kino.start_child/1 and apply supervisable flows somehow, but I’m not sure how to stitch those together. I surely can make a module containing the flow logic, but I still don’t know how to create a child_spec for it.

I’d like to collect the results back in the calling code-cell/process, to get the final result and print it in the cell. Any ideas for how to tackle this? It’s probably more generic in the sense that any process started this way won’t be linked properly.

[jonatanklosko]

Hey @linusdm!

Whenever we do start_link explicitly, it ties the given process to Livebook evaluation process, so if the linked process crashes, so does the evaluation process, and consequently all the state/variables is gone. Usually we would put such process in our app suprevision tree and in Livebook the counterpart is to use Kino.start_child/1, which makes sure the process is terminated when you reevaluate the cell.

When you do define your own module like MyFlow and do use Flow, it defines child_spec for you, so you can do this:

defmodule MyFlow do
  use Flow

  def start_link(_) do
    # ...
    |> Flow.start_link()
  end
end

Kino.start_child(MyFlow)

If you don't want to define a module, you can write down the child spec explicitly:

flow = ...

Kino.start_child(%{id: MyFlow, start: {Flow, :start_link, [flow]}})

In either case it runs in a separate process, so to get results back in the cell we would need to send them explicitly. @josevalim what's the best way to do that?

[josevalim]

My suggestion would be to add Process.flag(:trap_exit, true) in such cases.

[josevalim]

From Flow v1.2.1 you can also do Kino.start_child {Flow, Flow.from_stages() |> ...}. Note the Flow is permanent by default though (it will restart if it terminates).

[linusdm]

Thank you both for taking the time!

This makes things clearer to me, and got a version working. I can confirm that the background processes stop, when re-evaluating the same cell.

Process.flag(:trap_exit, true)

I don't see how trapping exits help with collecting the result back into the code-cell.

I had the assumption that calling Kino.start_child/1 would block and return the results, but that's obviously wrong. The code in the cell terminates immediately, and the background processes do their thing.

Note the Flow is permanent by default though (it will restart if it terminates).

Again, one step ahead of me (found out the hard way) :) A suitable :restart strategy worked for me:

use Flow, restart: :temporary

From Flow v1.2.1 you can also do Kino.start_child {Flow, Flow.from_stages() |> ...}

Is this an alternative for starting a flow without defining a dedicated module? Otherwise I'm a bit lost with that last remark.

Thanks again for taking the time, I really appreciate it! 💜

[linusdm]

Although this setup is perfect, and allows re-evaluation of code-cells, without having dangling processes, it would be nice to have UI controls to start and stop the underlying processes/flow. Then you could start the process on demand, and also stop it with a button or something. That's more flexible than only being able to re-evalute the code cell. Also, the state could be reflected in some kind of green/red dot (similar to how the execution of the cells is being visualized).
Infinite flows (GenServers, GenStages, Broadway, etc.) could be interrupted easily, as they otherwise would continue to process. And finite flows could be interrupted prematurealy, for whatever reason (maybe it takes too long, or an error popped up). Although in that last case, just re-evaluating the code-cell would suffice 🤔

Just thinking out loud... Probably doable in a smart cell?

[josevalim]

It would be a regular Kino.JS.Live output but sounds like doable. For a broadway version, you could get inspiration from the Broadway Dashboard project. :)