async-flow hazard: background async ops

[mhofman]

What is the Problem Being Solved?

In complex async flows it's tempting to start multiple operations in parallel or sometimes in the background.

E.g. Promise.all, or void asyncOp and later await some effect of the operation, or any variation.

The problem is that if the flow exits before this operations completes, e.g. on early error, the author might expect that background task to continue. And this is a failure mode without warning since the membrane is no longer paying attention to host vows resolving

Description of the Design

Some ideas:

• A way for an async flow to designate a promise as dontExitUntil to keep the flow active
• A way to diagnose any host vow that are still pending and will be ignored, either or both
  • proactively when the flow exits
  • when the vow actually settles

[erights]

@mhofman , can you create a minimal-enough self contained example?

[mhofman]

I think it can be reduced to the following guest code

const task1 = (async () {
  const result = await someHostObject.someOperationInTheBackground();
  // Do something with result, e.g.
  anotherHostObject.doFoo(result);
})();
const task2 = somePossiblyOtherHostObject.someOperationThatFails();

await Promise.all([task1, task2]);

The problem is that if the async flow exits because task2 fails first, when the task1 background operation vow settles, task1 never actually gets its result, because the membrane is gone. We effectively broke the promise without warning.

[erights]

Very helpful, thanks!

[erights]

What if you used Promise.allSettled instead?

[mhofman]

Of course making sure we never short circuit any promise resolution anywhere this wouldn't happen. But that's not how most complex async code is written.

A variation of the above is:

const task1 = ...;

const task2 = ...;

await task2;

await task1;

[mhofman]

One idea that came up was to have the orchestrate wrapper for async flow add a waitUntil method on the orchestrator object (injected as first arg to all orchestrated guest functions, and passed around in all calls). Any promises pass to that method would implicitly be awaited when the main guest promise settles.

The main complication here is that this is a behavior that should stay within the guest (guest created promises cannot pass back through the membrane) but the orchestrator object is a host object.

[erights]

The narrator is the one host object that guests can directly invoke, where those invocations are therefore not logged. What if that method were on the narrator?

[mhofman]

The narrator currently doesn't exist and so would need to be passed around like for the orch object. We likely do not need built-in help from async-flow / the narrator in this case, and could keep this user land side.

[erights]

As you already observed, waitUntil cannot be on the orchestrator. By "user land", do you also mean purely on the guest side? How?

(My suspicion is that it is best on the narrator.)

[mhofman]

I think we should keep the narrator only for powers that cannot be implemented purely on the guest side. Here in this case we can by simply wrapping the guest function with something that does something like an await Promise.allSettled(collectedWaitUntilPromises) as a 'finally` step.

[erights]

powers that cannot be implemented purely on the guest side

ok.

In any case, this provoked me to extract #12479 from #12329 as an easily reviewable step. #12329 is now staged on #12479 .