Extend Phlex algorithm lifecycle to include pre- and post-execution phases "initialize" and "finalize".

[brettviren]

I'd like Phlex to consider to add support for calling two methods on an algorithm: initialize() and finalize() before and after the DFP graph execution (just after start and just before end of job) and from the main thread.

A templated compile-time check can be used so that algorithms that are simple functions or classes that lack one or both of these methods could be exempt from attempting to call their non-existent methods and without the need for inheritance.

This would help support Wire-Cell Toolkit and Geant4 in Phlex and probably other future integrations.

Are there any barriers to adding these two algorithm entry points?

[knoepfel]

Are there any barriers to adding these two algorithm entry points?

The barriers are conceptual: adding these entry points would mean that graph execution would need to be informed by not only data dependencies and the available concurrency, but it would also depend on algorithms that are used (presumably) only for establishing state...making some of the dependencies implicit. We are very hesitant to go this route until we can understand specifically what necessitates it. We have not yet seen a compelling case to add such hooks (although they have been considered).

Some options you might consider are below. Those options assume that the purpose of the "initialize" and "finalize" are to handle state used by the algorithm¹. Happy to follow up on anything.

Phlex support for functions bound to stateful objects

Functions bound to stateful objects can be registered with the framework. See here for an example of calling m.make<SomeClass>(ctor args...) where a framework-owned object of type SomeClass is created during registration. The algorithm registered by that code is the member function &even_fibonacci_numbers::only_even bound to the framework-owned object. That framework-owned object is then destroyed just before the program ends. It may be possible for you to get what you need by using SomeClass's constructor to serve as the "initialize", and SomeClass's destructor to serve as the "finalize".

This approach means no changes to Phlex proper, and it keeps stateful classes local to the Phlex module.

Recasting the "initialize" and "finalize" as data dependencies

Depending on what the initialize and finalize are for, it may be possible to introduce algorithms that prepare the state and pass that state explicitly as an in-memory-only data product to a downstream algorithm that requires it. This keeps the data dependencies explicit, thus making it easier to understand the flow of the program. How a "finalize" is implemented would depend on what it is for.

Resources

We have a reasonably mature design for Phlex resources. These can be helpful when needing to coordinate the execution of algorithms that depend on a shared resource that may not be thread-safe. Depending on what you need to do with the "initialize" and "finalize", these may be reasonable approaches.

Footnotes

1. In Phlex parlance, an algorithm is always a callable entity, generally a free function, a member function bound to an object, or a closure object generated from a lambda expression. ↩

[brettviren]

Thanks Kyle. I'll look more into m.make<>().

Just to be clear, I'm explicitly wanting these new methods to NOT be intertwined by TBB flow_graph execution.

0. Enter main()
1. call initialize() from the main thread on all that define it.
2. run flow_graph
3. flow_graph is truly and forever finished executing
4. call finalize() from the main thread on all that define it.
5. exit() from main().

I guess you also do not need to know about the flow graph, per se, just about objects that will go on to serve as node instances.

Here are some use cases:

• WCT "configurable" components are written with a guarantee that configure() is called serially in a single-threaded context. When wire-cell CLI and (I think) art provides main() this also means main-thread context. I do not yet know if any components need specifically main-thread or just single-thread. I currently kludge serial single-thread by calling WCT's configure phase the first time a Phlex algorithm bearing a WCT graph is executed. This requires a mutex as multiple WCT-bearing Phlex nodes can be executing concurrently.

• WCT "finalizable" components expect to have their terminate() called serially and prior to process tear down. It may actually be okay to soften this call the terminate phase from my WCT/Phlex node destructor (or the type given to m.make) but since existing WCT components always from with terminate() called prior to main() exit, some may be sensitive.

• Geant4 has thread-local variables and so must "live" entirely inside a single thread for its lifetime (ignoring its own internal MT'ing features). Integrating Geant4 in Phlex can't just use the first-call trick but needs to start Geant4 in its own (non-TBB pooled) thread. This also requires careful in-thread Geant4 tear down to avoid some main-thread G4 singleton destructors from destroying objects that were constructed in other-thread. This so far works for my G4 integration of edep-sim which uses G4RunManager but SHiP tried this trick and still has to leak G4 memory to avoid end-of-job crash (I suspect that may be due to SHiP using G4MTRunManager or my use just being lucky so far). With a guaranteed main-thread entry for construction and destruction of Geant4, I think these problems can be cleared up.

[olantwin]

From reading edep-sim, the run manager seems to be leaker there as well, so this seems to affect also the single-threaded G4 run manager, not just the MT one we use in SHiP.

Two points from my side:

• For me so far (ab)using constructors and destructors has been sufficient.
• The Geant4 threading model is hard to work with. Maybe some accommodations on the Phlex side can help, but I would like to also work with Geant4 to fix some of the pain points (and frequently frankly bugs) there (e.g. https://bugzilla-geant4.kek.jp/show_bug.cgi?id=2747, which they already fixed). Ideally, it would be good to have interfaces to use Geant4 without a run manager (or a custom one) or properly working TBB task manager with a shared pool.

[knoepfel]

We believe we have a solution for the Geant4 threading model using Phlex resources, which are still under development (see #730).

[brettviren]

Another problem I just hit with the lack of a main-thread serial "configure" phase of Phlex's life cycle: calling go-jsonnet to parse WCT configuration file from each WCT-graph-as-Phlex-node crashes the go runtime ("fatal error: traceback did not unwind completely"). The immediate "fix" is to only run Phlex with a single thread. Next work-around would be that wcphlex nodes can add a mutex around Jsonnet parsing. But, this is a general class of problem.

I really think that Phlex need to support a single, post-construction, pre-execution, main-thread Algorithm class entry point.

I had opus analyze Phlex 0.3.2 to come up with a minimally invasive way to execute an initalize() method if the node type has one defined. Below I paste the core bits of the output. It generally matches what I expected is possible but gives specific Phlex bits so describes better than I probably could. Please let me know if providing this kind of generated content is objectionable.

Generated by Opus 4.8

Proposal: opt-in main-thread initialize() for Phlex nodes

Grounded in phlex v0.3.2 (reference/phlex, tag v0.3.2, commit 29b8b3a).

Goal: if a node's underlying algorithm type provides void initialize(), Phlex
calls it exactly once, on the main thread, after all nodes are constructed and
before graph execution begins. Otherwise nothing changes.

Where the typed object still exists (detection site)

Phlex erases the user's algorithm into a std::function very early, in
phlex/metaprogramming/delegate.hpp. algorithm_bits<Bound, Algorithm>
receives the concrete bound object and immediately erases it:

// delegate.hpp  (v0.3.2)
template <typename Bound, typename Algorithm>
class algorithm_bits {
  using bound_type = Bound;                 // = std::function<...>
  template <typename T>
  algorithm_bits(T object, Algorithm algorithm)      // T = std::shared_ptr<U>
    : bound_{delegate(std::move(object), std::move(algorithm))} {}
  auto release_algorithm() { return std::move(bound_); }
  Bound bound_;
};

The three delegate() overloads show the two categories that must both stay
legal:

• bare function / lambda → delegate(std::shared_ptr<void_tag>, ...); the
  bound object is void_tag — there is no instance, so no initialize().
• class method node (g.make<MyClass>().transform(&MyClass::op, ...)) →
  delegate(std::shared_ptr<T>, R (T::*)(Args...)); here a real T instance
  exists and MAY define initialize().

So algorithm_bits's constructor is the single point where a typed
std::shared_ptr<U> is in hand before erasure. That is where we detect and
capture.

(a) Detection trait

// delegate.hpp
template <typename U>
concept has_initialize = requires(U& u) { u.initialize(); };

Detection is on U, the element type of the bound object's shared_ptr. For
U = void_tag (bare functions/lambdas) the concept is false, so those nodes are
unaffected — this is the backward-compatibility guarantee, enforced by the type
system, not by convention.

(b) Capture the initializer alongside the algorithm

Have algorithm_bits build a std::function<void()> init thunk that shares
ownership of the same object (copy the shared_ptr before it is moved into
delegate). Empty thunk == "no initializer".

// delegate.hpp — helpers
inline std::function<void()> make_initializer(std::shared_ptr<void_tag> const&) {
  return {};                                   // bare function / lambda: no-op
}
template <typename U>
std::function<void()> make_initializer(std::shared_ptr<U> const& obj) {
  if constexpr (has_initialize<U>)
    return [obj] { obj->initialize(); };        // shares ownership; runs later
  else
    return {};                                  // class without initialize(): no-op
}

// algorithm_bits: add one member + one accessor; init it BEFORE bound_.
std::function<void()> initializer_;
template <typename T>
algorithm_bits(T object, Algorithm algorithm)
  : initializer_{make_initializer(object)},     // copies shared_ptr
    bound_{delegate(std::move(object), std::move(algorithm))} {}
auto release_initializer() { return std::move(initializer_); }

(Declare initializer_ before bound_ so it is constructed while object is
still intact; make_initializer takes it by const-ref and only copies the
shared_ptr.)

(c) Carry the thunk to the node, run it on the main thread

The node object is the right owner: it is exactly the set of things the
framework already tracks, and only successfully-registered nodes get one.

AlgorithmBits alg already flows into every node constructor, where the body is
extracted via alg.release_algorithm() (see declared_transform.hpp:84,
declared_observer.hpp:68, declared_predicate.hpp:72, declared_fold.hpp:97,
and the provider path registration_api.hpp:146). At each of those sites also
pull alg.release_initializer() and store it.

The consumer family (transform / observer / predicate / fold / unfold / output)
shares the base products_consumer (products_consumer.hpp:20), and the
framework already enumerates them via node_catalog::consumers()
(node_catalog.hpp:34). Add there:

// products_consumer (base)
void set_initializer(std::function<void()> f) { initializer_ = std::move(f); }
void initialize() { if (initializer_) initializer_(); }   // called once, main thread
std::function<void()> initializer_;

provider_node (provider_node.hpp:20) and source (source.hpp) are the two
node kinds outside that base; give each the same two-line addition and a catalog
accessor (or a tiny shared mix-in if preferred). These are the node types
register_source / sources produce, which the WCT shape executors use, so they
must be covered too.

Invoke on the main thread between construction and execution — a new step in
framework_graph::execute(), which itself runs on the caller's (main) thread:

// framework_graph.cpp
void framework_graph::execute() {
  ...
  finalize();          // main thread: builds all edges (existing)
  initialize_nodes();  // NEW, main thread: run each node's initializer once
  run();               // activates source; dispatches to TBB workers (existing)
}

void framework_graph::initialize_nodes() {
  for (auto* c : nodes_.consumers())            c->initialize();
  for (auto& p : nodes_.providers | std::views::values) p->initialize();
  for (auto& s : nodes_.sources   | std::views::values) s->initialize();
}

Ordering note: initializers run sequentially in an unspecified order, before any
worker task. If cross-node init ordering ever matters that is a separate
feature; the one-time, main-thread, pre-execution guarantee is what solves the
cases in #764 and the go-jsonnet crash.

(d) Backward compatibility

• Bare functions and lambdas: bound object is void_tag, has_initialize is
  false, make_initializer returns an empty thunk, initialize() is a no-op.
  No source change required for any existing node.
• Class nodes without initialize(): has_initialize<U> false → same no-op.
• Class nodes WITH initialize(): opt in simply by defining the method; no new
  registration syntax, no macro, no base class to inherit.
• Everything is resolved at compile time; the only runtime cost is one
  if (initializer_) per node at startup.

Why this fits Phlex's current design

• No new public registration API; opt-in is "define void initialize()".
• The detection lives exactly at the existing erasure boundary
  (algorithm_bits); nothing upstream of it needs the concrete type.
• The invocation point is a natural sibling of the existing finalize() step in
  execute(), on the same (main) thread, using structures the framework already
  holds (node_catalog).
• Total surface: delegate.hpp (trait + thunk + one member/accessor),
  products_consumer + provider_node + source (two lines + accessor each),
  the node constructors (one line each to pull the thunk), and one new
  framework_graph step.

Downstream payoff (this repo)

devel/wire-cell-phlex/modules/register_shapes.hpp builds
FunctionExecutor/SinkExecutor/SourceExecutor as std::shared_ptr captured
in a lambda. Refactor so the executor is the node object (a class method node)
and move its one-time wct_config Jsonnet evaluation (the go-jsonnet cgo call)
into executor->initialize(). It then runs once, on the main thread, before any
TBB worker — eliminating the concurrent-cgo crash and letting the split smoke
tests run at full -j.

[brettviren]

It looks like setting env var GODEBUG=asyncpreemptoff=1 makes go-jsonnet's go runtime robust against being called from different threads. This is after 15 successful test runs with it set and no crash while seeing 1 crash after 6 tests with it unset. So this latest problem appears to have a work-around.

I still think Phlex should provide deterministic, main-thread entries to Algorithms.

[brettviren]

This lack of main-thread initialization continues to cause me serious headaches.

I now get segfault due to races between mutex-protected initialization that I had to wedge in to first-time execution of a Phlex algorithm and the non-mutex protected subsequent executions.

I only went down this mutex hack to try to satisfy the ability to inject a Phlex "resource" into a WCT "service like" component.

This same workaround was actually the cause of the go-jsonnet problem.

I understand I can maybe use a shared mutex to perpetuate this ugly hack.

Working around Phlex's lack of flexibility in this matter is very frustrating.

[knoepfel]

@brettviren, oof...it looks like you're dealing with multiple issues.

This lack of main-thread initialization continues to cause me serious headaches.

I now get segfault due to races between mutex-protected initialization that I had to wedge in to first-time execution of a Phlex algorithm and the non-mutex protected subsequent executions.

Have you tried the m.make<T>(...) approach? It should satisfy steps 0–5 that you mention above, although the "spelling" is a little different wrt. your steps. Specifically, you should be able to create a wrapper like:

class AlgWrapper {
public:
  // The following special member functions are invoked on the main thread
  AlgWrapper(/* ctor args */) : alg_{/* ctor args */} { alg_.initialize(); }
 ~AlgWrapper() { alg_.finalize(); }

  auto execute(...) { alg_.execute(...); }  // invoked on TBB worker threads during graph processing

private:
  AlgType alg_;
};

and then register it via:

PHLEX_REGISTER_ALGORITHMS(m, config)
{
  m.make<AlgWrapper>(/* ctor args */)
    .transform("some name", &AlgWrapper::execute, ...)
    .input_family(...);
}

I'd appreciate hearing from you whether this approach works; and if doesn't, what specific problems you encountered. Assuming it works, then we can consider ways to make it simpler, like introducing a template that could invoke initialize() and finalize() as appropriate. If it does not work, then adding separate invocations of initialize() and/or finalize() is unlikely to help as the call sequence would be equivalent to the above (unless I'm missing something).

I only went down this mutex hack to try to satisfy the ability to inject a Phlex "resource" into a WCT "service like" component.

This same workaround was actually the cause of the go-jsonnet problem.

I understand I can maybe use a shared mutex to perpetuate this ugly hack.

Basic support for resources is intended for Phlex v0.4 (due at the end of September 2026). It does not surprise me that workarounds may lead to such issues.

Working around Phlex's lack of flexibility in this matter is very frustrating.

Thank you for your patience. We're working on many aspects of Phlex right now, and I hope that Phlex v0.4 will be able to address at least some of the issues you've encountered. Stay tuned.

[brettviren]

Hi @knoepfel

it looks like you're dealing with multiple issues.

Yes. I think I can disentangle somewhat now. The original title of this discussion is no longer correctly describing the problem that remains.

A few new understandings:

1. The m.make<> with AlgWrapper is essentially identical to what I am using when calling m.transform() with a lambda that captures a shared pointer to my underlying algorithm body class.

But, the m.make<> form has two possible benefits:

• It bakes initialize/execute/finalize into named methods while the lambda + shared pointer init/fini implicit in the ctor/dtor of the body class.
• It gives Phlex an actual type instead of a generic lambda type. Phlex doesn't (afaics) care about having this type right now but it could be a useful thing in the future.

2. Importantly, both have the same lifecycle w.r.t. which threads run which phases and it is exactly what I was calling for! Both call init/fini on the MAIN thread while exec is called on a variety of threads.

I confirmed this with some actual code that uses both m.make<> and m.transform() forms and different Phlex graph toploglies and number of threads.

So, this actually solves my initial problem. Running WCT as "introverted" Phlex algorithms is fine.

The problem comes when I try to add some extroversion so that WCT can implement a WCT service using information from a Phlex resource data product.

WCT assumes services are all ready to use after the "initialize" phase, but Phlex resource products are only available at each "execute" phase.

The initial solution was to delay all WCT initialization to the first execute call of each WCT island.

But, that fell apart once I scaled tests up to multiple events, multiple WCT islands and multiple threads. The mutex that I placed around the WCT initialization was not enough.

This is due to execute-time access to WCT's "named factory" was not mutex-protected so some WCT islands got there while others were inside the initialize mutex. A shared lock for the execution phase should fix this, (and perhaps fixing WCT NamedFactory to be thread safe for read-only queries). But, at that point I had given up and backed out support for Phlex resources in hopes that Phlex 0.4.0 might help something.

But, perhaps this relative scheduling problem is more endemic. If so, I can bring back the delayed first-execute and add the shared mutex.

[brettviren]

One more wrinkle before I forget it: the solution I describe at the end: to do WCT initialization from the first Phlex execution (even with shared mutex added) will initialize WCT on a non-MAIN thread. This allows the class of thread problems that Geant4 tickles. Ie, it will construct objects on a non-MAIN thread and destruct them on the MAIN thread. I don't know yet if this will tickle problems or not.

[knoepfel]

One more wrinkle before I forget it: the solution I describe at the end: to do WCT initialization from the first Phlex execution (even with shared mutex added) will initialize WCT on a non-MAIN thread. This allows the class of thread problems that Geant4 tickles. Ie, it will construct objects on a non-MAIN thread and destruct them on the MAIN thread. I don't know yet if this will tickle problems or not.

Correct. We have started work supporting tasks pinned to specific threads (#730), which we hope to have ready for Phlex v0.4.