Refactor: engine <-> client interface implementation

[eyelidlessness]

Closes #45, and addresses the broader set of goals behind that issue.

Initial writeup meta-commentary

If I did actually miss anything important in the writeup, it doesn't seem to have been important enough to resurface since. This definitely feels like more than enough for future archaeology.

---

I'm initially opening this as a draft with "[WIP]" in the title. It's very nearly ready and I want to make it visible now that it's so close.

TODO

There are a few last bits to wrap up:

• Prevent nested client reactive object creation
  • Map writes to node.state.clientState.children to an array of the nodes' nodeId
  • Map that same array of nodeIds back to their corresponding node objects in the client-readable node.state.currentState.children
  • Ensure JSDoc and any pertinent call site interfaces clearly explain this behavior and the reasoning for it
• Migrate remaining references to previous implementation of form instance state (and then remove it)
  • Consider whether ui-solid XFormsDetails component still has any value. If so, produce equivalent client-facing interfaces. (We will definitely be producing a subset of this—serialized representation of primary instance state; I'm just unsure if it'll be in this PR or a future iteration.)
  • Migrate ported JavaRosa tests
    • The Scenario class and related details. I am considering making this a separate package to emphasize the intent and fact that it is a client of the engine, just like any of the UI packages are.
    • Likely need to make a handful of type changes in the ported tests themselves
  • Migrate any other remaining tests pointing to EntryState or its descendants (a couple simple tests in ui-solid, quite a few in xforms-engine exercising XForms computation logic and reactivity)
  • Remove packages/xforms-engine/src/state (EntryState and its descendants), and all exports from there at the package entrypoint
  • Remove other assorted dependencies and aspects of implementation related to EntryState which will become dead code upon its removal (off the top of my head now: the @solid-primitives/set package, a couple of modules in packages/xforms-engine/src/lib/reactivity; possibly more)

I'll likely want to add even more detail after wrapping up the above TODOs, but I think I've captured many of the important details below.

But first: this is a lot. I'm writing this section last, before I open the PR... because I want to really recognize and acknowledge that this PR is huge. Not just in lines of code but the density of what's in those lines of code.

Writing everything below has re-emphasized that fact in my own mind. And I am quite certain I'm missing several important details which I'll want to update later.

I realize the hugeness of this PR is a burden, not to be taken lightly. I also think it's necessary. There are a lot of holistic and cross cutting concerns that came together in the design of #45 and everything it implies, and they would likely have been much less coherent without treating this as a singular effort.

I will say that I've also put considerable time and effort into making the commit history coherent, and trying to show the conceptual evolution from each of the foundational aspects into the set of changes as a whole. I highly recommend reviewing these changes commit by commit.

Okay, onto the details...

State: engine, client, current

To satisfy the interface goals in #45, this PR implements a tree of nodes corresponding to a form's primary instance state. Each node exposes a client-facing currentState object (and corresponding types), which is a read-only representation of the node's runtime state at any point in time.

Clients supply a state object factory on form instance initialization. This factory is used to construct a corresponding mutable object (clientState, internal to the engine) from which the client's currentState reads. Assuming a client's state object factory is backed by some client-specific implementation of subscribe-on-read reactivity, any writes performed by the engine to this internal mutable object are expected to propagate to that client's reactivity system and any pertinent reactive subscriptions.

To produce and maintain this clientState object, each node has a corresponding engineState object. This object is internally reactive—currently still using Solid, but this is expected to be an implementation detail encapsulated in the engine and opaque to clients (unless the client is also using Solid reactivity; more on that below)—and many aspects of XForms computations and semantics are built using this internal reactivity.

The engineState object itself is ultimately the source of each node's state. It is also notably derived from a specification object where:

• Properties assigned a signal or similarly shaped getter/setter tuple are mapped to an engineState getter/setter property. Writes to those signals are reactively written to the same property in clientState (and reactively readable by clients as the same property on currentState). Reading such a property is also internally reactive to facilitate pertinent XForms computations and semantics.
• Properties assigned a getter function (like a Solid memo, or any other thunk/zero argument function returning potentially reactive derived state) are mapped to a getter-only property. Any state changes which would update a reactive subscription to the provided getter function will, in turn, be propagated to the same property in clientState/currentState in the same way as signal-like state. Reading such a property is similarly internally reactive.
• Any other property value (in neither a signal-like or memo-like shape) in a node's state specification is assigned as a static, read-only value to each state object.

That's a lot of words. Here's an example:

import { reactiveObject } from 'some-ui-framework';
import { createSignal, createMemo } from 'solid-js';

const { engineState, clientState, currentState } = createSharedNodeState(node, {
  value: createSignal('foo'),
  relevant: createMemo(() => computeRelevanceSomehow(node)),
  hint: null,
}, { clientStateFactory: reactiveObject });

engineState.value // get/set property, reads internally reactive to writes
engineState.relevant // get property, reads internally reactive to `computeRelevanceSomehow(node)`
engineState.hint // null, writes to this property will fail with a `TypeError`

clientState.value // reactively written with `engineState.value` whenever it's written by the engine
clientState.relevant // reactively written with `engineState.relevant` whenever it reactively updates in the engine
clientState.hint // null, never updated

currentState.value // getter for `clientState.value`, client-reactive
currentState.relevant // getter for `clientState.relevant`, client-reactive
currentState.hint // getter for `clientState.hint`, never updated

XForms computations and semantics

With a couple of exceptions, the intent in this PR is to preserve the same form behavior as existed prior to #61 (also restoring ui-solid reactivity which was temporarily broken in that PR). As much as possible, nuances of these existing behaviors have been clarified both in terms of abstraction boundaries and in many cases much more extensive JSDoc/other commentary.

The most notable known exceptions:

• At least one repeat-specific bug I was aware of is now fixed: certain computations depending (even indirectly) on repeat position now behave correctly when repeat instances are removed. Specifically: this fixture originally taken from JavaRosa behaves as expected (when modified to use relative XPath repeat references, as is the behavior we're now targeting).

• The selection behavior with itemset filtering discussed in What is the expected behavior when a selected select value is temporarily removed? #57 is left unaddressed in this PR. In general the interaction between selection and itemset filtering should be considered incomplete and likely buggy. Given the behavior is an open question, it felt reasonable to defer implementation of this functionality rather than reproduce an opinionated behavior we may not want.

• When a field with calculate and relevant bind expressions:

  1. Is calculated (on load or update to dependencies)
  2. Has that calculated value overridden in a client (e.g. by user intervention)
  3. Becomes non-relevant
  4. Becomes relevant again

  Updated behavior: recalculate from the current state of the node's dependencies.
  Previous behavior: restore the overridden value from step 2.

  This previous behavior was inherited from Enketo (with excludeNonRelevant: true)—behavior which I had implemented because I understood it to be expected at the time. My understanding now, based on numerous conversations about pertinent spec details, is that the updated behavior is expected.

Changes to the runtime DAG

Generally speaking, this is now mostly deferred to the Solid reactivity implementation, which is itself a DAG. We no longer need to be concerned with dependency order, cycle detection, etc.

We do still need to:

• Track dependency references (for the most part as XPath reference sub-expressions as identified in a form definition). Nothing about this has changed in this PR.
• Resolve those references to their corresponding nodes in the instance state tree. This is now done dynamically, as part of the EvaluationContext interface, which each node implements.
• Establish reactive subscriptions to those resolved nodes. This was something the previous EntryState implementation did rather haphazardly without a clear understanding of how to handle node- or expression-specific semantics, and it was quite incomplete. This is now part of the SubscribableDependency interface, which each node implements. There will likely be more work here around expression-specific semantics, but it has significantly improved node-specific semantics (responsible for fixing the aforementioned repeat bug, likely fixing a slew of others we don't yet have tests for).

It's quitely likely that most or nearly all of these remaining DAG responsibilities could be handled by more conventional reactivity if we decouple XPath evaluation from the XML DOM.

For now, all graph and dependency related reactivity is now isolated to the createComputedExpression function (which depends on nodes' implementation of the aforementioned EvaluationContext and SubscribableDependency interfaces). This function builds a reactive memo, evaluating a supplied DependentExpression (introduced in #14, subject of much recent discussion when reviewing that PR), and re-evaluating the expression reactively as described above.

XForms semantics implementation details

Along with moving more of the DAG logic into the reactivity implementation, quite a lot of the implementation aspects of various XForms computations and semantics are now much closer to the way they'd be conventionally and/or idiomatically built with reactivity.

• Each node's readonly, relevant, required are basically implemented as a small wrapper around createComputedExpression.
• createValueState defers to those reactive relevant and readonly getters, and uses createComputedExpression to handle calculate where present
• createNodeLabel and createFieldHint also wrap createComputedExpression for ref/itext translations and <output>s, and produce the client-facing TextRange interface as a memo around that
• createSelectItems is a wrapper around createComputedExpression to resolve dynamic <itemset>s; select item labels use the same underlying logic supporting createNodeLabel (for both static <item>s as well as dynamic <itemset>s)

Value types

The aforementioned createValueState handles general XForms semantics around reading, writing and calculating values. It also introduces a ValueContext interface (which both current value/field node types implement), deferring to each node type to handle its own runtime value representation. For example, SelectField decodes the XForms string value to SelectItem[] and encodes those selected items back to an XForms string representation. It's easy to imagine how we'll expand this to similarly handle other arbitrary data types, regardless of their structural complexity (such as geo).

Solid-specific client details

There are a couple of special cases made for Solid (client) <-> Solid (engine) interaction:

• A separate Solid-specific xforms-engine build is produced which externalizes the solid-js package as a peer dependency, which ui-solid (or any other hypothetical Solid client) must resolve when importing the engine. This is a common practice in the Solid community, and necessary for our usage, because bundling Solid in a transitive dependency causes conflicts where Solid itself expects to have global singleton state.
  • Related to this, any downstream package which also uses Solid must ensure it consumes this Solid-specific build. For our usage with Vite, that means setting viteConfig.resolve.conditions: ['solid', ...]. This has tripped me up repeatedly (and it very well may have been the reason we had earlier issues with the SUID package[s]). We should think about how best to document this, as Solid has been gaining popularity, and the prospect of some part of the web-forms stack being consumed in a Solid stack will likely become more realistic in the foreseeable future.
• Dev mode in ui-solid currently references the xforms-engine source rather than its build, as a convenience for iterating on both packages simultaneously. It may be possible for ui-vue and other clients to be set up similarly if we think it would be beneficial. A case could also be made that we should revert this, because consuming the engine's build product will help us catch issues well ahead of any releases.
• A special case is made in the engineState -> clientState flow, which is treated as a noop if a client supplies Solid's createMutable as its reactive state factory (detected as an equality check, which would likely fail without the Solid-specific build). This exception has been reverted. I had previously considered such a special case safe, but I found it harder to predict nuances of reactive behavior in other clients with it in place.

[sadiqkhoja]

Done reviewing the src of xforms-engine. Overall this is great 👍

General comment: One pattern that makes me nervous is passing this to sub-routines in the constructors.

class Person {
   yearOfBirth: number;
   age: number;

   constructor(private name: string) {
      // this will fail because calculateAge depends on yearOfBirth
      // and method definition doesn't convey that information
      this.age = calculateAge(this);
      this.yearOfBirth = new Date();
   }
   
   calculateAge(p: Person) {
       // And if I am writing this method, I have no idea if p: Person is
       // fully initialized or not.
       return 2024 - p.yearOfBirth;
   }
}

[eyelidlessness]

General comment: One pattern that makes me nervous is passing this to sub-routines in the constructors.

I agree, and again this speaks to some of my discomfort with classes for some of this. I will say that I've tried to generally to make sure the input types to those sub-routines are narrower than the types of each node's this, to at least help make it easier to spot when an incomplete instance construction occurs. Off the top of my head, the main exception to this is in createSelectItems, which I'd be happy to revisit if that helps.

[sadiqkhoja]

I see few test files in src folder of common and xpath packages, we should move them to the test folder either in this PR or later.

[sadiqkhoja]

🚀