Package Compatibility in RoC Engine

[andy-haynes]

Motivation

NPM package support in RoC Engine is almost entirely contingent on a package's compatibility with the Engine's iframe-based container architecture. Accordingly, pure JS packages (lodash, uuid) function without issue, while React UI libraries (@chakra-ui/react, @radix-ui) tend not to function at all.

In exploring compatibility across the packages in the latter category, two prominent compatibility-breaking characteristics emerged:

• Broken JS bundles prevent iframe Components from being rendered.
  • Requires a fix in the upstream bundle provider.
  • Impacts: @radix-ui, @mui/material/*
• Non-interactive and unstyled elements in the outer DOM are indicative of package functionality (e.g. animations, styling) not being applied correctly.
  • Requires some combination of synchronizing styles between iframe and outer DOM and implementing missing React functionality (e.g. Portals).
  • Impacts: @chakra-ui/react, react-bootstrap, @nextui-org

These behaviors are described in detail below, along with their underlying causes and potential remediations.

JS Bundling

Context

RoC Engine relies on esm.sh, a CDN for NPM packages, to source external dependencies. Package CDNs like esm.sh or jsdelivr are an essential source of bundles, as browser-side bundling solutions are largely experimental and a server-based approach introduces new complexity while negatively impacting decentralization.

esm.sh is unique in that it supports URL parameters to rewrite import specifiers, the crucial component in Preact compatibility. This is what lets us swap out react and react-dom references with preact and preact/compat, enabling use of Preact within iframe containers, which in turn is necessary to facilitate Component lifecycle across containers. Without this functionality, packages with React dependencies could not be rendered.

While many NPM packages (including React-based packages) work with this method, testing revealed that several popular UI libraries have broken bundles in esm.sh. The dynamic nature of esm.sh makes it possible for directly-imported packages to have broken nested dependencies, which cannot be determined without attempting to resolve the package's dependencies via import. When this resolution fails, container initialization does not complete and Components in that container (along with its descendants) cannot be rendered.

Findings

Since different packages exhibit different bundle failures (e.g. bad import paths, transpilation issues), it's difficult to draw general conclusions from this behavior. Without a deeper examination of individual packages and a more thorough understanding of exactly how esm.sh constructs bundles, it's not clear whether this is a systemic issue.

Ostensibly there is nothing about these packages that makes them fundamentally incompatible with Preact. If that is the case, then some combination of the following should be possible:

• esm.sh can update its bundling/import rewrite logic to produce valid bundles
• the affected package can be updated to produce valid bundles
• we can pre-bundle affected packages and host/inject them ourselves
  • this implies that the cause is discovered but cannot be addressed by one of the above options

Non-Interactive/Unstyled Components

Context

Rendering in RoC Engine works by sending updates to the outer application when Preact nodes are rendered. This covers the Component lifecycle (hooks, rendering from props updates), but does not capture "side effects", i.e. changes to the iframe document made outside the scope of Component lifecycle are not reflected in the outer DOM. Similarly, usage of the window object in package code will not reflect that of the top window in the outer application.

In addition, coupling rendering to the Component lifecycle fails to implement React functionality outside the conventional render flow. React Portals in particular appear to be impacted by this, which are often used to facilitate advanced functionality prevalent in React UI libraries.

Findings

While this exploration has uncovered deficiencies in the Engine's rendering, they can largely be shored up by extending the current implementation to cover a wider scope of interactions. It's still not clear exactly to what extent parity could be achieved, given the limitations of the container architecture. However, these deficiencies can be addressed systematically to yield progressively better compatibility. Based on behaviors observed across packages, priorities should be in roughly the following order to maximize impact:

1. Synchronizing container documents with the outer DOM should address most of the issues with missing styles.
   1. An initial implementation could likely augment the current render lifecycle, though there may be advanced cases not captured.
   2. Doing this in a secure way will be challenging as it provides a path for containers to alter CSS in the outer DOM.
   3. Capturing document styles is expected to fix many of the issues around compatibility with simple styled Components (e.g. <Box />)
2. Implementing React Portal in the outer DOM is likely necessary to support more dynamic package usage.
   1. Given the success of the recent ref implementation, it's likely that RoC Engine could support this functionality.
   2. A working Portal implementation should unlock more dynamic functionality provided by React UI Components (e.g. <Tooltip />).
3. Proxies for properties and methods on window or document may be required to shore up other kinds of functionality.
   1. The necessity and viability of this are still big unknowns, and would likely be handled on a case-by-case basis.
   2. It may be viable to proxy asynchronous methods, and even synchronous request-based methods such as requestAnimationFrame.

Conclusions

The most promising path toward supporting a subset of popular libraries (e.g. @chakra-ui/react) would be to focus on widening the scope of captured DOM changes beyond the Component render lifecycle. This should enable most of the styled Components supported by compatible packages. Implementing React Portal, likely a more complex task, should enable much of the remaining functionality for advanced DOM interactions. These tasks can be addressed in parallel and should enable a substantial subset of Components provided by compatible React UI libraries.

Packages found to be incompatible due to broken bundles likely need to be addressed individually. In some cases, this may be a fix on the esm.sh side while in others it may be an issue in the package's bundling configuration. If the source of a broken bundle can be determined, then the issue can likely be addressed by the upstream provider. However, there may be other options available (e.g. forking, injecting pre-bundled packages) if an upstream fix is not viable. Ultimately this is a hard boundary to compatibility and touches on the tenuous nature of RoC Engine package imports.

cc @mpeterdev @thisisjoshford @charleslavon