[docs-infra] Add CodeHighlighter component for loading demos and codeblocks

[dav-is]

Adds an isomorphic (server and client loading/parsing) and build-optimizable (via a loader) code highlighting component.

Notable Features

• allows deferring highlighting to various points: init (no defer), stream, hydration, and idle
• Supports variants and transforms. Variants are manual, transforms happen at build time. Supports TS to JS transforms automatically without the need for a .js file in the repo.
• Supports MDX or JSX (TSX) usage
• Supports webpack and turbopack.
• Supports both the app and pages router in Next.js (no hard dependency on server components like other code highlighters)

Adds 1186 unit tests, covering 70% of statements, 85% of branches, runs in 4 seconds on my machine

Note

The docs are added in a separate PR: #431

Install instructions with variants (remembers preference in local storage)

A demo of a codeblock

A demo within a demo

A demo of a demo with multiple variants

Linking to Demo Title Consistent with File Linking

Note

See the address bar

Read the docs for this component

Precompute Loader Docs

Read the rest of the added docs

Notes

We use ESM only packages for some things (https://www.npmjs.com/package/unified, https://www.npmjs.com/package/jsondiffpatch, etc) so CJS publishing is disabled (at least for now)

[mnajdova]

Agree to have this merged and iterate. It is already been dogfooded by the Joy UI team internally. Next steps:

• update Joy UI to use the latest version of the feature & validate that everything works as expected
• update Base UI's branch to use the new version and check if there are any more regressions before we put it in production.

Very big effort, kudos for the patience :)