Repository navigation
feat(ai): syntax highlighting - #5
Merged
Merged
Conversation
MIT, pure Dart and Flutter — no native code, no bundled assets, no async initialization — with highlight.js grammars ported over. Its last publish was 2024, which the roadmap got wrong; taken anyway because grammars don't rot the way UI APIs do, and vendoring the MIT grammar files is a mechanical fallback if it is ever abandoned. Rejected alternatives: syntax_highlight (serverpod) forces a Rust/cargokit native build on every consuming app through super_clipboard, plus asset bundles and an async Highlighter.initialize(), all for 15 languages; flutter_highlight + highlight are five years stale. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Roadmap 2.1. The body becomes a highlighted span tree; a fence with no language, one outside the registered set, one over 20,000 characters, or one whose grammar throws still renders as plain monospace text. Either way it stays selectable, horizontally scrollable, and keeps its copy button. Highlighting is computed from the widget's fields rather than in build: a code block is rebuilt far more often than it changes — every typewriter tick while the text after it is still arriving. The 20,000-character cap is there because a fence that is still streaming re-highlights on every frame it grows by, which makes the total cost quadratic in the block's length. The new theme parameter is a Map<String, TextStyle> keyed by highlight.js scope name, so no re_highlight type reaches the public API. It defaults to the exported kDefaultCodeBlockTheme (vs2015, whose root is #1E1E1E on #DCDCDC — exactly the background the block already had, so the only visible change is coloured tokens). Its root entry now also supplies the header's label and copy-button colour at 60% opacity, which used to be a bare constant. Deliberately still dark regardless of the ambient Theme. Languages are a curated 31-grammar set, not re_highlight's builtinAllLanguages: every grammar is a top-level `final` holding a tree of Mode constructor calls, so naming that map makes 3.1 MB of Dart source reachable with nothing to tree-shake it back out of a host app. The curated set is ~870 KB. Aliases (js, ts, py, sh, yml, c++, cs, html, …) come free — registerLanguage reads each grammar's own aliases. Also fixes code blocks not being monospaced at all on iOS, macOS and Windows: `monospace` names a real family only on Android and, because Flutter hands family names straight to the browser, on web. The style now carries a fontFamilyFallback behind it. The test config still has to register a system font under `monospace`, since `flutter test` has none of the fallback fonts either — its comment is corrected to say so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fences are constructed by ai_markdown_body.dart's private _CodeFenceBuilder, so CodeBlockView.theme would be unreachable for the main use case without this — AIMarkdownBody.codeBlockTheme and StreamingMessageView.codeBlockTheme now carry it down. The fence widget cache keys on the theme's identity: two equal-but-distinct maps then miss each other's entries, which costs reuse rather than correctness, and hashing a 40-entry map for every fence on every typewriter tick would cost more than it saved. Hosts should pass a const or hoisted map. didUpdateWidget has to bump the config generation on a theme change too. Builders are consulted at parse time, and MarkdownWidget only re-parses when data or styleSheet changes — the same reason chartLanguages and useDollarDelimitersForMath are already checked there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Changelog, README and the example's canned reply, which now carries a second fence in another language so the showcase demonstrates more than one grammar. Not a json fence — json is in kDefaultChartLanguages and would be routed to the chart parser. The roadmap entry is closed out, with a correction. 2.1 was written believing Swift had "tokenized, per-language colored highlighting" to reach parity with. It doesn't: Splash highlights exactly one language, Swift, and CodeSyntaxHighlighter.swift applies that grammar to every labelled fence regardless of its label — a ```python block gets Swift keyword colouring — while rendering unlabelled fences plain. There was no per-language target here, so this is recorded under "Where Flutter already leads" instead. Also drops 2.3's stale forward-reference to 2.1 rewriting code_block_view.dart; 'Copy code' and 'Copied!' are still hardcoded there and still 2.3's job. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
renefloor
force-pushed
the
feat/syntax-highlighting
branch
from
September 3, 2026 09:00
7b810d8 to
1222dfc
Compare
renefloor
marked this pull request as ready for review
September 3, 2026 09:03
VelikovPetar
approved these changes
Sep 9, 2026
| photo_manager: ^3.9.0 | ||
| # Syntax highlighting for `CodeBlockView`. Only a curated subset of its | ||
| # ~197 grammars is imported — referencing `builtinAllLanguages` would compile | ||
| # 3.1 MB of Dart source into every host app. See |
Contributor
There was a problem hiding this comment.
Very small remark: here we state that 3.1 MB will be included, but in code_block_view_test.dart, line 393 we mention that 2.6 MB is added.
Both figures quoted for the curated-grammar decision were wrong, and the two rulers disagreed with each other — which is what Petar spotted on #5. - `builtinAllLanguages` has 194 keys (192 builtin + 2 community), not 197. That count came from directory entries, which include the `all.dart` barrel and a nested `languages/lib/` of shared helpers. - The reachable source is 2.7 MB (2,739,368 B), not 3.1 MB. The latter was `du -sh` — filesystem block allocation rather than source bytes, and a different ruler from the `~870 KB` in the same sentence, which is a true byte sum. The 2.6 MB in the test comment was closer but missed the nested helpers the grammars import. The ~3.1x ratio the argument rests on is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Highlighting was worth having, but not worth making every host pay for.
Measured on a release web build of the example app:
the package's own lib/ 198 KB of source
the curated 31-grammar set 872 KB (4.4x the package)
re_highlight's 194 grammars 2.7 MB
compiled cost of the curated set +636 KB of JS, +56 KB gzipped
73% of that grammar source survives into the bundle, because each grammar
is a top-level `final` holding a tree of constructor calls and nothing
tree-shakes an unused one back out. `swift` alone was 388 KB of the 872 KB.
So highlighting becomes a seam, exactly parallel to `mathBuilder`:
typedef CodeHighlighter =
TextSpan? Function(String code, String language, TextStyle baseStyle);
wired through `CodeBlockView.highlighter`, `AIMarkdownBody.codeHighlighter`
and `StreamingMessageView.codeHighlighter`. The package recognises fences,
frames them, caps them and renders the spans; it holds no grammars and no
highlighting dependency. `re_highlight` moves to the example, and
`lib/src/highlight/code_languages.dart` becomes
`example/lib/code_highlighter.dart` — all 31 grammars, ready to copy and
trim. A host that renders no code now pays nothing.
`CodeBlockView.theme` gives way to `backgroundColor`/`foregroundColor`. The
scope map only made sense while the package owned the tokenizer: it made
`root` a magic key, exposed ~25 `TextStyle` fields of which two were read,
and could be mutated behind the widget's back. This also drops the
incidental #D4D4D4 -> #DCDCDC foreground drift, since the default is a
plain constant again.
Two details worth keeping:
- The fence cache deliberately does not key on the highlighter. Hosts pass
a closure whose identity changes every build, so keying on it would mean
a fresh entry per fence per build and the cache would evict itself
continuously. `_CodeFenceBuilder` reads it through a closure instead,
with the same trade `_MathElementBuilder` documents.
- The `span.toPlainText() != code` check stays, and matters more now that
it guards third-party code. `re_highlight` runs in safe mode, so a
mid-parse grammar failure comes back on the result rather than as a
throw, carrying only the tokens produced before it; a try/catch never
sees it and rendering it drops the tail of the block or all of it. It
costs ~1% of the tokenizing it follows.
CI goldens still need regenerating via the update_goldens workflow: the
fixture changed, and the mixed-content golden's python fence now renders
unhighlighted because that golden supplies no highlighter.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Highlighting a prefix is linear, but a growing fence produces a longer prefix on every typewriter tick, so tokenizing each one makes the total quadratic in the block's length. Measured on a 1839-character Dart fence streamed through `StreamingMessageView` with a counting highlighter: 1487 calls, 1,372,042 characters tokenized -- 746x the block which is roughly a second of CPU on a fast desktop for one code block, and several times that on a phone. The 20,000-character cap did not help: it bounds a single pass, not how many passes there are, and a single pass at the cap already costs ~16 ms, a whole frame. `CodeBlockView` now re-highlights only once the code has gained 64 characters, plus one settling pass 120 ms after the last change so a stream that stops mid-threshold still ends up fully colored. Same fence: 33 calls, 27,896 characters tokenized -- 15x the block, 45x less work The subtle part is what to show in between. A stale span covers only the prefix it was built from, so rendering it alone would show a truncated block -- the exact failure the `toPlainText` check was added to catch. So the gained characters are appended as one plain run instead: the code stays whole, the already-visible part keeps its colors, and composing costs two children per build rather than a re-tokenize. The visible effect is up to about a line of uncolored text trailing the newest characters, settling as it arrives, the way coloring lags the cursor in an editor. Code that does not extend what was last highlighted -- a regenerated or edited message, which can be shorter -- invalidates outright rather than throttling, since the pending-tail arithmetic assumes an extension. Tests cover the small-gain reuse, the immediate pass past the threshold, the settling pass, the delay restarting while the fence still grows, non-extending replacement, disposal with a timer armed, and the property that matters most: no character is ever lost, asserted across the whole growth range. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The comments around highlighting had grown into essays, restating benchmark figures that belong in the changelog. Same reasoning, roughly half the lines. Also documents, and asserts, the one invariant the throttle depends on: `_highlightedCode` has a single writer that always sets it to the whole of `code`, so a pending tail can only arise in `didUpdateWidget`'s throttle branch, which arms the settle timer alongside it. A tail left unarmed would stay plain for the block's whole life and look exactly like a highlighter declining the language, so `_displaySpan` now asserts the two stay in step. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Submit a pull request
Linear: FLU-540
CLA
Description of the pull request
This improves the code syntax highlighting.
Screenshots / Videos
Simulator.Screen.Recording.-.iPhone.17.Pro.Max.-.2026-09-03.at.10.56.00.mov