Skip to content

fix(windows): redraw Flutter only when the captured WebView frame changed - #19

Merged
Y-PLONI merged 2 commits into
Otzaria:masterfrom
Nathaniel-260:fix/windows-texture-idle-frames
Sep 7, 2026
Merged

fix(windows): redraw Flutter only when the captured WebView frame changed#19
Y-PLONI merged 2 commits into
Otzaria:masterfrom
Nathaniel-260:fix/windows-texture-idle-frames

Conversation

@Nathaniel-260

Copy link
Copy Markdown

Connection with issue(s)

Issues are disabled on this repository, so there is no issue to resolve. Downstream context: Y-PLONI/otzaria-word-editor#53 (typing latency investigation in the Word plugin, where the host-side measurements below were made).

What

On Windows the WebView is shown through a Flutter texture: WebView2 draws into a composition visual, Windows.Graphics.Capture captures that visual, and the captured frame is copied into a DXGI shared texture that Flutter samples.

Capturing a composition visual yields a frame on every compositor tick, whether or not the WebView repainted. TextureBridge::OnFrameArrived forwarded each of them with MarkTextureFrameAvailable, so Flutter re-rasterized at the display refresh rate for as long as any WebView was on screen, and GetSurfaceDescriptor copied the frame again on the raster thread each time.

This PR:

  • compares every captured frame with the frame Flutter already has, on the GPU (one compute-shader dispatch, 4-byte readback), and only copies + notifies Flutter when a pixel differs;
  • stops copying on the raster thread (GetSurfaceDescriptor just hands out the shared handle);
  • serializes use of the shared immediate context with a mutex on GraphicsContext (several WebViews share it).

It falls back to the previous forward-every-frame behaviour when the compute shader cannot be built (feature level < 11_0, or d3dcompiler_47.dll missing — loaded on demand, no new link dependency). The frame pool stays dispatcher-bound: a free-threaded pool was tried and delivered no frames at all with the plugin's WRL event handlers.

Measurements

Otzaria, Debug build of the same commit, same machine, same 5-page document open in the same WebView; the only difference is this plugin. Flutter frames counted from the Dart VM timeline (Animator::DrawLastLayerTrees = texture-driven redraws); latency measured with real SendInput keystrokes into the WebView and screen polling of the caret line until its pixels change.

before after
Flutter redraws in 5 s, page idle, caret blinking 430 18
Flutter redraws in 5 s, page idle, caret blink paused 340 0
otzaria.exe CPU while idle (percent of one core, 10 s) 19–21% 0–4%
keystroke → pixels on screen, median (20 keys) 141 ms 153 ms
renderer paint → pixels on screen, median 135 ms 139 ms
renderer paint → pixels on screen, p90 341 ms 239 ms

Latency to the screen is unchanged within noise: the hop count (Chromium → DWM → capture → Flutter → DWM) is the same. What changes is that an idle WebView no longer keeps Flutter's raster thread and the GPU busy at 60 fps.

Verification

  • flutter build windows --debug of Otzaria against this branch; the WebView renders, updates on typing and caret blink, survives tab switches (pause/resume recreates the surface), and the dev-plugin reload.
  • Standalone MSVC compile of the three bridge translation units with the plugin's flags (/W4, _HAS_EXCEPTIONS=0), no warnings.

Testing and Review Notes

  1. Build Otzaria (Debug) against this branch and open any plugin tab.

  2. Record the Dart VM timeline for 5 s with the page idle: Animator::DrawLastLayerTrees should appear only when the page changes (caret blink, typing), not ~60 times per second.

  3. Type into the page, switch tabs away and back (pause/resume), resize the window, open two plugin tabs: the WebView must render and update in all cases.

  4. On a machine without D3D feature level 11_0 or without d3dcompiler_47.dll the log prints WebView frame compare unavailable and behaviour is the same as before this PR.

…nged

Windows.Graphics.Capture delivers a frame for the captured composition
visual on every compositor tick, whether or not the WebView repainted.
The texture bridge forwarded each one with MarkTextureFrameAvailable, so
Flutter re-rasterized at the display refresh rate for as long as any
WebView was on screen (measured in Otzaria: ~430 texture redraws per
5 s with the page idle, ~340 with even the caret blink paused), and the
raster thread copied the frame again on every one of those frames.

The GPU bridge now keeps the frame Flutter samples (`surface_`) as the
reference and compares every incoming frame against it with a small
compute shader (one dispatch, 4-byte readback; no CPU copy of pixels).
Only a frame with at least one differing pixel is copied into `surface_`
and announced to Flutter. `GetSurfaceDescriptor` no longer copies on the
raster thread; it just hands out the shared handle. With the page idle
Flutter now draws only when the caret blinks (18 redraws per 5 s) and
nothing at all when nothing changes; otzaria.exe idle CPU went from ~20%
of a core to ~4%.

The immediate context is shared by every WebView of the plugin, so the
compare/copy sequence is serialized with a mutex on GraphicsContext.
Falls back to the previous forward-every-frame behaviour when the
compute shader cannot be built (feature level below 11_0, or
d3dcompiler_47 missing; it is loaded on demand, no new link dependency).

The frame pool stays dispatcher-bound: a free-threaded pool was tried and
delivered no frames at all with this plugin's WRL event handlers.
@Y-PLONI
Y-PLONI merged commit d14e258 into Otzaria:master Sep 7, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants