Add native iPhone and Android WebGPU hosts and mobile WinUI support#29
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9d06f7b7e8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| { | ||
| _silkWindow.IsVisible = false; | ||
| } | ||
| NotifyHostVisibilityChanged(false); |
There was a problem hiding this comment.
When a desktop window is hidden, this new notification sets Window.Visible to false, but the Activate() path for an existing _silkWindow only sets IsVisible = true and focuses the native window; it never calls NotifyHostVisibilityChanged(true). Any app that hides and later re-activates a window will therefore keep the new WinUI-shaped Visible property false and miss the corresponding VisibilityChanged(true) event even though the native window is shown again.
Useful? React with 👍 / 👎.
| textAlignment: wrapping != TextWrapping.NoWrap && FlowDirection == FlowDirection.RightToLeft | ||
| ? ProGPU.Text.TextAlignment.Right | ||
| : ProGPU.Text.TextAlignment.Left); |
There was a problem hiding this comment.
With the default CellTextWrapping = NoWrap, this condition now forces every RTL DataGrid cell to TextAlignment.Left; before this change, RTL rows were right-aligned regardless of wrapping. In any right-to-left grid that does not opt into wrapping, cell text regresses to the wrong side of the cell, so the no-wrap drawing path needs to keep RTL alignment while avoiding the oversized no-wrap bounds issue.
Useful? React with 👍 / 👎.
wieslawsoltes
changed the title
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a9096cfe2e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| { | ||
| safeArea = NormalizeInsets(safeArea); | ||
| inputPaneOccludedRect = NormalizeOccludedRect(inputPaneOccludedRect); | ||
| if (_safeAreaInsets.Equals(safeArea) && _inputPaneOccludedRect.Equals(inputPaneOccludedRect)) return; |
There was a problem hiding this comment.
When the soft keyboard is already visible and focus moves to another text input, the mobile hosts keep reporting the same safe-area and occluded-rect values, so this early return skips the only call to EnsureFocusedElementInView(). A newly focused editor that is under the existing keyboard will therefore remain hidden until the keyboard geometry changes; the unchanged-insets path still needs to attempt the focused-element reveal while the input pane is visible.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 01e24c92f3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| FindSingleReplacement(_lastText, current, out int start, out int removedLength, out string inserted); | ||
| int selectionStart = Math.Max(0, _editText.SelectionStart); | ||
| int selectionEnd = Math.Max(selectionStart, _editText.SelectionEnd); | ||
| InputSystem.InjectTextReplacement( |
There was a problem hiding this comment.
When a mobile PasswordBox is focused, this non-composition edit path sends TextInputEventKind.ReplaceText for ordinary typing, but PasswordBox.ITextInputClient.OnTextInput only handles InsertText, delete, composition, and paste. Since PasswordBox.GetTextInputOptions() also exposes an empty native document, every normal Android software-keyboard character is dropped; the same replacement path exists in the iOS bridge, so password entry is unusable on native mobile until PasswordBox handles replacement/selection edits or the bridges special-case password clients.
Useful? React with 👍 / 👎.
| public override void DidBecomeActive(UIScene scene) => _host?.Resume(); | ||
|
|
||
| public override void WillResignActive(UIScene scene) => _host?.Pause(); | ||
|
|
||
| public override void DidEnterBackground(UIScene scene) => _host?.Pause(); |
There was a problem hiding this comment.
In the iOS scene lifecycle, these callbacks only pause/resume rendering and never call Application.Current.NotifyHostSuspendingAsync, NotifyHostEnteredBackgroundAsync, NotifyHostResuming, or NotifyHostLeavingBackgroundAsync (repo-wide, only the Android host invokes those new hooks). Apps that rely on the WinUI-shaped lifecycle events to save state or release resources therefore receive no Suspending/background/resume notifications when an iPhone app is backgrounded or reactivated.
Useful? React with 👍 / 👎.
| } | ||
|
|
||
| _controller.View!.LayoutIfNeeded(); | ||
| MetalViewMetrics metrics = _renderView.Metrics; |
There was a problem hiding this comment.
The iOS host reads native view metrics here but never publishes them through DisplayInformation.NotifyHostMetricsChanged (the only repo call is in the Android host). As a result, DisplayInformation.GetForCurrentView() on iPhone keeps the default landscape/96-DPI/zero-pixel values even after activation, so orientation- or DPI-adaptive UI sees incorrect display state on the new native iOS host.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9b208cad7e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| removedLength, | ||
| selectionStart, | ||
| selectionEnd - selectionStart); | ||
| _lastText = current; |
There was a problem hiding this comment.
When the focused client rejects or truncates this replacement, this still advances _lastText to the native current text. For example, a mobile RichEditBox with MaxLength returns without inserting once the limit is reached, so the hidden Android/iOS editor keeps extra characters; subsequent backspace/delete ranges are then computed past the ProGPU text and get clamped to no-ops until the user deletes the unaccepted native overflow. The bridge needs to resync from the focused client's TextInputOptions after dispatch instead of assuming every replacement was applied verbatim.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ab98041204
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| var row = (int)((y - _headerHeight + ScrollOffset) / _rowHeight); | ||
| float offset = y - _headerHeight + ScrollOffset; | ||
| int row = IsVariableRowHeight | ||
| ? EnsureRowSizeIndex().GetIndexAtOffset(offset) |
There was a problem hiding this comment.
When RowHeight is NaN, GetIndexAtOffset clamps offsets at or beyond the total size to Count - 1. Since GetRowIndexAt feeds any body y into it and then only checks that the returned row is in range, clicking or hovering the blank area below the final row (for example in a short grid or when scrolled to the bottom) incorrectly targets the last item and can select/edit it. Guard offset >= TotalBodyHeight before the variable-height lookup so empty grid space remains non-interactive.
Useful? React with 👍 / 👎.
| checked((ulong)Math.Max(0L, e.EventTime) * 1_000UL), | ||
| WheelDeltaX: deltaX, | ||
| WheelDeltaY: deltaY, | ||
| IsPreciseWheel: precise, |
There was a problem hiding this comment.
For Android 14+ touchpad pinch events that report GesturePinchScaleFactor but no scroll-distance axes, scroll.IsPrecise remains false, so the ctrl-wheel event injected here is not marked precise. Consumers such as DxfCanvasControl.OnPointerWheelChanged only use the exact 120 * ln(scale) zoom path when IsPreciseScrolling is true, so Android pinch zoom falls back to a single stepped mouse-wheel zoom instead of the actual pinch scale. Set precise in the pinch branch when the delta is rewritten.
Useful? React with 👍 / 👎.
| _editText.ImeOptions = MapImeAction(options.EnterKeyHint, options.AcceptsReturn); | ||
| _editText.SetSingleLine(!options.AcceptsReturn); | ||
| PositionNativeEditor(options); | ||
| SynchronizeNativeDocument(options); |
There was a problem hiding this comment.
Syncing the hidden native editor only from the focus-change path leaves its text/selection stale when the user taps or drags to move the caret inside an already-focused TextBox/RichEditBox (the focus setter returns without raising FocusChanged). The next IME edit is then diffed against the old native selection and inserted at the previous caret position, so ordinary mobile editing after repositioning the caret corrupts placement. The bridge needs an explicit resync from the focused ITextInputClient when the focused editor's selection/text changes, not just when focus changes.
Useful? React with 👍 / 👎.
1 participant
Summary
This change adds experimental native iPhone and Android hosts for ProGPU and runs the existing shared WinUI gallery directly on WebGPU over Metal and Vulkan. It also completes the mobile-facing WinUI surface needed by those hosts: lifecycle events, safe-area and keyboard occlusion, native text input and file/folder pickers, precise trackpad input, indirect-pointer drag/drop, wrapped variable-height DataGrid rows, reusable variable-size virtualization, and WinUI-shaped Page, Frame, panel, navigation, display, back-navigation, and deferral APIs.
Both mobile routes reuse the same retained renderer, scene compiler, controls, shaders, and sample pages as desktop and browser. iOS presents to a physical-pixel
CAMetalLayerthrough a statically linked, Metal-onlywgpu-nativeXCFramework. Android presents to a physical-pixelANativeWindow/SurfaceViewthrough the same Silk.NET WebGPU ABI and wgpu-native's Vulkan backend. Neither route uses a web view, JavaScript command serialization, MAUI, Uno, a second canvas, or a readback path.Why
ProGPU previously had desktop and browser hosts but no direct native Apple mobile surface. The browser architecture already provided a useful platform-neutral
IWindowHostand external-renderer boundary, but its WebAssembly/JavaScript command transport is browser-specific and would add unnecessary CPU work on iPhone.The shared control layer also assumed desktop-style window geometry, fixed-height virtualization, discrete mouse-wheel input, and hardware-keyboard text events. Those assumptions caused unsafe-area overlap, missing software-keyboard behavior, nonfunctional simulator trackpad gestures and mouse drag/drop, unusable pickers, wrapped DataGrid text overlapping adjacent rows, and an incorrectly stretched Compute FX offscreen texture on compact layouts.
Architecture and implementation
Native WebGPU/Metal backend
ProGPU.iOSUIKit host and a thinProGPU.Samples.iOSexecutable that launches the existingProGPU.Samples.App.CAMetalLayersized from logical bounds multiplied by the screen's native scale.CADisplayLinkat the screen's maximum supported refresh range.wgpu-nativeC ABI consumed by Silk.NET.WebGPU 2.23.0 and provide a reproducible script that builds device and Apple Silicon simulator XCFramework slices with only WGSL and Metal enabled.ProGPU.Backendbuild-transitive assets and provide the unreachable link stub required by browser AOT.Native WebGPU/Vulkan Android backend
SurfaceHolder,ANativeWindow,Choreographer, and direct wgpu-native Vulkan presentation.Bgra8Unorm/Rgba8Unormpresentation because ProGPU writes encoded sRGB channel values directly; this prevents double transfer-function encoding. Disable Android's full-SurfaceView default focus overlay so indirect input cannot wash out the presented frame.libwgpu_native.soartifacts with WGSL input, deterministic manifests/checksums, 16 KiB ELF/APK alignment, and no vendored upstream source.WinUI controls and panels
Page, reflection-free/AOT-safeFramenavigation, navigation events/options/cache modes,RelativePanel,VariableSizedWrapGrid,Viewbox,ItemsStackPanel,ItemsWrapGrid, group-header placement, display information, lifecycle deferrals, and system back navigation.Stretchto the officialMicrosoft.UI.Xaml.Medianamespace and preserve compatibility in image, rich text, markdown, and shared samples.UniformVirtualizingGridPanelwith variable item extents, prefix-sum lookup, viewport anchoring, bounded realization/recycling, horizontal/vertical layouts, and visible/cache index reporting.WinUI-shaped mobile window APIs
Windowderive fromDependencyObjectand align activation, close, size, visibility, title-bar, bounds, launch, and application-start contracts with the official WinUI API shapes.Windows.UI.ViewManagement.InputPanesurface with occlusion, showing/hiding events, focused-element reporting, and platform show/hide callbacks.WindowInsetscompanion for safe-area, keyboard occlusion, and visible bounds because official WinUI does not expose a portable safe-area API.ScrollViewerorDataGridwhen the keyboard covers them.Mobile text, pointer, gestures, and drag/drop
UITextInputbridge with exact replacement ranges, selection synchronization, marked-text composition, cancellation, deletion, return handling, input scopes, password mode, capitalization, spell checking, autocorrection, dictation edits, and hardware/software keyboard support.UIPasteboard.120 * ln(scale).Microsoft.UI.Input.GestureRecognizercontract with official settings, pointer metadata and transforms, tap/double-tap/right-tap/hold, mouse and pen drag, touch cross-slide, translation rails, scale, rotation, pivot rotation, wheel manipulation, completion, and configurable automatic/manual inertia.Microsoft.UI.Inputvalue/device types and emit timed routed inertial deltas with interruption plus desired deceleration/displacement/angle/expansion behavior.Native storage pickers
NSFileCoordinator.Shared variable-size virtualization and DataGrid wrapping
VariableSizeIndex, a reusable Fenwick prefix-sum geometry index shared byVirtualizingStackPanelandDataGrid.O(log N)measurement updates, item-to-offset lookup, and offset-to-item lookup withO(N)retained numeric state.ItemHeight/RowHeight = float.NaNplus explicit estimated sizes.TextWrappingsettings and measure wrapped cells into each realized row's actual height.Shared samples and regression fixes
docs/ios.md.Performance characteristics
O(1)with no per-frame discovery or allocation.O(log N)prefix-sum operations and bounded realization rather than scanning all rows.ViewConfigurationexactly once.Root causes addressed
CAMetalLayerhost, static resolver, and Metal-only XCFrameworkUITextInputbridgeMicrosoft.UI.Inputcontracts and run typed gesture/routed inertia state machinesValidation
dotnet test src/ProGPU.Tests/ProGPU.Tests.csproj --no-build --no-restoredotnet test src/ProGPU.Tests.Headless/ProGPU.Tests.Headless.csproj --no-build --no-restoredotnet build src/ProGPU.Samples.iOS/ProGPU.Samples.iOS.csproj -c Release -r iossimulator-arm64 --no-restorearm64-v8a, passed 16 KiB zip alignment, and packaged the audited wgpu-native digestb8102032e4e6b425327e507d8dfa48f2f2d898ac25524a9b0c57fa116b4109e7.ProGPU_API_36emulator using Apple M3 Pro -> Vulkan,Rgba8Unorm, and a 1280x2856 physical target at scale 3. The first frame contained 65 draws, 732 vector vertices, and 293 text vertices; the warm gallery HUD reported 56 FPS.git diff --check origin/main...HEADpasses.Follow-up and limitations
wgpu-nativecommit together; newer native releases are not ABI-compatible drop-in replacements.Research
The implementation is clean-room work based on public API contracts and primary sources. Detailed adopted/adapted/rejected decisions are recorded in
docs/ios.mdanddocs/android.md, including AppleCAMetalLayerand indirect input, AndroidANativeWindow/Vulkan/surface/input/IME/storage contracts, the WebGPU specification, wgpu/wgpu-native, .NET mobile AOT/linking, WinUI public APIs, Skia, Direct2D/Win2D, WebRender, Vello, Parley, HarfBuzz, and WinUI virtualization contracts.