Mobile support for document editing

[matthew-carroll]

Provide the mobile editing capabilities needed to ship Superlist and Clearful

Spec

• A toolbar to allow custom formatting displayed on top of the virtual keyboard. An example implementation should be part of the demo application
  • Provide UX for the sample application mobile toolbar, esp. if limited space require scrolling of toolbar @bk-one
• Support for a selection toolbars (copy, paste, share, etc.) as a platform agnostic variant defined in SuperEditor (non-configurable)
  • This selection toolbar should be expandable (add additional options)
• Provide drag handles for collapsed and expanded selections, including when a selection spans multiple nodes and varied content.
  • Also adjust gesture interactions to match mobile expectations
  • Spec document on the detailed gesture interactions @Jethro87 & @bk-one
  • "Trackpad" functionality on the virtual keyboard to place the caret, compare iOS and the Notes app (hold space) @bk-one
  • Magnifying glass on long-press to place the caret
• speech-to-text support if accessible via native dictation functionality of the host system -> Extract to separate milestone if not as trivial as anticipated
• option to close the keyboard (e.g. via done button or hide keyboard button, SuperEditor should provide an easy API for this)

Out of scope:

• Support for autocomplete and spell correction (type ahead support; auto-correct based on system settings) as this might be supplied by the IME in the future

Funding

Superlist, Clearful

[matthew-carroll]

@bk-one I'm starting to spec out and investigate mobile support. This first attempt is driven by Superlist requirements. Can we capture the minimum mobile requirements for Superlist to reach a shippable state?

[Jethro87]

@matthew-carroll Here's a list of capabilities we would need to be able to launch super_editor in Clearful in production.

Required

• Formatting bar
• Formatting options (Can we set the styling of headers, etc?)
  • Bold
  • Italic
  • Headers (1-3)
  • Ordered lists
  • Unordered lists
• Read-only document display
  • Read-only document display #303
• JSON format
  • JSON format as an alternative to Markdown #254
  • Export document as JSON
  • Import / create document from JSON
• Basic content selection (including copy/paste)
  • Content Selection #319
  • Formatting not required for initial release when pasting from another rich text source

Questions

• Keyboard input
  • Can a user pair a bluetooth keyboard to their mobile device and type?

Soon Afterwards

• Image support
• Placeholder support (on a per-node basis)
• Read-only document within custom scroll view
  • https://github.com/superlistapp/super_editor/discussions/304

[matthew-carroll]

HI @Jethro87 - I'd like to focus this milestone on mobile-specific capabilities.

Is there something mobile-specific about "JSON format", "Image support", "Placeholder support", "custom scroll view", and "read-only documents"?

For the "formatting bar", what kind of deliverable are you looking for from super_editor on that front?

For "formatting options" - the ability to apply and remove attributions in text should already work, regardless of platform. Can you elaborate on that?

[Jethro87]

@matthew-carroll Got it, I misunderstood. The above are all basic requirements for us to have the editor in production; not just mobile-specific requirements.

There is nothing mobile-specific about "JSON format", "Image support", "Placeholder support", "custom scroll view", and "read-only documents".

The "formatting bar", as it has already been demoed, would be fine as it is.

"Formatting options" are also fine as they are, I was just reiterating our need for them.

[matthew-carroll]

Sounds good. I updated the description to call out specific capabilities. Are there any other document editing behaviors or concerns that come to mind beyond those I've listed in the description?

[Jethro87]

The spec looks good.

From Ben's list, I'll second "precise cursor positioning", and I agree that it makes sense for super_editor to expose a "close keyboard" API. I would also be very concerned if the OS doesn't send "autocomplete and spell correction", but if there's nothing we can do, there's nothing we can do.

[matthew-carroll]

#341 - Will offer developers the choice of keyboard vs IME input, and mouse vs touch gestures.

For now, the IME input and touch input will be stubbed. This ticket only sets up the structure to allow for this separation.

[matthew-carroll]

#341 has a PR at #345

[bk-one]

The primary requirements for Superlist are:

• content selection
• precise cursor positioning (in document and in input fields, see for example https://www.youtube.com/watch?v=uMevwmsbrTk, happy to spec this further)
• copy/paste from/to device clipboard close to native behavior for iOS and Andriod (compare Apple Notes or Google Keep)
• Formatting bar with common formatting options (H1, H2, Bold, Italic, etc. - equivalent to desktop demo app) on top of keyboard
• Support for autocomplete and spell correction (type ahead support; auto-correct based on system settings)
• speech-to-text support
• read-only mode with smooth scrolling when no caret placed (and no virtual keyboard visible)
• option to close the keyboard (via done button or hide keyboard button, happy to spec further)

[matthew-carroll]

Hi @bk-one - similar to my comments on the Clearful requirements, my goal in this milestone is to capture mobile-specific functionality that you require. I think some of these items, while completely valid, are beyond the scope of this specific milestone. We should workshop these.

Perhaps, you'd like @miguelcmedeiros or someone else to interpret your list in terms of this milestone. I'm available to discuss, if there are clarifying questions.

To your points:

content selection

^ We certainly need to support content selection. This might be a dozen use-cases that are significantly different. Can we accumulate a representative sample for what this means?

precise cursor positioning (in document and in input fields, see for example https://www.youtube.com/watch?v=uMevwmsbrTk, happy to spec this further)

^ Yes, please, let's spec this further. I investigated this rather thoroughly for the text field work. There are a handful of different behaviors between iOS and Android. Also, based on the video, this line-item seems to refer to all gesture selection modalities, rather than a focus on position precision.

copy/paste from/to device clipboard close to native behavior for iOS and Andriod (compare Apple Notes or Google Keep)

^ I'm sure the OS is involved in this equation, but I wouldn't categorize clipboard integration as mobile editing. Or, at least, I think we're facing a cross-platform copy/paste problem that needs a high level of dedicated attention. Once that's solved, it will probably be trivial to apply that solution to mobile.

Formatting bar with common formatting options (H1, H2, Bold, Italic, etc. - equivalent to desktop demo app) on top of keyboard

^ This seems like an app-specific concern. I think we can make a default bar available, but odds are that Superlist will want a different one. Can we dig deeper into the tools that Superlist needs to accomplish this goal?

Support for autocomplete and spell correction (type ahead support; auto-correct based on system settings)

^ I don't know if we have any control over this. Once the IME is integrated, we'll get whatever composing region and deltas the OS sends us, but if the OS doesn't send us autocompletions and spelling changes, then I doubt there is much we can do about it.

speech-to-text support

^ Please feel free to correct me, but my initial reaction is that whatever you need for speech-to-text is well beyond the scope of super_editor. Perhaps it would be an add-on package that interfaces with speech-to-text and then translates the "text" part to super_editor nodes or deltas at the last moment.

read-only mode with smooth scrolling when no caret placed (and no virtual keyboard visible)

^ This is an entire feature set. For the most part, read-only is the editor minus a bunch of stuff, but I wouldn't be surprised if new requirements came up for read-only docs. Can someone investigate that? Also, you mentioned smooth scrolling. Is that meant as a juxtaposition to non-smooth-scrolling right now? If so, let's collect some reproductions so that everyone can see the perceived problem.

option to close the keyboard (via done button or hide keyboard button, happy to spec further)

^ This is fine. I can probably find somewhere to expose this. Technically you can do this now, but maybe it makes sense to provide a convenient API in super_editor. Once super_editor brings the API, your team should bring the button. For example, Notion places the button as an icon above the keyboard in the same place as other Notion-specific editing controls.

[bk-one]

@matthew-carroll would it probably most efficient to have a short workshop with @Jethro87, yourself and anyone else that has a stake in it to discuss the requirements and bring them in order?

[matthew-carroll]

Status update: here's a PR that adds Android-style and iOS-style selection handles, magnifiers, and toolbars:

#358

[matthew-carroll]

As I'm working on mobile document editing, I'm finding some Flutter issues. I'm going to log those here for our reference, and then possibly file them in the Flutter project, depending on how they turn out.

• The TextRanges reported from Flutter violate their contract. The reported end values are inclusive, but the TextRange API docs say that they should be exclusive.
• There seems to be a lot of unnecessary non-text changes reported from the IME. These include repeated deltas that are all the same, as well as some deltas that seem to change the composing region, and then immediately change it back.
• On Android, when I first position the caret, and send the text to the IME, I can't get a delete delta. For example, imagine that I send the following text and caret to the IME: "This is so|me text". Pressing the delete button should generate a delete delta that removes the "o". Instead, nothing happens when the delete button is pressed. If I move the selection anywhere else, then I'm suddenly able to get a delete delta. There seems to be something special about the very first blob of text and selection that's sent to the IME.
• On Android, as the user swipes around the keyboard to assemble words, the IME reports numerous deltas despite the fact that no content is changing during this process.
• RenderParagraph doesn't seem to measure emojis. Attempting to get the character box where an emoji sits, returns null. This prevents line-height measurement for the placement of a caret next to the emoji.

Something to look into:

A Hot Restart doesn't close the IME connection. Should Flutter add that behavior? It seems unlikely that a Hot Restart, which completely clears the Dart memory, would want to retain IME state.

[osaxma]

• The Hot Restart leaving the Keyboard up is a known issue: Hot restart while keyboard is up leaves the keyboard up flutter/flutter#10713

As a workaround, this hides the keyboard and clears the IME client if placed somewhere in the app's main():

  if (kDebugMode) {
    // Hide the keyboard after a hot restart
    // issue tracker: https://github.com/flutter/flutter/issues/10713
    SystemChannels.textInput.invokeMethod('TextInput.hide');
    SystemChannels.textInput.invokeMethod('TextInput.clearClient');
  }

• For unnecessary reports, Flutter's TextEditingControllersState.updateEditingValue seems to be aware of the issue and it handles it by caching the remote editing value and checks if it has changed for each event:

    if (value == _value) {
      // This is possible, for example, when the numeric keyboard is input,
      // the engine will notify twice for the same value.
      // Track at https://github.com/flutter/flutter/issues/65811
      return;
    }

[matthew-carroll]

This milestone is complete. I'm locking the conversation.