feat(web): implementation of modipress, integration with multitap 🐵

[jahorton]

Another goal from #5029: introducing the "modipress". (#5029 (comment) for the original mention.)

Short for "modi-fier long-press", this is a gesture interaction in which the user may hold down any of our recognized modifier keys to temporarily shift the keyboard's active layer... as if it were a longpress key. (Such behavior exists in iOS's default keyboard, for example.) Its layer will then be locked in place until the interaction ends, allowing the user to type with the keys of that layer. Lifting the finger holding the modifier key will then return to the original layer.

Bonus: this also works with layers reached through multitapping on such a key! Tapping once, then holding on a second tap will activate the layer corresponding to the first multitap key under the modifier key - the key that'd normally be reached by a second tap.

At present, this is locked by the following logic:

keyman/web/src/engine/osk/src/input/gestures/specsForLayout.ts

Lines 82 to 94 in facafe3

	function keySupportsModipress(key: KeyElement) {
	// Future enhancement idea: allow some extra way for a key to say "hi, I'm modipressable".
	const keySpec = key.key.spec;
	const modifierKeyIds = ['K_SHIFT', 'K_ALT', 'K_CTRL', 'K_NUMERALS', 'K_SYMBOLS', 'K_CURRENCIES'];
	for(const modKeyId of modifierKeyIds) {
	if(keySpec.id == modKeyId) {
	return true;
	}
	}
	// Allows special-formatted keys with a next-layer property to be modipressable.
	return keySpec.sp > 0 /* ButtonClasses.normal */ && keySpec.sp < 8 /* ButtonClasses.deadkey */ && !!keySpec.nextlayer;
	}

The main thing: it's a lot simpler if we don't have to rely on key-event rule processing to determine whether or not a key should be considered for modipresses. If an approach can fit within the method posted above ("... the following logic"), it'll be very easy to integrate. We can even add access to the compiled keyboard object (say, for a "here's all keys for modipress" property) if needed.

As for the keys included above... including K_NUMERALS, in particular, gives us numeric-layer modipress for sil_euro_latin.

K_LOWER (the numeric-layer 'abc' key) is not currently included, but perhaps I should correct that. That said, the

Possibly relevant codes for consideration:

keyman/common/web/keyboard-processor/src/text/codes.ts

Lines 71 to 74 in 6aaf58a

	"K_LOPT":50001,"K_ROPT":50002,
	"K_NUMERALS":50003,"K_SYMBOLS":50004,"K_CURRENCIES":50005,
	"K_UPPER":50006,"K_LOWER":50007,"K_ALPHA":50008,
	"K_SHIFTED":50009,"K_ALTGR":50010,

As for this part:

if(key.sp > 0 && key.sp != 8 && key.nextlayer) - if the key is marked for special styling that doesn't match deadkeys and has a nextlayer property, we'll trust that key "in good faith" as a modipressable layer-swap key.

Reference:

keyman/common/web/types/src/keyman-touch-layout/keyman-touch-layout-file.ts

Lines 67 to 89 in 6aaf58a

	export const enum TouchLayoutKeySp {
	normal=0,
	/** A 'frame' key, such as Shift or Enter, which is styled accordingly; uses
	* the 'KeymanwebOsk' font on KeymanWeb */
	special=1,
	/** A 'frame' key, such as Shift or Enter, which is styled accordingly and is
	* highlighted to indicate it is active, such as the shift key on a shift
	* layer; uses the 'KeymanwebOsk' font on KeymanWeb */
	specialActive=2,
	/** **KeymanWeb runtime private use:** a variant of `special` with the
	* keyboard font rather than 'KeymanwebOsk' font */
	customSpecial=3,
	/** **KeymanWeb runtime private use:** a variant of `specialActive` with the
	* keyboard font rather than 'KeymanwebOsk' font. */
	customSpecialActive=4,
	/** A styling signal to indicate that the key may have 'deadkey' type
	* behaviour. */
	deadkey=8,
	/** A key which is rendered as a blank keycap, blocks any interaction */
	blank=9,
	/** Renders the key only as a gap or spacer, blocks any interaction */
	spacer=10
	};

(The version directly above has not yet reached this branch, though.)

It'd be a "heuristic" of sorts, though - there's nothing stopping someone from special-styling a text-output key that would swap the layer too. Hopefully folks don't consider that a "feature" and instead wisely design their layouts accordingly.

---

One extra-fun interaction:

If a subkey menu has been opened during a modipress and the modipress key is lifted, subkeys still work... but any nextlayer attribute on the subkeys will not take effect, as it's from a subkey of the modipressed layer.

While we could "just" cancel the subkey-menu operation... I think this scenario is both niche enough and consistent enough to be reasonable. I think it could be possible to maintain the layer-lock until the modipressed longpress is complete, but that will take extra effort to accomplish.

---

User Testing

TEST_BASIC_SIMPLE_SHIFT: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

1. Select "Norwegian" (for the SIL EuroLatin keyboard)
2. Tap the spacebar. (The keyboard should be on the default [lowercase] layer after this.)
3. Tap the shift key once.
4. Verify that the keyboard changes to, and stays on, the shift layer.

TEST_BASIC_MODIPRESS: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

1. Select "Norwegian" (for the SIL EuroLatin keyboard)
2. Tap the spacebar. (The keyboard should be on the default [lowercase] layer after this.)
3. Tap and hold the shift key.
4. Verify that the keyboard changes to the shift layer.
5. Tap the A key, verifying that it produces an A.
6. Release the shift key.
7. Verify that the keyboard changes to the default layer as a result of step 6.
8. Repeat steps 3 through 7 quickly, holding shift just long enough to tap the shift layer's A key, then releasing quickly.

TEST_BASIC_MODIPRESS_HOLD: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

1. Select "Norwegian" (for the SIL EuroLatin keyboard)
2. Tap the spacebar. (The keyboard should be on the default [lowercase] layer after this.)
3. Tap and hold the shift key.
4. Verify that the keyboard changes to the shift layer.
5. Wait at least one second.
6. Release the shift key.
7. Verify that the keyboard changes to the default layer as a result of step 6.

TEST_NUMERIC_FROM_SHIFT: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

1. Select "Norwegian" (for the SIL EuroLatin keyboard)
2. Tap the spacebar. (The keyboard should be on the default [lowercase] layer after this.)
3. Tap the shift key, making the shift layer active.
4. Tap and hold the numeric key (123) underneath the shift key.
5. Verify that the keyboard changes to the numeric layer - numbers in the top row, mathematical symbols elsewhere.
6. Longpress the % key and select the ‱ subkey.
7. Verify that ‱ is emitted as text.
8. Release the numeric key.
9. Verify that the keyboard changes back to the shift layer.
10. Tap the numeric key, releasing it immediately.
11. Repeat steps 6 and 7, longpressing the % key and select the ‱ subkey.
12. Verify that the keyboard automatically changes back to the default (lowercase) layer.

TEST_DELAYED_SUBKEY: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

1. Select "Norwegian" (for the SIL EuroLatin keyboard)
2. Tap the spacebar. (The keyboard should be on the default [lowercase] layer after this.)
3. Tap the shift key, making the shift layer active.
4. Tap and hold the numeric key (123) underneath the shift key.
5. Verify that the keyboard changes to the numeric layer - numbers in the top row, mathematical symbols elsewhere.
6. Longpress the % key, but do not select a subkey yet.
7. Release the numeric key.
8. Verify that the keyboard changes back to the shift layer, with the subkey menu still displayed.
9. Select the ‱ subkey.
10. Verify that ‱ is emitted as text and that the layer does not change as a result.

TEST_DOUBLETAP_CAPS: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

1. Select "Norwegian" - you should then see "Norwegian - EuroLatin (SIL)" in the space bar.
2. Type All .
3. Double-tap the shift key. Afterward, the shift key should be highlighted, and its up-arrow should have a thin line underneath, as if underlining it.
4. Without touching the shift key at any point, type CAPS .
   • If this is impossible - i.e., the layer shifted out from underneath you - FAIL the test.
5. Single-tap the shift key. Afterward, you should be returned to the base layer.
6. Type work .
7. Tap the shift key once.
8. Without touching the shift key at any point, type Well.
   • If this is impossible - i.e., the layer did not drop the the base layer when expected - FAIL the test.
   • Expected final result, in total: All CAPS work Well

TEST_CUSTOM_MULTITAP_MODIFIER: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

1. Select "English...", then "Gesture Prototyping".
2. Tap and hold the ◌́ key. (The key between the spacebar [left] and the Enter key [right].)
3. A number of letters, including all of the vowels, should change to versions using the ◌́ accent mark.
4. Tap the à and verify that the corresponding letter is output.
5. Release the ◌́ key; you should be returned to the default layer.
6. Note the hints on the ◌́, then triple-tap it, holding on the third tap.
7. A number of letters, including all of the vowels, should change to versions using the ◌̂ mark
   • This corresponds to the second diacritic from the ◌́-key's hint.
8. Tap the â and verify that the corresponding letter is output.
9. Release the ◌́ key; you should be returned to the default layer.

TEST_MISCELLANEOUS: If anything you consider an error or bug occurs that seems unrelated to the tests, please note it here.

• If you see something you'd normally report as a bug when regression testing, FAIL this "test" and document the bug here.
• An actual regression-test run will be requested later on, so don't feel the need to perform a full regression-test set at this time.

[keymanapp-test-bot]

User Test Results

Test specification and instructions

• ✅ TEST_BASIC_SIMPLE_SHIFT (PASSED): Tested in an Android mobile device (Samsung Galaxy A23 5g - Android Version 13) using the standard "Test unminified KeymanWeb" test page and here is my observation: 1. Set "Norwegian - EuroLatin" keyboard. 2. Tapped the spacebar. The default [lowercase] layer appears on this keyboard. 3. Tapped Shift key. Verified that the keyboard changed to shift layer. (notes)
• ✅ TEST_BASIC_MODIPRESS (PASSED): 1. Set the "Norwegian - EuroLatin(SIL)" keyboard. 2. Tapped the space bar. verified that the keyboard shows default (lowercase) layer. 3. Tapped and hold the shift key. Verified that the keyboard changed to the shift layer. 4. Tapped the 'A' key. Verified that it produces an A on the text box. 5. Release the Shift key. Verified that the keyboard changed to the default layer. 6. Repeat the steps from 3 through 7 quickly, output the A character on the text box and verified that the keyboard is in the default layer. (notes)
• ✅ TEST_BASIC_MODIPRESS_HOLD (PASSED): 1. Set the "Norwegian - EuroLatin" keyboard. 2. Tapped the spacebar. The keyboard shows the default layer. 3. Tapped and hold the shift key. Verified that the keyboard changed to the shift layer. 4. After a second, I released the Shift key. Verified that the keyboard changes to the default layer. (notes)
• ✅ TEST_NUMERIC_FROM_SHIFT (PASSED): 1. Set the "Norwegian - SIL Eurolatin" keyboard. 2. Tapped the spacebar. The keyboard shows the default layer. 3. Tapped the shift key to make the shift layer active. 4. Tapped and hold the numeric key (123). Verified that the keyboard changed to the numeric layer. 5. Long-press the % key then clicked the ‱ subkey. Verified that it outputs ‱ on the text box. 6. Released the Numeric key. Verified that the keyboard changed back to the shift layer. 7. Repeat the steps 6 and 7 and the expected output ‱ appear again and again on the text box. Also, verified after releasing the numeric key, the keyboard automatically changed back to the default (lowercase) layer. (notes)
• ✅ TEST_DELAYED_SUBKEY (PASSED): 1. Set the "Norwegian - SIL Eurolatin" keyboard. 2. Tapped and hold the numeric key underneath the shift key. 3. Verified that the keyboard changed to the numeric layer. 4. Long-pressed the % key, and did not select the subkey. 5. Released the numeric key. Verified that the keyboard changes back to the shift layer, with the subkey menu still displayed. 6. Tapped the ‱ subkey. Verified that ‱ is emitted as text and that the layer did not changed in the keyboard. (notes)
• ✅ TEST_DOUBLETAP_CAPS (PASSED): 1. Set "Norwegian - EuroLatin(SIL)" keyboard. 2. Typed "All". The text 'All' appeared on the text box. 3. Double-tapped on the shift key. Verified that the shift key was highlighted and its up-arrow appeared with a thin line underneath, as if underlining it. 4. Typed 'CAPS' without touching the shift key. Verified that the corresponding output appear on the text input screen. 5. Single-tapped the shift key. Verified that the keyboard shows the default layer. 6. Typed 'work'. It appears on the screen. 7. Taped the shift key once, then type 'Well' and it shows the corresponding output on the screen. Able to see all the expected outputs on the screen. Seems to be working fine. (notes)
• ✅ TEST_CUSTOM_MULTITAP_MODIFIER (PASSED): 1. Set the "English - Gesture Prototyping" keyboard. 2. Tapped and hold the ◌́ key. Verified that a number of letters, including all of the vowels are changed to versions using the ◌́ accent mark. 3. Tapped à key and verified that the corresponding letter is output. 4. Released the ◌́ key and verified that the keyboard was in the default layer. 5. Triple tapped on ◌́ key and hold on to the third tap. Tapped the â key and it shows the corresponding output on the screen. Release the ◌́ key and verify that it returns to the default layer. (notes)
• ✅ TEST_MISCELLANEOUS (PASSED): I haven't encountered any significant problems so far in this test. Anyhow, it would be better to add this test to our regression test. Thanks.

[mcdurdin]

It'd be a "heuristic" of sorts, though - there's nothing stopping someone from special-styling a text-output key that would swap the layer too. Hopefully folks don't consider that a "feature" and instead wisely design their layouts accordingly.

I think we need to make sure our keyboard design guidance notes this.

[mcdurdin]

A few more thoughts on this one but LGTM once those are addressed

[mcdurdin]

Why do we need try here? Are we expecting exceptions to be thrown?

[jahorton]

Expecting? No, not during standard use. But that's why they're "exceptions". Note: I don't even try to catch them - I'm just looking to ensure a critical recovery step happens even if they occur.

this.modelKeyClick triggers code that links into rule processing, which isn't exactly a narrow area of code. Should something unexpectedly break, we want to at least "unlock" the layer even if an error occurs... otherwise, the OSK will appear to "lock up" to end users, and that's far from ideal.

Sometimes we've seen errors from individual keystrokes that don't break things for other keystrokes. If KMW isn't completely broken, just failing that one keystroke can allow a form of recovery... in which we want to ensure that any layer-lock (modipress) manipulations we do here also recover if possible.

[mcdurdin]

Okay, we just haven't got anywhere near that level of robustness elsewhere in KeymanWeb, so this is a bit of a new pattern.

[jahorton]

A text-search across the repo for } finally { actually turns up 8 pre-existing instances of this exact pattern. It's not the first time I've used it like this.

[bharanidharanj]

Test Results

• TEST_BASIC_SIMPLE_SHIFT (PASSED): Tested in an Android mobile device (Samsung Galaxy A23 5g - Android Version 13) using the standard "Test unminified KeymanWeb" test page and here is my observation: 1. Set "Norwegian - EuroLatin" keyboard. 2. Tapped the spacebar. The default [lowercase] layer appears on this keyboard. 3. Tapped Shift key. Verified that the keyboard changed to shift layer.

  

• TEST_BASIC_MODIPRESS (PASSED): 1. Set the "Norwegian - EuroLatin(SIL)" keyboard. 2. Tapped the space bar. verified that the keyboard shows default (lowercase) layer. 3. Tapped and hold the shift key. Verified that the keyboard changed to the shift layer. 4. Tapped the 'A' key. Verified that it produces an A on the text box. 5. Release the Shift key. Verified that the keyboard changed to the default layer. 6. Repeat the steps from 3 through 7 quickly, output the A character on the text box and verified that the keyboard is in the default layer.

  

• TEST_BASIC_MODIPRESS_HOLD (PASSED): 1. Set the "Norwegian - EuroLatin" keyboard. 2. Tapped the spacebar. The keyboard shows the default layer. 3. Tapped and hold the shift key. Verified that the keyboard changed to the shift layer. 4. After a second, I released the Shift key. Verified that the keyborad changes to the default layer.

• TEST_NUMERIC_FROM_SHIFT (PASSED): 1. Set the "Norwegian - SIL Eurolatin" keyboard. 2. Tapped the spacebar. The keyboard shows the default layer. 3. Tapped the shift key to make the shift layer active. 4. Tapped and hold the numeric key (123). Verified that the keyboard changed to the numeric layer. 5. Long-press the % key then clicked the ‱ subkey. Verified that it outputs ‱ on the text box. 6. Released the Numeric key. Verified that the keyboard changed back to the shift layer. 7. Repeat the steps 6 and 7 and the expected output ‱ appear again and again on the text box. Also, verified after releasing the numeric key, the keyboard automatically changed back to the default (lowercase) layer.

  

• TEST_DELAYED_SUBKEY (PASSED): 1. Set the "Norwegian - SIL Eurolatin" keyboard. 2. Tapped and hold the numeric key underneath the shift key. 3. Verified that the keyboard changed to the numeric layer. 4. Long-pressed the % key, and did not select the subkey. 5. Released the numeric key. Verified that the keyboard changes back to the shift layer, with the subkey menu still displayed. 6. Tapped the ‱ subkey. Verified that ‱ is emitted as text and that the layer did not changed in the keyboard.

  

• TEST_DOUBLETAP_CAPS (FAILED): 1. Set "Norwegian - EuroLatin(SIL)" keyboard. 2. Typed "All". The text 'All' appeared on the text box. 3. Double-tapped on the shift key. Noticed that the shift key was not been highlighted. The Keyboard still appear on the default (lowercase) layer. Seems to be an issue.

  

• TEST_CUSTOM_MULTITAP_MODIFIER (FAILED): 1. Set "English - Gesture Prototyping" keyboard. 2. Tapped and hold the ◌́ key. Verified that a number of letters , including all of the vowels are changed to versions using the ◌́ accent mark. 3. Tapped à key and verified that the corresponding letter is output. 4. Released the ◌́ key and verified that the keyboard was in default layer. 5. Triple tapped on ◌́ key and hold on the third tap. Noticed that it did not show up the second diacritic from the ◌́ key's hint. Noticed that à key appears on the text box. Seems to be an issue.

  

[bharanidharanj]

Test Results

• TEST_MISCELLANEOUS (PASSED): I haven't encountered any significant problems so far in this test.

[jahorton]

@keymanapp-test-bot retest all

While it's nice that some of the tests passed, I've noticed that fixing one aspect sometimes has led me to break others. It's best to do a full retest to ensure nothing is left broken by accident.

[bharanidharanj]

Test Results

• TEST_BASIC_SIMPLE_SHIFT (PASSED): Tested in an Android mobile device (Samsung Galaxy A23 5g - Android Version 13) using the standard "Test unminified KeymanWeb" test page and here is my observation: 1. Set "Norwegian - EuroLatin" keyboard. 2. Tapped the spacebar. The default [lowercase] layer appears on this keyboard. 3. Tapped Shift key. Verified that the keyboard changed to shift layer.

  

• TEST_BASIC_MODIPRESS (PASSED): 1. Set the "Norwegian - EuroLatin(SIL)" keyboard. 2. Tapped the space bar. verified that the keyboard shows default (lowercase) layer. 3. Tapped and hold the shift key. Verified that the keyboard changed to the shift layer. 4. Tapped the 'A' key. Verified that it produces an A on the text box. 5. Release the Shift key. Verified that the keyboard changed to the default layer. 6. Repeat the steps from 3 through 7 quickly, output the A character on the text box and verified that the keyboard is in the default layer.

  

• TEST_BASIC_MODIPRESS_HOLD (PASSED): 1. Set the "Norwegian - EuroLatin" keyboard. 2. Tapped the spacebar. The keyboard shows the default layer. 3. Tapped and hold the shift key. Verified that the keyboard changed to the shift layer. 4. After a second, I released the Shift key. Verified that the keyboard changes to the default layer.

  

• TEST_NUMERIC_FROM_SHIFT (PASSED): 1. Set the "Norwegian - SIL Eurolatin" keyboard. 2. Tapped the spacebar. The keyboard shows the default layer. 3. Tapped the shift key to make the shift layer active. 4. Tapped and hold the numeric key (123). Verified that the keyboard changed to the numeric layer. 5. Long-press the % key then clicked the ‱ subkey. Verified that it outputs ‱ on the text box. 6. Released the Numeric key. Verified that the keyboard changed back to the shift layer. 7. Repeat the steps 6 and 7 and the expected output ‱ appear again and again on the text box. Also, verified after releasing the numeric key, the keyboard automatically changed back to the default (lowercase) layer.

..NumShift1

..NumShift2

• TEST_DELAYED_SUBKEY (PASSED): 1. Set the "Norwegian - SIL Eurolatin" keyboard. 2. Tapped and hold the numeric key underneath the shift key. 3. Verified that the keyboard changed to the numeric layer. 4. Long-pressed the % key, and did not select the subkey. 5. Released the numeric key. Verified that the keyboard changes back to the shift layer, with the subkey menu still displayed. 6. Tapped the ‱ subkey. Verified that ‱ is emitted as text and that the layer did not changed in the keyboard.

  

• TEST_DOUBLETAP_CAPS (PASSED): 1. Set "Norwegian - EuroLatin(SIL)" keyboard. 2. Typed "All". The text 'All' appeared on the text box. 3. Double-tapped on the shift key. Verified that the shift key was highlighted and its up-arrow appeared with a thin line underneath, as if underlining it. 4. Typed 'CAPS' without touching the shift key. Verified that the corresponding output appear on the text input screen. 5. Single-tapped the shift key. Verified that the keyboard shows the default layer. 6. Typed 'work'. It appears on the screen. 7. Taped the shift key once, then type 'Well' and it shows the corresponding output on the screen. Able to see all the expected outputs on the screen. Seems to be working fine.

  

• TEST_CUSTOM_MULTITAP_MODIFIER (PASSED): 1. Set the "English - Gesture Prototyping" keyboard. 2. Tapped and hold the ◌́ key. Verified that a number of letters, including all of the vowels are changed to versions using the ◌́ accent mark. 3. Tapped à key and verified that the corresponding letter is output. 4. Released the ◌́ key and verified that the keyboard was in the default layer. 5. Triple tapped on ◌́ key and hold on to the third tap. Tapped the â key and it shows the corresponding output on the screen. Release the ◌́ key and verify that it returns to the default layer.

  

• TEST_MISCELLANEOUS (PASSED): I haven't encountered any significant problems so far in this test. Anyhow, it would be better to add this test to our regression test. Thanks.