Mouse in driver 2.0

[tznind]

User expectation

Users interact with computers via mouse. We want to make the experience as smooth as in GUI environments e.g. Windows. We also want the programmer making the app to have a good experience.

So events like clicked, double and click-drag should be easy to hook without having to think too hard.

But lets look at some advanced scenarios:

User clicks down in

• User clicks down in button
• Then changes mind and drags mouse off to release
• No Click should be raised

User clicks down wobbles then releases

• User clicks down in button
• Then changes mind and drags off for a second
• Then realizes it is definetly what he wants so moves back over button for release

Here is two examples of that, in the first case user waits too long before release so UI does not close. In second case the gap is shorter so behaviour is treated as click.

To support this we need to gather and retain information about what the mouse is doing as long as any mouse button is down - and for a short time afterwards, before relasing certain events e.g. click.

The Console

The console communicates mouse state to us via ANSI escape sequences:

ESC[<C;X;YM

C indicates button
X/Y indicates coordinates
M or m is terminator and indicates press/release

State	Meaning*
0	Button released
1	Left button pressed
32	Dragging with left button pressed
35	Movement with no buttons pressed

* this is my understanding currently, it may be slightly off as I'm not an expert yet.

A futureproof mouse model

I think we can separate the ANSI layer, the state (interpretation) and the library user facing layers:

For each button we need a class to model its state at any given in time (current or past):

public class ButtonState
{
    /// <summary>
    ///     When the button entered its current state.
    /// </summary>
    public DateTime At { get; set; }

    /// <summary>
    /// <see langword="true"/> if the button is currently down
    /// </summary>
    private bool Depressed { get; set; }

    /// <summary>
    /// The screen location when the mouse button entered its current state
    /// (became depressed or was released)
    /// </summary>
    public Point Position { get; set; }

    /// <summary>
    /// The <see cref="View"/> (if any) that was at the <see cref="Position"/>
    /// when the button entered its current state.
    /// </summary>
    public View? View { get; set; }

    /// <summary>
    /// True if shift was provided by the console at the time the mouse
    ///  button entered its current state.
    /// </summary>
    public bool Shift { get; set; }

    /// <summary>
    /// True if control was provided by the console at the time the mouse
    /// button entered its current state.
    /// </summary>
    public bool Ctrl { get; set; }

    /// <summary>
    /// True if alt was held down at the time the mouse
    /// button entered its current state.
    /// </summary>
    public bool Alt { get; set; }
}

But we also need to be able to accumulate these between when a mouse is clicked down and when it is released. We could store all states or just the first and last. So far I am considering the word Narrative i.e. we are telling a story about what the mouse did while the button was pressed.

This is my initial sketch of the class:

/// <summary>
/// Describes a completed narrative e.g. 'user triple clicked'.
/// </summary>
/// <remarks>Internally we can have a double click narrative that becomes
/// a triple click narrative. But we will not release both i.e. we don't say
/// user clicked then user double-clicked then user triple clicked</remarks>
public class ButtonNarrative
{
    public int Button { get; set; }
    public int NumberOfClicks { get; set; }

    /// <summary>
    /// Mouse states during which click was generated.
    /// N = 2x<see cref="NumberOfClicks"/>
    /// </summary>
    public List<ButtonState> MouseStates { get; set; }

    /// <summary>
    /// <see langword="true"/> if distance between first mouse down and all
    /// subsequent events is greater than a given threshold.
    /// </summary>
    public bool IsDrag { get; set; }
}

There is a timing element and decision engine that must interpret clicks and must have regular 'check ins' regardless of whether there are more mouse actions.

For example imagine we get a click

We must wait a short time to see if there is another click, if so then we need to accumulate that in the narrative and release it as a double click (assuming no third click).

We need to make sure not to release click then double click then tripple click.

Here is my initial pseudocode WIP on the topic:

public class MouseStateManager
{
    /// <summary>
    /// Function for returning the current time. Use in unit tests to
    /// ensure repeatable tests.
    /// </summary>
    private Func<DateTime> Now { get; set; }

    /// <summary>
    /// How long to wait for a second click after the first before giving up and
    /// releasing event as a 'click'
    /// </summary>
    public TimeSpan DoubleClickThreshold { get; set; }

    /// <summary>
    /// How long to wait for a third click after the second before giving up and
    /// releasing event as a 'double click'
    /// </summary>
    public TimeSpan TripleClickThreshold { get; set; }

    /// <summary>
    /// How far between a mouse down and mouse up before it is considered a 'drag' rather
    /// than a 'click'. Console row counts for 2 units while column counts for only 1. Distance is
    /// measured in Euclidean distance.
    /// </summary>
    public double DragThreshold { get; set; }

    public MouseState CurrentState { get; private set; }

    private ButtonNarrative? [] _ongoingNarratives = new ButtonNarrative? [4];

    public MouseStateManager (
        Func<DateTime>? now = null,
        TimeSpan? doubleClickThreshold = null,
        TimeSpan? tripleClickThreshold = null,
        int dragThreshold = 5
    )
    {
        Now = now ?? (() => DateTime.Now);
        DoubleClickThreshold = doubleClickThreshold ?? TimeSpan.FromMilliseconds (500);
        TripleClickThreshold = tripleClickThreshold ?? TimeSpan.FromMilliseconds (1000);
        DragThreshold = dragThreshold;
    }


    /// <summary>
    /// 
    /// </summary>
    /// <param name="e"></param>
    /// <returns></returns>
    public IEnumerable<ButtonNarrative> UpdateState (MouseEventArgs e)
    {
        // TODO: manage transitions

        for (int i = 0; i < 4; i++)
        {
           // Update narratives

           // Release stale or naturally complete ones based on thresholds
        }
    }

    /// <summary>
    /// If user double clicks and we are waiting for a triple click
    /// we should give up after a short time and just assume no more
    /// clicks are coming. Call this method if the state for a given button
    /// has not changed in a while.
    /// </summary>
    /// <returns></returns>
    public ButtonNarrative? ReleaseState (int button)
    {
        
    }

    private void PromoteSingleToDoubleClick ()
    {

    }
    private void PromoteDoubleToTripleClick ()
    {

    }

    public static double DistanceTo (Point p1, Point p2)
    {
        int deltaX = p2.X - p1.X;
        int deltaY = p2.Y - p1.Y;
        return Math.Sqrt (deltaX * deltaX + deltaY * deltaY);
    }
}

[BDisp]

User clicks down in

• User clicks down in button
• Then changes mind and drags mouse off to release
• No Click should be raised

This was always my opinion too. It's wrong to invoke a clicked event in these situations, but only mouse down and mouse up (pressed/release).

[tig]

Here are some existing use-cases that demonstrate both good/bad behavior of the current impl:

• Border and View dragging. Note how if you start dragging outside a border, then cross the border, it starts dragging. This is bogus. I couldn't figure out how to fix this with the current design. I think your design will address.

• Also Border: Arrangement - Just a nice complex scenario. We need to ensure your design supports how this currently works.

• HighlightStyle.Hover - e.g. for Button. For the most part it works exactly as you describe the ideal above. I worked hard to get this right, feeling like I was fighting the library the whole time. I think your design will make this much easier.

• Drag-Then-Move-Outside - ScrollBar and Slider do it right, If you drag outside, the drag keeps doing its thing:
  

• ColorPicker (ColorBar) does it wrong (when you drag outside after starting the drag transitions to something else:
  

• MouseEnter/Leave - now works pretty well. The Mouse Scenario demonstrates.

[BDisp]

It may have to do with the MouseGrabView. If initially you start grab from a view that view is grabbed, so MouseGrabView is set to that view. If you move to another view, by still pressing the mouse we must check if the MouseGrabView isn't null and thus the grab event only can respond to the first view. What I think it's happening is maybe inn the MouseEnter is assigning the MouseGrabView to the new view without the first view ungrab it.

[tznind]

I think we can improve on MouseGrabView, it is defined as:

Gets the view that grabbed the mouse (e.g. for dragging). When this is set, all mouse events will be routed to this view until the view calls or the mouse is released.

I think we can make this implicit. We divide mouse events into two types:

• Atomic
• Narrative

Atomic events are like click, mouse wheel, mouse down and mouse up etc. They are 'point in time' events i.e. single state.

Narrative events are ongoing i.e. involve the accumulation of multiple mouse states over time e.g. drag.

For these Narrative events we would automaticaly raise the event for each view in the narrative (i.e. the view the mouse was pressed down over and potentially any views over which the mouse passes).

This would mean no need to 'grab' the mouse, you just look at the stack of events and decide what to do.

This lets for example:

• The dragged view know it is being dragged
• Any view you drag over know its happening e.g. to indicate to you it is a valid drop target
• Any view you release on know its been dropped into

If the term Narrative is confusing, we can just call them MouseSequences or Drag - if we can't think of any other kinds of 'over time' events.

[BDisp]

I like these sequences and also, any view that the mouse pass over know that a drag operation is happening. For a validation the view can add to the Data property the object it is dragging and if the view under the mouse check the Data property and if it's a valid object maybe a colored view as green or a check glyph indicates that is valid, otherwise a colored view as red or a X glyph.

[tznind]

Ok super prototype click selecting is in place

[tznind]

Ok here is the MouseInterpreter as implemented in #3837

I've renamed narrative MouseButtonSequence instead as it is clearer.

A sequence accumulates events then conditionally resolves them into a single 'resolved' event. Similarly to AnsiParser there are 2 ways you can end up releasing (if the mouse moves away or if time has passed without seeing events).

For example:

• Mouse pressed at 0,0
• Mouse released at 0,0
• time passes

Would result in a click.

Likewise

• Mouse pressed at 0,0
• Mouse released at 10,10

Would result in nothing because the mouse is moving around - its clearly a click and drag not a click.

Its not fully implemented but it will be:

• Super testable
• Not driver specfic
• Tolerant to different terminal reporting capabilities (e.g. if mouse released is not supported)

For now the input and output objects for this subsystem are MouseEventArgs because that is what everything else in Terminal.Gui is dealing in. I dont want to tackle that beast just yet.

Benefits over current system

• More testable
• Less code
• We wont send Mouse1Clicked then Mouse1DoubleClicked then Mouse1TripleClicked (only Mouse1TripleClicked)