The Leap Motion controller provides a protocol for communicating high-level hand data to applications using Websockets. This document outlines the usage of this protocol.
leapd is a service which interprets data coming from the Leap Motion
controller and reconstructs hand position. That hand data is made available
in via a web socket server within leapd. This web socket communication is
done through an local service on your computer.
A websocket client can connect and communicate with leapd via this web socket
connection. Several clients exist already for this data, for example:
- https://github.com/leapmotion/leapjs - Javascript client for Nodejs and browsers.
- https://github.com/logotype/LeapMotionAS3 - Flash client
The data sent by the websocket server is versioned. As new fields and features are added, the version of data sent gets incremented. Currently there are four versions available. The requested version is encoded in the URI used to conenct.
The current URL format is /{version number}.{format}. The only supported format
is json.
As of now (November 2013) the following protocols are supported. Each is described in a specific documentation sheet.
- Version 1: accessed by '/' or '/v1.json'
- Version 2: accessed by '/v2.json
- Version 3: accessed by '/v3.json'
- Version 4: accessed by '/v4.json' -- the current (as of November 2013) latest protocol.
When you send this first message, you get back a single JSON object echoing the version that the leapd will be using in its communications with your client: {version: 4}
And from then on, you will receive frame data and events over that webSocket channel, as detailed below.
One leapd can have zero, one or many clients. It may only have one piece of hardware that it reports for. Whether a given client receives frame and gesture information depends on which client has focus and which clients have elected to listen in the background; this is detailed below. It also relies on the leapd's connection to a Leap Motion Controller hardware.
In a windowed GUI, mouse state is "blocked" by overlapping windows. You would not expect to receive mouse events if another window was wholly covering your own. With leap, however, every client expects to receive all hand/finger/pointer data continuously. There is no automatic blocking of leapd data when you (in your window/mouse environment "defocus" an application window, cover it with another window, minimize it, etc.
There may not even BE an active window or visible application tied to a particular client.
It is up to each client to decide when and if to start and stop listening to the stream when the application context changes.
You can tell the leapd server one of four things:
- "I have gained focus" -- _place_holder;{focused: true}. Take focus away from the last "on top" client and send me frames.
- "I have lost focus" -- {focused: false}. I do not want to be on top. this will NOT _place_holder;give any other client the focus but it will un-register this client as the focused client.
- "I always want to get frames" -- {background: true }. I want to receive frame data no matter which client has the focus.
- "I only want to get frames when in focus" -- {background: false}. I do not want to receive frame data unless I have the focus. (note - this is the default status of all clients until you explicitly send {background: true}.
The client is not told which state they are in _place_holder;by the leapd engine, except if the hardware is disconnected or reconnected. However beneath the surface, each client has the following possible states:
Connected Focused Listening Gets Frames
YES YES YES YES
YES YES NO YES
YES NO YES YES
YES NO NO NO
NO YES YES NO
NO YES NO NO
NO NO YES NO
NO NO NO NO
In any of these scenarios, if the hardware is disconnected, a "deviceDisconnect" event is sent to them to inform them that there is not a current connection to the physical hardware. (but that by implication they ARE still connected to leapd).
If hardware is then plugged in, they will receive another message letting them know that the hardware is connected.
State events about connected/disconnected hardware are not repeated. They are only sent once to each client for each state change; that is, when the Leap hardware is plugged in/detected or when the hardware is unplugged/loses connection.
Almost all of the communication between the leapd and the client is one-way. The leapd broadcasts frames or state changes to the listening clients.
The client to leap communication happens when the client wants to change its state or turn gestures on and off.
Messages regarding focus and background from ONE client may interfere with ANOTHER client's leapd communication.
When a client grabs focus by setting {focused: true} to leapd, other clients will no longer receive frames. However other clients may also grab focus; there is no way to "Lock" the focus of the leapd to your connection.
The leapd will NOT inform the client when it has lost focus. However you can infer that this is so when you are in a client state that would ordinarily be getting frames and you don't get frames.
Clients can elect whether or not to receive gestures. This choice is at a per- client basis; if one client elects to get gestures, it will not affect whether another client does or does not receive gestures.
By default, you will not receive gestures unless you "opt in".
The array of gestures may be empty, have one gesture, or have many gestures.
Some gestures are only connected with a single event. Some are continuous, such as the circle gesture. Continuous gestures' states will be one of three (string) values: 'start', 'continue' or 'stop'.
You opt in or out of gestures by sending {enableGestures: true} or {enableGestures:false} to leapd. (it will not give you any feedback in response. )
As of November, 2013, this is the most current schema that leapd provides.
There are two sets of messages -- leapd to client and client to leapd
As described in the leapd documentation, the client establishes connection to the leapd by connecting to '/v4'. This connection is made to the leapd daemon at port 6347.
The server will respond with {version: 4}. It will then send a series of frames (see below).
If the server CANNOT support the requested api version it sends back a message with a status code of 1001. No JSON block is provided if the version is not supported.
By sending {background:true} or {background:false} to leapd you can elect to listen to frames regardless of which client has focus.
no direct response -- but you may see a change in your frame stream.
By sending {focused: true} or {focused: false} you can attempt to commandeer the stream of frames from other clients of the leapd. This is detailed in the leapd communication.
no direct response -- but you may see a change in your frame stream.
Events inform the client about changes to the state of the connection, etc. There is currently only one event: deviceConnect.
state is either true (you can expect frames) or false (the hardware is not communicating with the daemon, so no frames can be communicated.
## frames
Frames are streamed to active or listening clients as JSON blocks. The schema
of the json block is as follows. Note, the "r", "s" and "t" values for all
elements described below are for internal use and should not be used in your
application.
## id: float
a unique identifier of the frame
### timestamp: integer
a Unix timestamp.
### hands: [hand]
an array of hand data
### interactionBox: InteractionBox
the dimensions of the measurable area of the Leap Motion Controller hardware
### pointables: [Pointable]
An array of all your pointables: fingers, sticks, from all hands in the
interaction box.
### gestures [Gesture]
an array of 0..many Gestures currently detected
### r: [[floats]]
(rotation) a 4 x 4 matrix; mainly for internal use, reflects overall motion of
detected objects.
### s: [float]:3
(scale) a 3 value array of floats indicating relative size of the space
occupied by detected objects; for internal use
### t: [float]:3
(translation) a three value array of the net relative motion of detected
objects, for internal use
## Hands
### id: integer
the identity of the hand
### direction: [3x float]
euler angles of the hand
### palmNormal: [3x float]
euler angles of the palm
### palmPosition: [3x float]
the position of the center of the hand, in millimeters, relative to the
Controller hardware
### palmVelocity: [3x float]
The change in position of the palm relative to the last measured position of
this hand.
### sphereCenter [3x float]
The position of an imaginary "ball" being held by the hand, in millimeters,
relative to the Controller hardware
### sphereRadius float
The distance from the imaginary "ball" to your palm
r: [[float]]
rotation relative to last known position of this hand; for internal use.
s: float
size relative to the last frame of the extent of the hand; for internal use
### t
[float]: 3
relative motion of the hand and its fingers since the last frame; for internal
use
## InteractionBox
The interaction box describes the measured area of space in which Leap can
detect things. It is not going to change over time.
### center [3x float]
the position, in millimeters, of the origin relative to the Controller
hardware
### size [3x float]
the extent of the Controller detection area in millimeters. The three values
are width, height, and depth.
Note, it is not in reality a square; the closer you get to the Controller, the
narrower the detection region gets.
### Gestures
### center [3x float]
the position, in millimeters, of the gesture. For circular gestures it is the position of the center of the gesture.
### duration: integer
the number of microseconds that the gesture has been maintained.[1]
### handIds [integer]
the ids of the hands making the gestures; at this point most gestures are made by a signle hand.
### pointableIds: [integer]
the ids of the pointables making the gestures
### progress: float
?????
### radius: float
for the circle gesture, the approximate distance in millimeters between the center and the
pointable (or the mean location of the pointables).
### state: string -- 'start', 'update', 'stop'
Whether the gesture has started or stopped this frame. Not all gestures will express in all three states -- one-frame
gestures may only express 'stop'.
### type: string -- 'circle', 'swipe', 'keyTap', 'screenTap'
the type of gesture.
# Leap Protocol Version 3
Leap Protocol Version 3 does not feature background events. You cannot expect to change your frame stream by sending
`{focused: true}` or `{focused: false}` to the leapd server. Likewise, `{background: true}` and `{background: false}`
have no effect or use in version 3. Leap version 3 also features the heartbeat described below.
# Leap Protocol Version 2
In this version a client establishes unique control of the Leap by sending `{heartbeat: true}` messages within 100ms
(1/10th of a second) often. The heartbeat is present in version 3 but removed in version 4.
Leap Version 2 does not have focus or background messages.
# Leap Protocol Version 1
No heartbeat, focus or background messages.
* * *
[1] _place_holder; _place_holder;(1 million microseconds == 1 second).