Kinect Overview

Platform for Situated Intelligence supports reading from a Microsoft Kinect V2 Depth Camera (for the new Azure Kinect device, see the Azure Kinect component). This includes capture of video, audio, body tracking, and face tracking streams from the Kinect. A sample application here demonstrates its use.

Please note:
Support for Kinect is currently limited to Windows only.
Support for Face tracking via Kinect is limited to Windows 64 bit applications.

Basic Components

Basic Kinect capabilities are provided by instantiating a KinectSensor component which is part of the Microsoft.Psi.Kinect namespace.

Support for Kinect Face tracking is provided by instantiating a KinectFaceDetector component which is part of the Microsoft.Psi.Kinect.Face namespace.

Common Patterns of Usage

The following are some examples of how to use the Kinect sensor in \psi.

Capturing video from the Kinect device.

The following example shows how to create a KinectSensor and to receive images from the Kinect's color camera.

using (var pipeline = Pipeline.Create())
{
    var kinectSensorConfig = new Microsoft.Psi.Kinect.KinectSensorConfiguration();
    kinectSensorConfig.OutputColor = true;
    var kinectSensor = new KinectSensor(pipeline, kinectSensorConfig);
    kinectSensor.ColorImage.Do((img, e) =>
    {
        // Do something with the image
    });
    pipeline.Run();
}

Capturing audio from the Kinect device.

The next example shows how to receive audio from the Kinect and convert the audio stream into 16KHz-16b PCM format.

using (var pipeline = Pipeline.Create())
{
    var kinectSensorConfig = new Microsoft.Psi.Kinect.KinectSensorConfiguration();
    kinectSensorConfig.OutputAudio = true;
    var kinectSensor = new KinectSensor(pipeline, kinectSensorConfig);
    var convertedAudio = kinectSensor.Audio.Resample(WaveFormat.Create16kHz1Channel16BitPcm());
    convertedAudio.Do((audio, e) =>
	{
        // Do something with the audio block
	});
    pipeline.Run();
}

Performing face tracking with the Kinect device.

This final example demonstrates how to use the Kinect to perform face tracking. This simple example will print out for each face detected whether the person's mouth is open or closed. Note: That face tracking on the Kinect relies on enabling the body tracking, and thus we need to enable OutputBodies in the KinectSensorConfiguration.

using Microsoft.Psi.Kinect;
using (var pipeline = Pipeline.Create())
{
    var kinectSensorConfig = new KinectSensorConfiguration();
    kinectSensorConfig.OutputFaces = true;
    kinectSensorConfig.OutputBodies = true;
    var kinectSensor = new KinectSensor(pipeline, kinectSensorConfig);
    var faceTracker = new Face.KinectFaceDetector(pipeline, kinectSensor, Face.KinectFaceDetectorConfiguration.Default);
    faceTracker.Faces.Select((List<Face.KinectFace> list) => 
    {
       for (int i = 0; i < list.Count; i++)
       {
          if (list[i] != null)
          {
              string mouthIsOpen = "closed";
              if (list[i].FaceProperties[Microsoft.Kinect.Face.FaceProperty.MouthOpen] == Microsoft.Kinect.DetectionResult.Yes)
                  mouthIsOpen = "open";
              Console.WriteLine($"Person={i} mouth is {mouthIsOpen}");
          }
       }
    });
    pipeline.Run();
}

A Note About Coordinate Systems

The KinectSensor component emits all its calibration, joint, and body orientation information in the coordinate system basis of MathNet.Spatial. This is a different basis assumption from that used by the sensor technology underneath.

All coordinate systems are immediately rebased inside the component such that the X-axis represents "forward", the Y-axis represents "left", and Z-axis represents "up". All coordinate system information emitted by these components adhere to this basis.

    	Z
    	|   X
    	|  /
    	| /
 Y <----+