Kinect & Processing: A Beginner's Tutorial

Hey guys! Ever wanted to dive into the world of interactive art and design? Combining the Kinect sensor with Processing, a flexible software sketchbook and a language for learning how to code within the context of the visual arts, offers a fantastic way to do just that. This tutorial will guide you through the basics of getting your Kinect to talk to Processing, opening up a realm of possibilities for creative projects. So, buckle up, and let’s get started!

What You'll Need

Before we jump into the code, let's make sure you have everything you need:

• A Kinect sensor (either the original Kinect for Xbox 360 or the Kinect for Xbox One). Also, you can use Azure Kinect.
• The Kinect SDK (Software Development Kit) for your Kinect version. For the original Kinect, you'll need the Microsoft Kinect SDK. For the Kinect for Xbox One, you'll need the Kinect for Windows SDK 2.0. For Azure Kinect, you need Azure Kinect SDK.
• Processing installed on your computer. You can download it from the official Processing website.
• The SimpleOpenNI library for Processing. This library allows Processing to communicate with the Kinect. For Azure Kinect, you may need another library.

Setting Up Your Environment

First, you've got to install Processing. Head over to processing.org and download the version that matches your operating system. The installation process is straightforward, just follow the instructions on the site. Once Processing is installed, launch it. Next, we need to install the SimpleOpenNI library. In Processing, go to Sketch > Import Library > Add Library. Search for SimpleOpenNI and install it. This library acts as the bridge between Processing and your Kinect, allowing them to communicate effectively. If you are using Azure Kinect, you might need a different library. Search the library in Processing.

With Processing and SimpleOpenNI set up, it's time to connect your Kinect. Plug your Kinect into a USB port on your computer. Depending on your Kinect version, you may also need to plug it into a power source. Once connected, your computer should automatically recognize the Kinect. If not, make sure you've installed the Kinect SDK correctly. The SDK provides the necessary drivers for your computer to communicate with the Kinect. Now, let’s ensure that everything is running smoothly. Open Processing and navigate to File > Examples > Libraries > SimpleOpenNI. Choose one of the example sketches, such as "SimpleOpenNI". Run the sketch. If everything is set up correctly, you should see a window displaying the Kinect's depth data, showing a live feed of what the Kinect sees in shades of gray based on distance. If you encounter any errors, double-check that you've installed the Kinect SDK and the SimpleOpenNI library correctly.

Writing Your First Kinect + Processing Sketch

Okay, let's dive into writing some code! We'll start with a simple sketch that displays the depth data from the Kinect. This will give you a visual representation of the Kinect's capabilities and how Processing can interact with it. First, open Processing and create a new sketch. Here’s the basic code structure:

import SimpleOpenNI.*;

SimpleOpenNI context;

void setup() {
  size(640, 480);
  context = new SimpleOpenNI(this);
  context.enableDepth();
  context.enableUser();
  context.setMirror(true);  
}

void draw() {
  context.update();
  image(context.depthImage(), 0, 0);
}

Let's break down this code:

• import SimpleOpenNI.*;: This line imports the SimpleOpenNI library, allowing us to use its functions in our sketch.
• SimpleOpenNI context;: This declares a SimpleOpenNI object named context. This object will be our interface to the Kinect.
• size(640, 480);: This sets the size of the Processing window to 640x480 pixels, which is the resolution of the Kinect's depth image.
• context = new SimpleOpenNI(this);: This creates a new SimpleOpenNI object, passing this (the current sketch) as an argument.
• context.enableDepth();: This enables the depth data stream from the Kinect.
• context.enableUser();: Enables the user tracking capabilities of the Kinect, allowing the system to identify and track human figures within the scene. User tracking is essential for creating interactive installations and applications that respond to human movement and gestures.
• context.setMirror(true);: Mirrors the image, so it's easier to use.
• context.update();: This updates the Kinect data, retrieving the latest depth information.
• image(context.depthImage(), 0, 0);: This displays the depth image from the Kinect at the top-left corner of the Processing window.

Copy and paste this code into your Processing sketch and run it. You should see a grayscale image representing the depth data from the Kinect. Closer objects will appear brighter, while farther objects will appear darker. This is a raw depth image; we'll refine it and make it more interactive later.

Accessing Depth Data

Now that we can display the depth data, let's learn how to access it directly. This will allow us to create more sophisticated interactions and effects. The depth data is stored in an array of integers, where each integer represents the distance from the Kinect to a specific point in the scene. To access this data, we can use the depthMap() function. Modify your sketch to include the following code:

import SimpleOpenNI.*;

SimpleOpenNI context;
int[] depthData;

void setup() {
  size(640, 480);
  context = new SimpleOpenNI(this);
  context.enableDepth();
  context.setMirror(true);
  depthData = new int[640 * 480];
}

void draw() {
  context.update();
  depthData = context.depthMap();

  loadPixels();
  for (int i = 0; i < depthData.length; i++) {
    int depth = depthData[i];
    pixels[i] = color(depth);
  }
  updatePixels();
}

Here's what we've added:

• int[] depthData;: This declares an integer array named depthData to store the depth values.
• depthData = new int[640 * 480];: This initializes the depthData array with a size of 640x480, matching the resolution of the depth image.
• depthData = context.depthMap();: This retrieves the depth data from the Kinect and stores it in the depthData array.
• The loadPixels(), for loop, and updatePixels() functions are used to draw each pixel based on its depth value. This is a more direct way to visualize the depth data.

With this code, you're now directly accessing the depth data and using it to draw the image. Each pixel's grayscale value is determined by its distance from the Kinect. Now that we can access the raw depth data, we can start doing some cool things with it. For instance, we can filter the data to only show objects within a certain range, or we can use the depth data to control other parameters in our sketch.

User Tracking

One of the coolest features of the Kinect is its ability to track human users. This opens up a world of possibilities for creating interactive installations and games. Let's add user tracking to our sketch. Modify your code to include the following:

import SimpleOpenNI.*;

SimpleOpenNI context;
int[] depthData;
int userId = 0; 

void setup() {
  size(640, 480);
  context = new SimpleOpenNI(this);
  context.enableDepth();
  context.enableUser();
  context.setMirror(true);
  depthData = new int[640 * 480];
  context.requestFocus(this); 
}

void draw() {
  context.update();
  depthData = context.depthMap();

  if (context.isNewUser()) {
    userId = context.getFirstID();
    println("New user detected with ID: " + userId);
    context.startTracking(userId);
  }

  loadPixels();
  for (int i = 0; i < depthData.length; i++) {
    int depth = depthData[i];
    pixels[i] = color(depth);
  }
  updatePixels();

  // Draw a bounding box around the user
  if (userId > 0) {
    PVector jointPos = new PVector();
    context.getJointPositionSkeleton(userId, SimpleOpenNI.JOINT_HEAD, jointPos);
    PVector projPos = new PVector();
    context.convertRealWorldToProjective(jointPos, projPos);
    
    float x = projPos.x;
    float y = projPos.y;

    ellipse(x, y, 50, 50); // Draw a circle at the head position
  }
}

void keyPressed() {
  if (key == ' ') {
     context.requestCalibrationSkeleton(userId, true);
  }
}

void onNewUser(SimpleOpenNI curContext, int id) {
  println("New user detected with ID: " + id);
  println("Press space to calibrate user: " + id);
  context.startTracking(id);
}

void onLostUser(SimpleOpenNI curContext, int id) {
  println("Lost user: " + id);
}

void onFocusGained(SimpleOpenNI curContext) {
 println("Focus gained");
}

void onFocusLost(SimpleOpenNI curContext) {
 println("Focus lost");
}

Here’s what this code does:

• context.enableUser();: Enables user tracking in the Kinect.
• context.isNewUser(): Checks if a new user has entered the Kinect's field of view.
• context.getFirstID(): Retrieves the ID of the first detected user.
• context.startTracking(userId): Starts tracking the specified user.
• context.getJointPositionSkeleton(userId, SimpleOpenNI.JOINT_HEAD, jointPos): Retrieves the 3D position of the user's head.
• context.convertRealWorldToProjective(jointPos, projPos): Converts the 3D position to 2D coordinates that can be used in the Processing sketch.
• ellipse(x, y, 50, 50): Draws a circle at the user's head position.
• context.requestCalibrationSkeleton(userId, true): Allows you to calibrate the skeleton tracking by pressing the space bar.
• onNewUser, onLostUser, onFocusGained, onFocusLost: These functions handle events related to user detection, loss, and focus changes.

When you run this sketch, it will display the depth data as before. Additionally, it will detect and track any users in the Kinect's field of view, drawing a circle around their head. This is a basic example, but it demonstrates the power of user tracking. You can expand on this by tracking other joints, such as hands and feet, and using this data to control various aspects of your sketch.

Conclusion

Alright, guys! You've now taken your first steps into the exciting world of Kinect and Processing. You've learned how to set up your environment, display depth data, access raw depth data, and track users. These are the fundamental building blocks for creating interactive installations, games, and art projects. The possibilities are truly endless!

Don't be afraid to experiment and explore. Try modifying the code, adding new features, and combining the Kinect with other sensors and technologies. The more you play around, the more you'll discover. So, go forth and create something amazing! This journey has only just begun, and I'm excited to see what you'll come up with. Happy coding!