Processing Library Tutorial: JMyron

Elie Zananiri
last update :: 17/03/2008

introduction and setup

JMyron (aka WebcamXtra) is an external library for Processing that allows image manipulation without having to hard code everything. This is great because we can extend what we saw in the Video Library tutorial and add other aspects to it like motion tracking and color tracking. To use JMyron, you have to download the JAR and put them in the path/to/Processing/libraries folder. You must then import the library in each Processing sketch where JMyron objects will be used.

playing live video

The first thing we'll do is simply capture video from the camera and display it in the window. To do this we'll use a JMyron object. We have to call JMyron's start() method to start capturing and its stop() method to stop. It is extremely important to call JMyron.stop() when we're done or else we won't be able to call JMyron.start() next time we need to. It's a good idea to call JMyron.stop() in the applet's stop() function, so that we'll stop capturing when we quit the applet.

We'll use the JMyron.update() method in the draw() loop to refresh our image at every cycle. A call to the JMyron.image() method returns a one-dimensional integer array where each element represents the RGB value of a captured pixel. We still need to draw the image, so we'll copy each element from JMyron.image() to Processing's pixels[] array. This array must be loaded with loadPixels() so that we can access it, and saved with updatePixels() so that we can modify it. Finally, we'll turn off glob processing (which is used for motion tracking; see below) with JMyron.findGlobs(0) to improve our performance.

import JMyron.*;

JMyron theMov;

void setup() {
  size(320, 240);
  
  theMov = new JMyron();
  theMov.start(width, height);
  theMov.findGlobs(0);
}

void draw() {
  theMov.update();
  int[] currFrame = theMov.image();

  // draw each pixel to the screen
  loadPixels();
  for (int i = 0; i < width*height; i++) {
    pixels[i] = currFrame[i];
  }
  updatePixels();
}

public void stop() {
  theMov.stop();
  super.stop();
}

pixel manipulation

Next up we will see how to modify our image by analyzing and processing each pixel separately. We will display a negative version of our feed by going through all the pixels and subtracting each channel's value from 255 before displaying them on screen. We will extract the color channel values using Processing's red(), green(), and blue() functions, and we will display the resulting color using the built-in pixels[] array.

import JMyron.*;

JMyron theMov;

void setup() {
  size(320, 240);

  theMov = new JMyron();
  theMov.start(width, height);
  theMov.findGlobs(0);
}

void draw() {
  theMov.update();
  int[] currFrame = theMov.image();

  // draw each pixel to the screen
  loadPixels();
  for (int i = 0; i < width*height; i++) {
    float r = red(255-currFrame[i]);
    float g = green(255-currFrame[i]);
    float b = blue(255-currFrame[i]);

    pixels[i] = color(r, g, b);
  }
  updatePixels();
}

public void stop() {
  theMov.stop();
  super.stop();
}

We can make this more interesting by pixelizing our image. The following example will transform the video into a 20x20 grid of colours. We'll analyze a bunch of pixels at the same time and use their average colour as the corresponding cell's fill. Note that we are not using the pixels[] array anymore, but drawing rectangles of the corresponding colour.

import JMyron.*;

int NUM_SQUARES = 20;

JMyron theMov;
int sampleWidth, sampleHeight;
int numSamplePixels;

void setup() {
  size(320, 240);

  theMov = new JMyron();
  theMov.start(width, height);
  theMov.findGlobs(0);

  sampleWidth = width/NUM_SQUARES;
  sampleHeight = height/NUM_SQUARES;
  numSamplePixels = sampleWidth*sampleHeight;
}

void draw() {
  theMov.update();
  int[] currFrame = theMov.image();

  // go through all the cells
  for (int y=0; y < height; y += sampleHeight) {
    for (int x=0; x < width; x += sampleWidth) {
      // reset the averages
      float r = 0;
      float g = 0;
      float b = 0;

      // go through all the pixels in the current cell
      for (int yIndex = 0; yIndex < sampleHeight; yIndex++) {
        for (int xIndex = 0; xIndex < sampleWidth; xIndex++) {
          // add each pixel in the current cell's RGB values to the total
          // we have to multiply the y values by the width since we are 
          // using a one-dimensional array
          r += red(currFrame[x+y*width+xIndex+yIndex*width]);
          g += green(currFrame[x+y*width+xIndex+yIndex*width]);
          b += blue(currFrame[x+y*width+xIndex+yIndex*width]); 
        }
      }

      r /= numSamplePixels;
      g /= numSamplePixels;
      b /= numSamplePixels;

      fill(r, g, b);
      rect(x, y, sampleWidth, sampleHeight);
    }
  }
}

public void stop() {
  theMov.stop();
  super.stop();
}

motion tracking

Now we'll look at ways to track movement in our video feed. This is done with glob processing, which is a type of pattern matching. In our case, we're trying to match pixel colour values by analyzing their neighbours and previous values. This may sound complicated but it's quite easy with the JMyron library. All we have to do is turn on glob processing with JMyron.findGlobs(1), fine-tune the search with JMyron.minDensity(int val) and JMyron.maxDensity(int val) (which both take pixel-counts as input), and we're ready to grab some data. Data is returned as lists of globs in the form of multi-dimensional arrays with the first index always representing the globs.

The following are the functions that we can use with a brief explanation of their return arrays:

Here is an example that will draw bounding boxes around the globs. The fill color of the bounding boxes will be the average color of all the pixels. Instead of calculating this value like we did in the previous example, we'll use the JMyron.average(int left, int top, int right, int bottom) method that's built-in to the library and that does the same thing (notice its parameters are the four limits of a bounding box). This method returns a single integer representation of a color, so we'll have to use the red(), green(), and blue() functions to extract each channel. Finally, we'll use the JMyron.trackColor(int red, int green, int blue, int tolerance) method to determine which color range the globs should match. Since we are setting all these parameters to 255, it's like not setting any limits on the color matching.

import JMyron.*;

JMyron theMov;
int[][] globArray;

void setup() {
  size(320, 240);

  theMov = new JMyron();
  theMov.start(width, height);
  theMov.findGlobs(1);
  theMov.trackColor(255, 255, 255, 255);

  stroke(255, 0, 0);  // red outline
}

void draw() {
  theMov.update();
  int[] currFrame = theMov.image();

  // draw each pixel to the screen
  loadPixels();
  for (int i = 0; i < width*height; i++) {
    pixels[i] = currFrame[i];
  }
  updatePixels();

  // draw the glob bounding boxes
  globArray = theMov.globBoxes();
  for(int i = 0; i < globArray.length; i++) {
    int[] boxArray = globArray[i];

    // set the fill colour to the average of all colours in the bounding box
    int currColor = theMov.average(
      boxArray[0], 
      boxArray[1], 
      boxArray[0] + boxArray[2], 
      boxArray[1] + boxArray[3]);
    fill(red(currColor), green(currColor), blue(currColor));

    rect(boxArray[0], boxArray[1], boxArray[2], boxArray[3]);
  }
}

public void stop() {
  theMov.stop();
  super.stop();
}

We'll modify the last example slightly to draw the glob contours instead of bounding boxes. Since this is a little messier, we'll use JMyron.minDensity(int value) to filter our data. I'm also only going to track shades of black, partly because the room I'm in right now is dark. Notice that the globArray[][][] is now three-dimensional because we need to match the return type of JMyron.globEdgePoints(int segmentLength). The last thing that needs mentioning is that we have to make sure our array of contours is not empty before we start drawing shapes. This is so we don't get a null pointer exception the first time the program loops.

import JMyron.*;

JMyron theMov;
int[][][] globArray;

void setup() {
  size(320, 240);

  theMov = new JMyron();
  theMov.start(width, height);
  theMov.findGlobs(1);
  theMov.minDensity(100);
  theMov.trackColor(0, 0, 0, 100);

  stroke(255, 0, 0);  // red outline
}

void draw() {
  theMov.update();
  int[] currFrame = theMov.image();

  // draw each pixel to the screen
  loadPixels();
  for (int i = 0; i < width*height; i++) {
    pixels[i] = currFrame[i];
  }
  updatePixels();

  // draw the glob contours
  globArray = theMov.globEdgePoints(10);
  for (int i=0; i < globArray.length; i++) {
    int[][] contourArray = globArray[i];
    if (contourArray != null) {
      beginShape();
      for(int j=0; j < contourArray.length; j++) {    
        vertex(contourArray[j][0], contourArray[j][1]);
      }
      endShape(CLOSE);
    }
  }
}

public void stop() {
  theMov.stop();
  super.stop();
}

This final example will be our personal take on motion tracking, without using the built-in glob processing of JMyron. We will track movement in the frame by analyzing the pixel colors for the current and previous frames. If the values are different, it means we have movement. The following code tests the difference in the red, green, and blue channels of each pixel against the tolerance to judge whether or not the color difference is significant enough. We need to use a tolerance range in our analysis because color values will never be exactly the same. The tolerance value can be modified with the < and > keys, and the tracking can be toggled between motion and stillness using the spacebar.

import JMyron.*;

JMyron theMov;
int[] currFrame;
int[] prevFrame;
int tolerance;
boolean revealing;

void setup() {
  size(320, 240);

  theMov = new JMyron();
  theMov.start(width, height);
  theMov.findGlobs(0);

  // initialize the pixel arrays to avoid a NullPointerException
  loadPixels();
  currFrame = prevFrame = pixels;

  tolerance = 15;  
  revealing = true;
}

void draw() {
  // erase the previous image
  background(255);

  theMov.update();
  // save the last frame before updating it
  prevFrame = currFrame;
  currFrame = theMov.image();

  // draw each pixel to the screen only if its change factor is 
  // higher than the tolerance value
  loadPixels();
  for (int i=0; i < width*height; i++) {
    if (comparePixels(i)) {
      pixels[i] = currFrame[i];
    }
  }
  updatePixels();
}

boolean comparePixels(int index) {
  if (Math.abs(red(currFrame[index])-red(prevFrame[index])) < tolerance) 
    if (Math.abs(green(currFrame[index])-green(prevFrame[index])) < tolerance)
      if (Math.abs(blue(currFrame[index])-blue(prevFrame[index])) < tolerance)
        return !revealing;

  return revealing;
}

void keyReleased() {
  if (key == '.' || key == '>') {
    // increase tolerance
    tolerance += 2;
  } else if (key == ',' || key == '<') {
    // decrease tolerance
    tolerance -= 2;

  // toggle the revealing mode
  } else if (key == ' ') {
    revealing = !revealing;
  }
}

public void stop() {
  theMov.stop();
  super.stop();
}

references

WebcamXtra/Myron Official Project Page
WebcamXtra/Myron Discussion Forums