Processing Library Tutorial: Video

Elie Zananiri
last update :: 17/03/2008

introduction and setup

Processing makes it extremely easy to use video in your applets, whether it's a file loaded off memory or a feed coming from a USB or FireWire camera. The only extra software you need is Quicktime for Java. This should be included when you install the newest version of Quicktime/iTunes, so chances are you already have it on your computer. If not, download and install Quicktime/iTunes again and you should be all set.

loading and playing video files

To use a video file in your applet, you have to place a copy of it in the data folder of your project. Also, the videos you use must be compatible with Quicktime. MOVs and MPGs will work for sure, but you will probably need to download some codecs (especially if you're working on a Mac) to use AVIs. This is generally not recommended, as you cannot be sure other users have the same codecs. If you're not sure if your file will work, try loading it in the Quicktime Player. If it works there, it will play in Processing.

Unfortunately the video handling in Processing is buggy, so you should run some tests to see how the Video Library works out for you before launching into a time-consuming application implementation . For example, if your video file also has an audio track the audio will begin playing whether you start the video or not. This in turn influences the playhead position, so stops and pauses might not work as expected.

loading

Processing uses Movie objects to handle video files. This means that you have to load your movie file into a variable of type Movie before you can play it, display it, etc. This is done using the Movie() constructor. Here is an example of how this would be done where we are loading the dgma2005.mp4 file in a Movie object called theMov. Notice that we have to import the Video library before we use any Video objects or commands.

import processing.video.*;

Movie theMov; 

void setup() { 
  theMov = new Movie(this, "dgma2005.mp4");
}

playing and displaying

Next up we want to play the movie. The sound from the video starts playing automatically when the Movie object is created, but this will most probably not suit your needs. There are 2 options to play the video file: Movie.play() plays the movie once and stops on the last frame and Movie.loop() plays it over and over again. At this point, we will not see anything on screen because we need to tell Processing where to display our video. We use the image() function to do so, which accepts (in this order) the Movie to display, the x-coordinate of the video, and the y-coordinate of the video.

We now have a video that plays but the screen only shows the first frame and stops. We need to override the movieEvent() function, which gets called whenever a new movie frame is available. We will use the Movie.read() method to capture the current frame of the video.

import processing.video.*;

Movie theMov; 

void setup() { 
  size(320, 240);
  theMov = new Movie(this, "dgma2005.mov");

  /* only use 1 of the following options */
  theMov.play();  //plays the movie once
  //theMov.loop();  //plays the movie over and over
}

void draw() { 
  image(theMov, 0, 0); 
} 

void movieEvent(Movie m) { 
  m.read(); 
} 

The following list is an overview of some of the playback methods of the Movie object:

Here is an example that uses all these features. We will load the same file as before and display it in a Processing window that is double its size. The video will move around the frame following the mouse. The user will use the keyboard to control the playback: 'p' for play/pause toggle, 'l' for loop toggle, 's' for stop, and 'j' for jump (to a random position). We will use Movie.duration() in the jump to make sure we do not exceed the length of the video. Note that the sound will keep playing even if the video is paused; this is a bug.

import processing.video.*;
   
Movie theMov; 
boolean isPlaying;
boolean isLooping;

void setup() { 
  size(640, 480);

  theMov = new Movie(this, "dgma2005.mp4");
  theMov.loop();  //plays the movie over and over
  isPlaying = true;
  isLooping = true;
}

void draw() { 
  background(0);
  image(theMov, mouseX-theMov.width/2, mouseY-theMov.height/2);
} 

void movieEvent(Movie m) { 
  m.read(); 
} 

void keyPressed() {
  if (key == 'p') {
    // toggle pausing
    if (isPlaying) {
      theMov.pause();
    } else {
      theMov.play();
    }
    isPlaying = !isPlaying;

  } else if (key == 'l') {
    // toggle looping
    if (isLooping) {
      theMov.noLoop();
    } else {
      theMov.loop();
    }
    isLooping = !isLooping;

  } else if (key == 's') {
    // stop playing
    theMov.stop();
    isPlaying = false;

  } else if (key == 'j') {
    // jump to a random time
    theMov.jump(random(theMov.duration()));
  }
}

capturing video feeds

Capturing live video from a FireWire or USB digital camera works well in Processing and is straightforward to implement. The following tests were done with a FireWire camcorder; there should not be a problem using something like a USB webcam. If it works with Quicktime, it will work in Processing.

The one thing you have to be careful with is that your camera has to be connected and turned on before you start Processing. If it is not, the capture will not work and Processing will most likely crash.

setup

Live video works in a similar fashion as movie files, but it requires the use of the Capture object instead of Movie. The Capture() constructor takes as parameters a parent PApplet (typically this), the width and height of the frame, and optionally, the frame rate and the camera name (useful if multiple cameras are used). Here is a simple example where the captured feed is displayed in the Processing window. We use the Capture.read() method in the captureEvent() function to grab the current frame whenever it is available.

import processing.video.*;

Capture theCap; 

void setup() { 
  size(320, 240);

  theCap = new Capture(this, 320, 240);
}

void draw() { 
  image(theCap, 0, 0);
} 

void captureEvent(Capture c) { 
  c.read(); 
}

controlling the display

The following example creates a grid of stills from the video feed, which can be paused and restarted by the user with the spacebar. Note that the frame coordinates are determined in the captureEvent() function, so that they only increment whenever the next video frame is ready to be displayed.

import processing.video.*;

Capture theCap; 
int currX, currY;
int capWidth, capHeight;
boolean isCapturing;

void setup() { 
  size(640, 480);

  currX = 0;
  currY = 0;
  capWidth = width/5;
  capHeight = height/5;

  theCap = new Capture(this, capWidth, capHeight, 50);
  isCapturing = true;
}

void draw() { 
  image(theCap, currX, currY);
} 

void captureEvent(Capture c) {
  if (isCapturing) {
    c.read(); 

    currX += capWidth;

    // if we're at the end of the line...
    if (currX >= width) {
      // ...go to the beginning of the next line
      currX = 0;
      currY += capHeight;

      // if we're at the bottom of the window...
      if (currY >= height) {
        // ...restart from the top
        currY = 0;
      }
    }
  }
} 

void keyPressed() {
  if (key == ' ') {
    isCapturing = !isCapturing;
  }
}

This next example is a slight modification of the previous one, where the entire image is composed at different points in time, using the Capture.crop() method.

import processing.video.*;

Capture theCap; 
int currX, currY;
int capWidth, capHeight;
boolean isCapturing;

void setup() { 
  size(640, 480);

  currX = 0;
  currY = 0;
  capWidth = width/5;
  capHeight = height/5;

  theCap = new Capture(this, width, height, 30);
  isCapturing = true;
}

void draw() { 
  // keep the motor running
}

void captureEvent(Capture c) {
  if (isCapturing) {
    c.read(); 
    c.crop(currX, currY, capWidth, capHeight);
    image(c, currX, currY);

    currX += capWidth;

    // if we're at the end of the line...
    if (currX >= width) {
      // ...go to the beginning of the next line
      currX = 0;
      currY += capHeight;

      // if we're at the bottom of the window...
      if (currY >= height) {
        // ...restart from the top
        currY = 0;
      }
    }
  }
} 

void keyPressed() {
  if (key == ' ') {
    isCapturing = !isCapturing;
  }
}

references

Processing Video Library Reference