Software
We are aiming to code the front-end of the piece (graphics and animation) in Processing/Java and the back-end (blob detection and tracking) in openFrameworks/OpenCV, and have both components communicate with each other using the OSC protocol. The information to be transmitted will be very minimal: a list of blob IDs and xy-coordinates. If Processing turns out to be too slow, we will convert the front-end to openFrameworks/OpenGL code and have both components running in the same application.
Computer Graphics
The growth algorithm will consist of a central vine which follows the path drawn by the fingers. The direction of the growth is determined by generating a path following a finger as it travels on the surface. In order to keep the path as smooth as possible, the following algorithm is used to register the least amount of control points, using threshold values for the distance and slope between the last few points.
IF (the new point is far enough from the last key point) {
IF (there are less than 2 key points total) {
add a new key point at the end of the list;
}
ELSE IF (the slope between the new point and the last key point is
different enough from the slope between the last 2 key points) {
add a new key point at the end of the list;
}
}
The leafs/flowers particles will be created in an external graphics program, and imported as SVG graphics into flora. These will be placed along the central vine, rotated and scaled depending on their position in the overall plant. The particle animations will be computed automatically based on keyframes for the birth, alive, and dead states. Pre-rendered animations cannot be used because of the initial growth and nurturing features, which require being able to stop at any intermediate state between the birth, alive, and dead states. The keyframes will consist of positions for a fixed number of control points which determine the shape of the particle. Each control point will have a birth, alive, and dead position, and will tween from its current position to one of these targets depending on the triggered animation.
Surface
We will use a single horizontal surface as the touch interface and the display, which will be cut to custom shape as represented in the design document. This surface will consist of a sheet of acrylic covered with a thin film such as vellum or mylar. The purpose of the film is twofold:
• It will make the surface more pleasant to touch and will allow the fingers to move smoothly across it.
• It will act as a diffuser for the IR light if we end up using the Diffused Illumination technique for touch detection (see Sensing Device section).
Display Device
The application will be projected on the touch/display surface using a projector. Depending on the setup used (see Sensing Device section), the image will either be front-projected or rear-projected.
Sensing Device
Touch detection can be achieved using one of the two following options.
The first is to build a touch interface following the Diffused Illumination technique. This works by shining infrared light using IR emitters below the screen, which is covered by a diffuser such as vellum or mylar. When the finger touches the surface, it reflects more light than the diffuser, and can therefore be detected by the camera. The camera and projector used to display the software are both also placed below the screen. The advantage of using this technique is that we can accurately detect points of contact on the touch surface, and that we do not have to worry about shadows covering the projected image. The disadvantage is that the table needs to be built, and this will cost time and money.
The second option is to use the pre-existing setup at ITP. Because the projector and camera are on top of the surface, the user’s entire arm will be tracked and not just the point of contact with the screen. One solution to this problem would be to track from which edge the arm enters the tracked area and to assume the finger is on the opposite end. For example, if the arm comes in from the right, we know that the fingers are at the leftmost edge of the tracked blob. This however does not solve the problem of the shadows covering the projected image. We also cannot detect the moment when the finger touches or releases the screen, which is necessary for our interface. This can be solved by installing a contact mic on the surface, which could set a flag in the software whenever it “hears” the finger tapping the surface.
Computer Sensing
The computer will identify blobs using a background subtraction algorithm. The blobs will be tracked and identified so that their movement can be followed from frame to frame. This will be done using a customized version of Stefan Hechenberger’s extension to the openFrameworks OpenCV libraries. The way the tracker works is by comparing the current list of blobs with the list from a previous frame, and assigning the same ID to those that are near each other. This is limited by a maximum distance a blob can travel per frame, and inconsistencies are resolved by using more than one previous frame when performing the comparison. Once the blobs are identified, their position will be transmitted to the front-end software, which will generate graphics.
Points of Failure
• The background subtraction could stop functioning properly if the light conditions in the room change. If this is the case, we would need to research adaptive background subtraction techniques.
• The blob detection could be imprecise if the blobs and the background have similar colour. This potential bug can be eliminated if we use the Diffused Illumination setup.
• The software may lag if there are many input points and/or growths at the same time. If this happens using Processing, we can switch to openFrameworks for a speed increase. If the problem is still not solved, we may have to revert to OpenGL vertex buffers and display lists to perform more processing on the graphics card.
• The way the directions will be conveyed may not be as clear as we want it to be. User-testing and multiple iterations will eventually solve this problem.
View the PDF
