Data Tutorial 3: Gesture Capture

Introduction

This tutorial focuses on data capture and playback – in essence, we will be performing sequencing. However, unlike the sequencing found in the MIDI Tutorials, this is sequencing of arbitrary data (mouse movement, in this case). We will be expanding on our use of coll, this time collecting as well as reproducing data for an interesting drawing patch.

The heart of many performance tools is the ability to capture and play back movement, whether motion capture data, sensor data or mouse movement. This tutorial will present the basis for capture and playback systems, and will introduce a few interesting design patterns as well.

To open the tutorial patch please open 03dGestureCapture.maxpat from the the zip archive, which is available for download at the top of this page.

Run the program, view the data

Take a look at the tutorial patcher. This is a fairly complex patch, and the whole thing serves one purpose: capturing mouse movement and turning it into an interesting drawing. The interesting thing is that while receiving mouse movements and drawing the picture, the patch is also recording the movements. Because this patch depends on regular timing regardless of other computer activity, you may want to turn on Overdrive under the Options menu.

An important hint: you don’t want to interact with this patch using the mouse, since it will tend to mess up your drawing. Rather, notice the key object-based section just above the lcd object: this maps three keys (space, “r” and “p”) to three patch functions (“clear”, “record” and “playback”). To begin, hit the “r” key (lower-case is important!) to begin the recording process. Now, move you mouse around the screen – looping and large movements are especially effective. When you are done, hit the “r” key again to turn off recording.

Now you have an interesting, script-like drawing on the lcd display. Hit the space bar to clear the display, then hit the “p” key to begin playback. You will see the drawing occur: not only was the drawing captured, but the actual mouse movement (and timing) were captured and reproduced.

There are three coll objects in this patch, but all of them refer to the same data: the data in a coll named thegest. (The -1 argument prevents Max from searching for a file of that name when the patch is loaded.) The copy at the right is most convenient to access, so double-click on it to reveal the captured data. The contents of the coll shows our movement being tracked and captured in 20 millisecond intervals. Let’s look at how this capture happened.

The recording part

• Double-click the recordpatcher (p) object to open it.

The RECORD section of the patch is started with the big red toggle object. When it is turned on, it resets the patch (via the sel1 object connected to the right outlet of the trigger), then starts a metro that fires every 20 milliseconds. This determines the sampling rate of the mouse tracking, and also serves as a trigger for the other objects. The metro sends a bang message to two objects: a subpatcher (the scaled version of WTHITM that we've been using) and an int (abbreviated "i") object.

The WTHITM abstraction should be familiar by now – it tracks the current mouse position, and outputs a set of X and Y coordinates scaled to the range given as arguments. In this case the subpatcher will output values in the range of 0 - 320 and 0 - 240 any time it receives a bang message. In order to provide immediate feedback, the output of the subpatcher is sent to a drawing subpatcher that generates drawing commands for the lcd object.

The int object attached to the metro, and the + object connected to it, perform a counting function. Incoming bang messages will output the current value (as seen in the number box). However, the output value will also go to the + object, which will add one to the value and store it in the right inlet of the int (i) object to wait for the next bang message. The result is that every bang message will output a value, but also increment it by 1 – sort of like a counter object, but with no upper limit. This is a useful way to get an ever-increasing number in situations where you don’t know what the limits might be.

The values from these two objects (the current count and the scaled X and Y positions) are assembled with a pack object and then sent to the coll object - the first value in the list (the current count) acts as an index, telling coll to store the remaining values at that position. As a result, the coll acts as a data recorder, taking the indexed values and storing them for later use. The collection of data continues until the record function is turned off, either by clicking on the red checkbox, or by hitting the “r” key.

Take a moment to look at what the select1 object connected to the recording trigger does. When the recording is begun, a bang gets sent to a second trigger object that, from right to left, turns off the "play" toggle box (stopping any playback), sends a clear message to the lcd, erases the contents of the coll (with another clear message), and zeroes the int object to reset the index counter.

The playback part

• Double-click the playpatcher (p) object to open it.

The playback part of the patch mirrors much of the recording section, except it doesn’t do any mouse tracking. Rather, it just requests indexed values from the coll named thegest, and uses the same drawing patch to recreate the mouse drawing movements. It is able to use the same incrementing integer technique used in the record section; this has the benefit of not requiring an upper limit to be defined, but a downside of never stopping or looping on its own.

Since the record function didn’t timestamp the captured data (it just indexed it with the output of the integer object), it means that we can manipulate the playback of the drawing by altering the speed at which we request the coll ’s content. The easiest way to do this is to change the speed of the playback metro.

Unlock the patch and add an integer number box in the area of the playback section. Connect its output to the right inlet of the metro (the right inlet controls the output interval). Enter 2 into the number box, then hit “p” for playback. The drawing will occur as before, but will run 10 times as fast. Stop the drawing (hit "p") and change the number box to 40, hit the “p” key again, and the drawing will run at half speed. As you can see, this data capture technique gives you a lot of flexibility at playback time.

Once you have an interesting data capture, you will probably want to save it for later use. The coll at the far-right (outlined in gol) shows the simple method for saving and retrieving coll contents; a read message will allow you to import data from disk files, while a write message will take your captured data and store it to disk. You can append an actual file name and path to the message if you have a specific file to read or write, while a read/write message with no arguments will open a file dialog.

Make a few drawings and save them using the write message to the coll on the right. If you read them back in using the read message, you will be able to play back your saved drawings. Notice that the data files created by coll objects are simply text files; you can open then in any text editor or import them into another program for viewing or manipulation.

Summary

In this tutorial, we’ve seen how to use the coll object as a data collection device and we've learned how to design a counter with no upper limit. We’ve also seen how the coll can be used for disk storage and retrieval, allowing us to save and reuse our captured data. Finally, this patch is a good example of a single data set being used by three coll objects, a method for data sharing that becomes very useful with large patches.

See Also

• coll - Store and edit a collection of different messages