https://vimeo.com/311367320
This project is a basic implementation of a motion interface for the software synthesiser Maximillian, using openFrameworks. It uses the webcam on a laptop to track motion and links the data received to various (pretty arbitrary at the moment) parameters of the synth. As well as all the noise, it also provides a bit of visual feedback on the data, and a camera image so the operator has a chance of developing a strategy for playing it.
I’m interested in developing gestural means of controlling or performing with software, especially creative software, as it’s this element of physical expression which often seems missing from the relationship between performer and machine. Equally I’m interested in the idea of ‘features’, particularly in how they can be abstracted a long way from categories a human observer might use to describe an image or a sound.
This example uses a standard technique of frame-differencing – subtracting a frame of video from the previous frame – on the assumption that any identical (ie static) pixels will be removed. It’s a fast algorithm and provides a lot of information, particularly applied to sections of the image a time, so regions of the image then can be mapped to different events. I’d hoped to use frame-differencing a first step to calculating a histogram of oriented gradients, but on the frame-differenced image, so a kind of histogram of movement. This seemed like it could be a good way to react to both the shape of a gesture and its development in time.
This turned out to be a bit too ambitious and in the end I relied on the averaged movement values across tiled segments of the image to drive the synth, though i’d like to come back to this and properly implement the HOG feature. I speent a lot of time rummaging around the underlying memory operations of objects like vectors and deques, as the program would consistently crash when i got things wrong (does a vector make a copy of an object or move it or reference it and what happens if you then reference the original object, or some other part of the framework wants to delete it..)
So it doesn’t feel like a finished piece but it would be good to be able to improve on it if I get the time.