Thomas Knott (242993)
Christian Guggenmos (261701)
Andreas Hennings (229802)
Image Resonator
Max/MSP Patch (as zip file, including audio sample)
Basic Ideas
control: webcam
use current video image (or filtered version) as a resonator. Large scale = low frequencies. Dark objects are fixed, bright pixels move free in z-direction.
Vibrations are initiated from local movements.
To speed up, use only x-dimension of the image as a resonator (take average of y) – use y as activation dimension.
Output wave: get from left and right ends of the image.
How can it produce predictable sounds?
Imagine a dark room, and you handling some "glow in the dark" device (let's say a glowing drumstick or so). The system should then simulate how the air would vibrate in a pipe or guitar belly with this shape. The projected length of the device implicitly defines a wavelength, so the control is really somehow predictable.
The vibrations are initiated by movements in the image - with the glowing device, this means you have to shake it and move it around to make sound.
Simulation Details
We imagine as the physical model something different than air vibrations - maybe the vibrations of an elastic membrane spanning the canvas and trying to adapt to the local brightness value of the input image - with inertia to keep it swinging beyond the equilibrium, and friction to stop it after a while.
parameters like local elasticity and friction can be defined by colors in the image, for instance.
We also have to define which position of the image defines the output vibrations...
To make it faster, I think we have to reduce the number of pixels, maybe by restriction to one dimension.
Steps in implementing the Image Resonator
Finding a mathematical model for the simulation process.
Implementing the discretized differential equation in Max-MSP as a “feedback loop”.
Fine-Tuning of the parameters like friction (damping), elasticity, coupling spring constants, timestep
Finding a way to produce audio from the resonating image. As the simulated vibration was too slow to use as a signal, we chose to use it for manipulating the playback rate of audio samples. (which is far from our original idea, but at least it makes noise)
Problems
We had communication problems within the team, as the idea behind is highly mathematical, and not easily understandable.
MAX/MSP too slow for real-time audio signal generation. We should have seen this from the start, but we were bedazzled by our brilliant-seeming idea.
A C++ experiment (with a 16 pixel resonating image) showed that the hardware can do faster, but is still not fast enough. It would have been necessary to make at least one simulation step per audio sample, but this would require a factor ~128 faster processing.
One might think that this is a camera problem, but it's not – we intended to run multiple simulation steps per camera grab.
Tuning the parameters is not easy: Bad choice of parameters can drive the oscillation to infinity. To restore reasonable values, the patch has to be restarted.
Extensibility / Future Ideas
You could apply the image resonator to a disco where a camera films the dance area and people can influence the music by their movements
An interesting idea is also to use the system to create audio from a movie, which would be an “audiolization” of movie data.