A Multimodal Media Madness project
by:
Sören Busch
Niklas Hoppe
Bahram Maleki-Fard
| Audio | Video |
|---|---|
| time/length | |
| speed(audio: bpm or samplerate, video: fps) | |
| tone frequency | colour values |
| volume | brightness/contrast/saturation |
| instrumentation | image size/resolution |
| rhythm | |
| Audio | Video | |
|---|---|---|
| Red Value | <-> | frequency or volume of 1st tone |
| Green Value | <-> | frequency or volume of 2nd tone |
| Blue Value | <-> | frequency or volume of 3rd tone |
| Brightness/Contrast/Saturation | <-> | Volume |
| Image Width | <-> | Audio Length |
| Image Heighth | <-> | Number of frequencies |
| Image quality (how blurry the image is) | <-> | Sound quality (sinus, sawtooth or square waves?) |
| framerate, rate of changes or movement | <-> | bpm or samplerate |
| colour | <-> | instrument |
First, we agreed to do a sonification. We also agreed that we wanted to do something that doesn't generate random tones that sound weird. Our mapping ideas in mind (especially the one with the instruments), we came up with the idea to do something that generates background music for a movie.
We want to cut the frames of a movie into 10 pieces (see picture). Every piece refers to an instrument. The upper pieces will be connected to a piano, the lower pieces will be connected to a fiddle and the overall picture will be connected to a bass or cello.
For every piece (and for the whole picture) our application will calculate the average colour value(s) and the average brightness value. The colours will be mapped to the pitch of the instruments and the brightness will be mapped to the volume. We have 11 instruments playing. A bass or cello for the background (connected to the overall average values) 5 fiddles for the lower pieces and 5 pianos for the upper pieces. In order to represent movement, we decided to give each instrument a separate balancing (see picture for percent values - Left/Right).
In case of the piano we have no possibility of a continuous tone, so for the piano pieces our application has to react to immense brightness changes within the pieces with an attack for the piano.
Our goal is to develop an application that generates background music that represents the mood of the movie in an adequate way. A smooth fade-in of a picture will lead to a smooth fade-in of the instruments, etc. Furthermore, we want that a (possibly blind) person can distinguish foreground from background and is able to hear movement.
To build our project we used Max/MSP and jitter, because their video and audio editing qualities seemed really nice and Quartz Composer seemed to be just too limited. The instruments should all be played through MIDI
The first thing we had to get done, was splitting the movie frames into ten parts and calculating the average RGB-Values of each of them (and the background). Discovering Cyclops made life a lot easier, because it is a really nice tool, which is just made for this job.
The most important part was calculating notes out of the given colour-values. We agreed to use 3 different 'harmonics', each of them 3 tones in 3 different octaves, and assign each of them to one of the colours red, green and blue. For every frame the dominating colour gets calculated and all instruments only play the notes, associated with that colour. This was done to prevent too extreme dissonances.
With a dominating colour given, our patch "rgb-converter" calculates a note for any given colour, based on the simple formula: Brighter Colours = Higher Notes. In the same way, the velocity is connected to the overall brightness. Now a note can be played and with "ctlout" it gets assigned the right balance (see picture above)
To calculate the attack times of the piano, we have another patch, called "attack". It takes a stream of RGB-Values and any time the colour-changes cross a given/adjustable tolerance, it sends out a bang. For the strings we tried several variations, like using a metronome to have them play all the time for a fix duration. In the end, we used change-based attacks for them as well, because it just sounds best.
This idea was used once again, when we added percussions. We used a modified "attack"-patch so that everytime the image brightens in a extreme way a cymbal is played or a tom if it darkens. We decided to use WAV-files for these sounds instead of MIDI, because it just sounds way better
This is the starting patch and for reasons of clearness, everything that isn't absolutely needed here is put into "core". It loads and displays the movie and tolerances can be adjusted.
This one is really a big mess, so to clear things up a bit, this is what happens:
The movie and the tolerances are received through the various inputs. Cyclops does its job and calculates the RGB-Values. The overall-RGB-averages are calculated as well and based on the a dominating colour is picked through "domcolour". All the instrument patches receive their tolerances, RGB-values and their respective midi-channel to further process the data
Those three all work the same way. They receive the data and calculate a note with "rgb-converter". The note (pitch and velocity) is saved in the "0"-boxes. If "attack" sends out a bang, that note is send out to makenote and it gets played in its MIDI-channel. At start set_balance and pgmout set the right balance and program for that channel
The heart of everything. It takes 3 colourvalues and, based on the dominating colour, only lets one of them through. For that colour-value one of 9 notes (3 notes in 3 octaves) gets sent out. There are 2 outlets, one for pitch and one for velocity
This one takes 3 colour values and a tolerance. It saves those values and if new ones are received it compares them. If the tolerance is crossed it sends out a bang
These two should be pretty self-explanatory
The final product works pretty much as we expected it to do. It doesn't produce real music, but it catches the mood of the movie nicely. You can hear things happening, which is pretty much what we aimed for
First thing we'd do, if we went on with the project would be completely abandoning MIDI and instead using samples of real instruments, because MIDI just doesn't sound good
Another thing that could be done, would be refining the calculation of the notes, so that in the end the output sounds like real music. This one should be really complicated
For reference on Max/MSP and jitter we pretty much relied completely on the documentation and the help files delivered with the programs
For testing purposes we used trailers, downloaded from the Apple-Website. The following ones worked particularly well:
Getting the patch to work is really simple: Just open the file "mmm.mxb" in Max/MSP, hit the read-button, select a movie to play and start playback by activating the toggle. Everything else is done automatically. If you feel, that attacks occur too often (or too few) in your movie, you can adjust the tolerances by dragging the number boxes.