In this section we'd like to give some information about what's happening in the background inside the patch when you play the Draw Piano and what kinds of problems we encountered during the development.
The first thing we had to think about was how we should generate the music. Should we make use of pre-recorded WAVE-files or FM-synthesized music? With the latter Immo experimented but we didn't really end up with sounds that do not annoy the audience after some time...
So we decided to make use of MIDI-instruments which provide a variety of different sounding devices. They also vary a bit in playing the individual sample (e.g. piano vs. drums or synthie pads). As we wanted to detect the mouse mouvements, we made use of a mouse-adapter provided by the MAX/MSP package. We completely made use of this package because we did not need to make use of image recognition elements (Cyclops) or manipulate digital samples (Jitter). We found that MIDI-instruments represent the best solution for our purposes: To get (relatively) good sounding notes out of our device with a dynamic component so that certain parameters can be changed at runtime (see below).
There are some calculations made that are performed in the background and that do not become obvious while you are playing. So one issue is the explicit screen mapping, that means that we fit the pitch, velocity, metronome and left-right-half-position calculation to the actual resolution of the smartboard. A second issue is the removal of the semi-notes within an octave because with them left, it becames harder to hit the "good-sounding" notes and, of course, the drawing area becomes denser and therefore the notes are harder to hit. That's another reason why we decided to only make use of five octaves. Besides, they provide a good accustical basis for most of the instruments. The next task to perform was a smoothing of the values. As we read out the actual screen position, it is not recommended to directly map the position coordinates to the values we need for our instrument. So we had to to think about calculations in order to achieve an equi-distant note distribution over the y-axis and also equi-distant intervals for the velocity and metronome values. The latter now has a scale ranging from 500ms down to 0ms in steps of 100ms. The velocity changes in steps of ten. This was needed because otherwise the values would always change slightly while you move the pen. This is a non-desired effect. As mentioned before, also the pitch values are remapped from the input values. So we start with an offset of 36 (deep "c") and every 768/(5×7) pixels a new note region begins. The 768 describes the height value of the resolution, we make use of 5 octaves with 7 full notes in them.
Now we had to find a way to influence the velocity dynamically at play time. No big deal: Just make use of the x-axis and smoothen the values to equi-distant intervals.
But then the next problem occurred: While drawing, all the notes in between the first and the next desired note are played. This is not always desired - especially if you want to play a more sophisticated piece. So we made use of the metronome in order to create a little gap between two selected notes. But in terms of dynamic playing, the musician may want to change that gap. This is where the dynamic metronome value comes into the game. But all the possible axes were already used and because of limitations of the SMARTboard drivers and no existing interface to Max/MSP we could not simply use another pen. Furthermore, the SMARTboard is not pressure-sensitive, so using the pressure as a third input parameter was not option either. Finally we got the idea of splitting the x-axis into two parts. The functionality of the two axes is not described here again. See the manual section for that.
Then Immo wanted to be able to control the sustain of a note. That means, as long as you remain on the screen at certain position, the note should be played for a pre-determined time. That's why we make use of the sustain slider in the top-left part of our patch. Because of the lack of better input devices like pressure-sensitive pens, for example, we chose that static solution. However, what is important to mention is that the played note stops immediately when you release the pen from the screen. When you drag it around (drawing mode) all the other notes until the last one under your current position are played with the pre-determined sustain value (until the note-off signal is sent via the pipe object).
The final tasks on the one hand were to find appropriate instruments for staccato and drawing mode. We recommend to make use of piano-like instrument for the first and synthesizer pads for the latter mode. Then we had to create some interface so that the musician can get in touch with the instrument. He should be enabled to directly see what he's doing without always watching the cryptic cord and reading out the values displayed. This is why we make use of a fullscreen image that exactly fits the screen resolution and the mentioned intervall. So you can directly see what note at what velocity you are playing and what the tact rate is. However, you have to decide what sustain time you want to give to your instrument beforehand. But as described, this slider is a static component and cannot be changed while playing.
We think reading through this sections also marks out what challenges we encountered in order to create a feeling for the musician to be directly involved and intuitively knowing what he is doing and how to perform a piece.
If the SMARTboard technique was a little more sophisticated or drivers were available for MAX/MSP which give information about more than one pen position and its pressure it would be possible to think of some extensions of the Draw Piano:
- For example, multi-user support could be added so that each player plays a different instrument depending on the pen he's holding (for example distinguished by the color of the pen).
- The axes could be remapped. Whereas the pitch direction remains, pressure-sensitive pens would make it possible to influence the velocity of the current note to be played.
- The x-axis could then be used to influence the sustain value for each note dynamically.
- One could make use of some tracking software that tracks the movements and different parameters in order to generate note sheets or any other exchangeable formats so that others could replay self-made pieces or just listen to them in a playback mode.
|
||
 |
The colors indicate the different notes, the velocity is controlled in the left half of the x-axis, the metronome in the right one. Only one value of the latters can be changed at a time - the other remains at its last value.