Project: Music Garden

IDEA

The idea of the project is to represent music as a garden. Music is played depending on which flowers are in the garden and how far they have grown. Every flower will grow on its own but the growth can also be affected by the user adopting the role of a gardener.
Interaction between the user and the garden takes place with a real watering can / garden scissor.

So there are two scenarios:

1. If there is no user interaction, the garden evolves naturally (flowers die after a while and new ones emerge). This is more or less audio-visual fortuity.
2. By interacting with the garden the user is able to create an individual sound scenery.

CONCEPT

Detailed Description

Each flower type (Figure 1) is associated with a certain instrument category (flower type 1 - guitars, flower type 2 - drums, flower type 3 - horns, flower type 4 - piano). That means that each plant is associated with an instrument out of the flower types' category (e.g. the red tulip, that belongs to flower type 1, would be associated with a guitar sound). Furthermore each flower type is situated on a fixed position (Figure 2) so that the user is able to recognize the connections flower type - instrument category. To make this process of recognition easier, the sound of the two plants on the left (right) is output on the left (right) channel.

Figure 1, the four flower types finally used

Figure 2, arrangement of the four flower types

Scenario 1: Natural Evolving

In this scenario the garden evoles naturally, what means that flowers run through their life cycle of emerging, blossoming and degenerating (Figure 3).
The volume of their sound corresponds with the state of the plants' life cycle, i.e. the volume grows with a flower and sinks with it's degeneration. After the death of a flower a new one of the same type emerges and with it a new sound.

Figure 3, life cycle of a flower

Scenario 2: User Interaction

In this scenario the user is able to take control about what is going on in the garden. After a short time the user should be in the position to tell which instrument category and flower type belong together. By then he can make use of well known, real world tools as a watering can and a garden scissor to create the music he likes (the following should be understood as altering both, a flower and it's sound):

• watering an evolving flower to hurry growth up and maintain it over a longer period of time
• watering a degenerating flower to reprogress in life cycle
• cutting an evolving or degenerating flower until its complete removal

TASKS ACCOMPLISHED

• Created patch to coordinate audio (volume) and video (plant's state of life) including "watering" and "cutting"
• Created patch to track the gardening tools
• Connected gardening tools to Max/MSP using Teleo
• Found a way to import plants from PlantStudio (a free software that comes with a large plant library and is able to export plants in every state of their life cycle as 3D models) with GEM (first export with PlantStudio, then convert with Biturn)
• Created a set of 16 flowers (à 30 life cycle states) with PlantStudio and exported and converted them
• Designed and rendered an appealing background image with Vue Esprit
• Created patch to put all visual stuff together
• Figured out how to import sound samples from Garageband to Max/MSP
• Created a set of sound samples for each instrument category we chose (guitars, drums, horns, piano)
• Created patch to play sound samples and control volume

CHALLENGES

Constructing a patcher for the tracking of the two gardening tools was quite similar to an assignment we already built during our lab, so we just modified it, added a few mathematical expressions and a "one click automation" to assign the color to the jit.bounds object.
But then we had to deal with different problems like bad colors and so on. Our main problem was, that we wanted the watering can and the scissors to be intuitive - so we could not use colored dots and let the camera get their distance because the camera can only see the scissors from one side and if a user decides to use it bottom up or even worse: Bottom to camera, what should we track then? So we forbeared using tracking to determine wheter the scissors were used or not.
In the case of the watering can we also had an idea of tracking it's use by scanning it's height. So we thought, that we could be able to see, when it becomes greater. But this method was unrelieable in a too high degree for our project, so we used a teleo pushbutton in both instruments. More intuitive in the scissors and kind of unattractive on the can's head.

After finding PlantStudio, we exported some models (in Wavefront OBJ format) and tried to import those with Jitter. This was not possible although Jitter is basically able to read that format. This was the reason we had to switch to GEM, which is an open-source graphic library for Max/MSP and PureData. Unfortunately, GEM was not able to load those models, either. So we had to convert the models with a free conversion tool (from Wavefront OBJ to Wavefront OBJ) whichs sounds silly but was necessary.
Another problem that came up was, that our patch did not run smoothly at all times. This was caused by the fact that we had to load the model of every state of a flower from hard disk which is clearly inferior to the way of loading all needed models at once. GEM provides an object ("multimodel") which could do that but, unfortunately, this object is not working currently.

At the beginning it was a great challenge for us to make up a way to create harmonic music. If you want to do this by the classic approach, you must even observe many musical laws in order to create a harmonic melody for only one instrument. And what we had in mind was much more difficult: we wanted the harmony to stay even if we took a sound/melody (at that point we did not knew exactly) out and brought a new one in. So we had to come up with something new and because we all knew the famous "Magix Music Maker" we took a look at GarageBand and were pretty pleased with the results.

FUTURE WORK

• Improve tracking of the gardening tools and make use of watering can more intuitive
• Improve loading of the models (piece of cake if multimodel is implemented correctly sometime)
• Change/improve the way music is created (e.g. build something like a realtime composer which is capable of creating harmonic music on the fly)

REFERENCES

• 3D models created with PlantStudio and converted with Biturn
• Background image rendered with Vue Esprit
• 3D scene built together using GEM
• Sound samples created with GarageBand

DOWNLOAD