The Music PuZzle
21.7.2005

The Music PuZzle is a project at the RWTH-Aachen as an assignment of the Media Computing Group. The music puzzle intends to be a new way to interact with music, letting the user learn about structures in musical pieces while having fun.

At the beginning the user is presented a complete piece of music which then breaks into pieces. The task presented to the user is to reassemble the musical piece just using sound.

Accomplished tasks

- Gesture interface through color recognition of two distinguishable colors

- Interchangeable musical piece in form of wave file

- Java Interface representing puzzle pieces

- Finding two colors which are recognized well

- Creation of gravity well within the java interface

- computing puzzle piece length for arbitrary music pieces

- connection of MAX/MSP patch with java interface

Some Pictures of the Software

Design Challenges/Problems

The most challenging aspect of this project was getting the gesture recognition to work as we wanted it to. The main problem here was that the color recognition we used in Max/MSP was very susceptible to background "interference", i.e. the recognition needs a controlled environment to grant best results. Also finding colors which were not present within the background and at the same time very independent of lighting proved to be a great challenge. Empirically colors with high values in two of the three RGB/channels were most suited for the task at hand.

Compared to the challenge with the gesture recognition the connection of the java interface with the Max/MSP patch was relatively easy. However, the correct mapping of the "grab" event to the change in one color/value was non trivial.

Also, puzzling musical pieces proved to be far more difficult than we initially thought as - since audio is a time-dependent media - the cognitive load of the user is quite high. It became clear that the music-puzzle would not be a game for short-time interaction - as would maybe be preferred for expositions - but rather an application which consumes a lot of time.

Possibilities of future work

There is a list of features we would have liked to implement, which look promising for future work:

Different play modi: cooperative play, competitive play

Different degrees of difficulty; shorter and less puzzle pieces for beginners, medium length and piece count for intermediate players and many pieces and split of musical piece into voices

Physical representation of puzzle pieces

Using .mp3 files as input

Improvement on the gesture recognition

Links to Source Code

MAX/MSP-Patch
Java GUI
Java Controller
Java Snippet

Project members

• Peter Radics
• Alex Lachmann