DIS II - Final project proposal - Hop-A-Tone™

Introduction

Usually creating music involves using your mouth, arms or digits. Wouldn't it be an interesting experience to create music with just your feet for a change? By using a dance mat, you can select the rhythms of your choice and can accompany the background beats with your own musical composition. To make compositing your own music even more rewarding, our input device reacts to certain input sequences of yours (also known as "Combos" in professional circles), and adds its own little tunes. Sounds interesting? Keep on reading for a more detailed overview...

Motivation

Usually, while listening to music playing, people react with feedbacks, either consciously or subconsciously..
This feedback includes stepping the feet on the floor or moving the, the foot up and down. Another more complex feedback would be moving the feet harmonically with the same speed as the music -- basically dancing at its simplest form. By considering this natural reaction of human beings we thought of getting use of the Dance Mat, which fits this natural concept and also includes having fun and excitement while jumping up and down and hearing the produced music from it.

The device

The Hop-A-Tone musical instrument uses a standard issue dance mat - available on E-Bay for only a couple of bucks: Image of a dancing mat
The model we use has 10 buttons: The SELECT and START buttons, and the 8 action buttons that form a circle around the center.

Usage

We have decided to go with a 2-modal input. Another possible input method is also described, but was discarded. A very early implementation exists, though.

2-Modal Mode

In this input method the dance mat works in two modes, which you can toggle by using the SELECT button:

When in rhythm mode, you can switch between different background rhythms simply by stepping on one of the 8 action buttons. The rhythm will keep playing until you select the currently playing rhythm again.

When in compose mode, you can play the different keys C,D,E,F,G,A,H,C by stepping on one of the 8 action buttons. If you want to play a chord, just push two buttons belonging to one C-Major chord at the same time. The device will then add the third note by itself.

Mixed Mode (implemented, but not used)

Step in the middle of the mat. Now, step on a button of your choice with both feet. This selects the background rhythm you will be using. Each of the 8 buttons plays a different beat. Should you wish to change the background, simply put your feet on another button. Should you ever wish to stop any rhythm, place both your feet in the center of the mat.
Should you wish to change the rhythm on the fly, just step with one foot to the button triggering the new rhythm, and then lift the other foot from the previous button. The rhythm will then change to the your newly selected one.

While keeping one foot on the button which selects your background rhythm, use the other one to play notes. The note which gets played is always relative to your own position: From your rhythm-button clock-wise, the buttons play the notes C, D, E, F, G, A, H.
Don't worry about keeping the rhythm running: As long as you keep one foot on the button, it will keep playing.

While this method of input sounds cool in theory, it's quite confusing in practice and unnecessary since you don't really change the background that often during composition.

Completing combos

To make the dancing experience a little bit more rewarding and to convey a feeling of accomplishment, it is possible to program combos into our instrument. This means that once you've completed one of a few specific sequences of notes within a certain time period, the instrument will react by "answering" to your input. Why don't you try it out yourself?

Combo Reaction
E, G, C2 Ha Do Ken!
A, G, D, D Speed of Sound

Challenges

A rough outline of our milestones:

In our first brainstorming session, we came up with fantastic ideas like if you hop a combo, the computer finishes the music sequence you started, or the background rhythm is adapted to the speed of your played music. But soon we realized these thoughts were very "mission impossible". Therefore we concentrated on our main goal: producing music and not producing wonderful sounding music ;-)

From a technical point of view, we needed a special USB driver to get the mat running, since the Macs did not identify our device correctly. We looked for a third-party alternative and stumbled over "USB Overdrive". This driver maps the mat-buttons to keys from the keyboard.

In class we got a short introduction to Max/MSP and all of us had programming experiences with Java. But we had to learn how to connect external java classes to Max/MSP. Our approach was to do the main logic part in Java and the MIDI signals in Max/MSP.

Our first protoypes arised from playing around with Java and Max/MSP. With the "trial&error" method we went along and implemented our first prototypes.

In the following usertests we became aware of our mistakes. We had to adjust the reaction time and delays according to human capabilities. The usability of a dance mat is completely different from typing on a keyboard. And this difference doesn't consist only of some ten miliseconds... So during our user-self-tests we did incremental changes to our prototype.

For further refinement, we put more thought into the actual mappings. On the one hand we had the very limited user input of the mat (8+2 digital buttons), and on the other hand the high variety of an instrument. The speed of the music should of course be proportional to the speed of pressing the buttons on the mat.
One of our biggest problems was the arrangement of the notes: what arrangement feels the most natural? What is the starting position of a user -- in the middle of the mat or right in front of the mat? We implemented two solutions and came to the conclusion that the tones should be arranged from left to right, buttom to top according to their pitch. The button in the upper-right corner of the mat has the highest pitch. 
To make our instrument more versatile, yet still simple to use, we added chords as well. Since it's very impractical to the hands and  humans have no third leg (except for a lucky few), we decided to add chords of the major scale in the way, that if two notes are pressed, the third chord tone is added by the computer.

In the upcoming usertest, when we were actuallytrying to play a song on our instrument, we had to adjust the tolerance of our device to detect a chord according to human reaction times

Our latest prototype now maps to each button a full tone from C to C and gives the user the possibility to play some chords of the major scale if two buttons are pressed together.

Future work

To improve and upgrade Hop-A-Tone we thought about a group experience. Creating music is much more fun in a group, therefore connecting several dance mats to a computer to get a band of hopping artists sounds like a blast! Each musician could play his favourite instrument and all together compose a common score.
Another approach could be to add another class of sensors. For example, an observing camera could produce some rhythms or tones if the actor does a special movement with his arms or if he loses ground contact (= jumps).
A better way to integrate the background rhythm would certainly go a long way to improve the experience.
Some kind of visualization could be implemented as well. The produced sound could be displayed in curves like an oscillograph does.
One could also think about a dance mat with more buttons to address a bigger scale of tones and/or to mix several instruments.

References

Perhaps you wonder why anybody would have such a mat at home??! Here is a possible answer...
http://en.wikipedia.org/wiki/Dance_Dance_Revolution

First of all, we needed a driver for the dance mat to map the buttons to keys from the keyboard
http://www.usboverdrive.com/

Since we are all unexperienced in doing music, we needed a short introduction of scales and chords
http://www.looknohands.com/chordhouse/piano/

For our second prototype we chose a different instrument: the guitar. But we recognized piano chords sound terrible on a guitar and therefore we looked for guitarchords
http://www.guitaretab.com/

To implement the Max/MSP Patch, we needed the corresponding MIDI settings to the pitch of the used notes.
http://www.dancetech.com/aa_dt_new/tables/pitch-freq-note.cfm

Downloads

Here you can test our prototype on your own.

Max/MSP Patch

Java Stuff

Installation & Running Instruction

You have to copy the Java classes into your MAX/MSP classfile-directory (on Windows usually c:\Program Files\Common Files\Cycling '74\java\classes). You can install the patch itself in any directory.
When running, remember to enable the toggle button setting the groove-object to loop. An empty Java window will popup, which you have to give focus to process the input. Please note that while keyboard input is possible using the keypad and the /-button for SELECT, there are problems with press/release events due to the driver hacking with USB Overdrive.
Should you want to use the Event Heap or Debug Version, edit the file 'dance_main' to use the output you want and start it as a standalone Java application. Please note that the Event Heap version is not well tested.

If you have some improvements, please let us know.
( Remove the NOSPAM in the adresses )

Mail to Mahsa, Christian and David

Implementation Overview

Java part:

Combo.java Implements object which contains state for one combo
ComboManager.java Handles Combo-recognition
ComboStorage.java Static class, combo data is loaded from here
dance_main.java Main class for Debug and Event Heap Output
Dance.java Handles the actual event input and processing. Generates the output data.
DanceEHMax.java Use this class as object in Max/MSP if you receive events by Event Heap
DanceEHUtils.java Constants used in Event Heap Output
DanceOutput.java Interface which specifies device-output
DanceOutputDebug.java Implements DanceOutput; Only console information
DanceOutputEH.java Implements DanceOutput; Sends events through Event Heap
DanceOutputMax.java Implements DanceOuput; Use this class as object in Max/MSP to receive the device data