Anna Koster
Student Assistant
This email address is being protected from spambots. You need JavaScript enabled to view it.
Project Involvement
Diploma Thesis with advisor Thorsten Karrer: "WiiCon: Acceleration Based Real-Time Conducting Gesture Recognition for Personal Orchestra"
Personal Orchestra is a research project which allows laymen to conduct an audio
and video recording of an orchestra. The orchestra adapts to the tempo of the conductor
and plays in the given speed. Up to now, a robust gesture recognition of
authentic conducting gestures for professional conductors was missing.
In this thesis, a new real-time gesture recognition plug-in for Personal Orchestra
is developed, which is able to process three-dimensional acceleration data of the
NintendoWii remote.
Due to the time-based character of conducting gestures, the hidden Markov model
is chosen to solve the task of automatic learning and recognition of 1-beat to 4-beat
gestures.
To achieve good recognition results, a large amount of training data is needed. For
this reason, two user studies for data acquirement were organized. Over 16000
conducting bars were recorded from 70 differing participants of the studies.
The hidden Markov model was trained with these conducting gestures for each
type of bar separately with the help of dynamic programming. After some
iterations of time alignment, the attributes in the training phase were optimized
for the recognition phase.
The recognition phase of conducting gestures works in real-time with unknown
conductors. Some adjustments were made in the recognition algorithm, so that the
recognition works while the conducting is executed. It is possible to recognize at
which position the conductor is in the bar at each time-point.
The hidden Markov model was trained with half of the conductors, the recognition
phase was evaluates with the other half, respectively. The correct position of beats
in a given type of bar is recognized correctly up to a rate of 92%.
Additionally a second task was solved: Where to recognize the beats by the system?
We identified the beats at the maximum peaks of the acceleration data, especially
in the down and forward acceleration of theWii remote.
Publications
- Anna Koster. WiiCon:Acceleration Based Real-Time Conducting Gesture Recognition for Personal Orchestra. Diploma Thesis, RWTH Aachen University, Aachen, December 2008.
- Elaine M. Huang, Anna Koster and Jan Borchers. Overcoming Assumptions and Uncovering Practices: When Does the Public Really Look at Public Displays?. In Pervasive 2008, Lecture Notes in Computer Science, pages 228-243, Springer Link, Sydney, Australia, May 2008.
