Personal Orchestra

 

(Also known as "The Virtual Conductor", or "Der Virtuelle Dirigent")

Experience Personal Orchestra 14 at the House of Music in Vienna. Logo Haus der Musik Wien

The goal of the Personal Orchestra series of projects is to adapt this centuries-old interaction metaphor to digital multimedia so that everyone may enjoy the wonderful experience of conducting world-famous orchestras.

Our primary research goals for this line of research are:

  • Innovative interfaces for musical expression
  • Gesture recognition (capturing the user's physical movements)
  • Gesture modeling and interpretation
  • Real-time interactive audio and video time-stretching without pitch-shifting artifacts
  • Time design and frameworks for time design of multimedia

Our conducting simulation software builds on this research. Its newest iteration, version 14, added dynamic movement of audio focus, support for 4K video, and is the first version that runs natively on macOS.

Current Team

Prof. Dr. Jan
Borchers

Sebastian
Hueber

Adrian
Wagner

News

  • In celebration of the HOUSE OF MUSIC VIENNA's 20th anniversay, the Personal Orchestra exhibit will be updated with completely new recordings in 2021 (Sep 30, 2020).
  • A permanent demo installation of the Personal Orchestra exhibit is now set up in our lab (operating instructions) (Aug 14, 2014)
  • Today, the HOUSE OF MUSIC VIENNA launched our redesigned Personal Orchestra exhibit. Using an infrared baton, visitors can conduct the Vienna Philharmonic by controlling speed, volume, and the emphasis of different instrument sections. The new system now features our PhaVoRIT continuous real-time high-quality audio stretching, six new recordings of the orchestra in full HD video, an electronic music stand displaying the score, and helpful hints by maestro Zubin Mehta. Our original exhibit was installed in 2000, and has been used by 1.5 million visitors since, making it the most successful station in the house, and a top tourist attraction - see the TV news report (Nov 27, 2009 --video cached on our server, 11.6 MB).
  • Documentary about "Haus der Klänge" (House of sound) and a interview on personal orchestra (ZDFinfokanal, July 2007, video, 28 MB)
  • Coverage of Personal Orchestra 1 in Vienna City Guides (English+German, 2006)
  • Documentary about Personal Orchestra 1 from "Abenteuer Erde" on 19 April 2006 (high (41 MB) /low (18 MB) resolution)
Click here to show previous versions of Personal Orchestra.

Prior Versions of the System

Personal Orchestra 13: The Virtual Conductor 2015 (2014–2015, currently on exhibition)

Casa de la Musica de Vienna en Puebla, Mexico

The 2015 version of PO is almost identical to the 2014 version, thus sharing its version number of PO13. The main difference is the change of the VMWare version, which required some internal code changes.

Team:


Personal Orchestra 13: The Virtual Conductor 2014 (2013–2014)

HOUSE OF MUSIC VIENNA, Austria

The 2014 version of this system was developed primarily with the following goals in mind:

  • Exchanging the active infrared baton for a passive baton, which will be tracked using standard motion caption hardware, to reduce maintenance load on the active batons
  • Developing reliable tracking and conducting motion classification algorithms
  • Updating the software to be compatible with newer versions of the underlying operating system
  • Updating video and audio rendering to use modern APIs and provide smoother playback
  • Tightening the closed loop for the conducting speed tracking to allow better and more fine grained tempo control
  • Allowing a wider spatial range of conducting gestures without losing dynamic range

These changes made the most successful exhibit in the Vienna House of Music even more enjoyable for visitors and significantly less maintenance-prone for the museum staff.

Team:


Personal Orchestra 6: The Virtual Conductor Reloaded (2008–2009)

HOUSE OF MUSIC VIENNA, Austria

This significantly upgraded version was again designed for the HOUSE OF MUSIC VIENNA. It opened to the public on Nov 27, 2009. It replaced our original PO1 exhibit installed there in 2000, which had been used by 1.5 million visitors since, making it the most successful exhibit in the museum, and a top Vienna tourist attraction.

Using an infrared baton, visitors can conduct the Vienna Philharmonic by controlling speed, volume, and the emphasis of different instrument sections. The new system features

  • the MICON electronic music stand that shows the score, highlighting the current position and emphasized instrument section, and turning pages in a 3D animation,
  • six new recordings of the orchestra in full HD video,
  • greatly improved conducting fidelity, giving visitors the tightest control over the orchestra of all our systems, including the ability to stop conducting for a fermata,
  • our PhaVoRIT continuous real-time high-quality audio stretching,
  • up to six different instrument sections that can be emphasized by conducting towards them,
  • helpful hint video scenes by maestro Zubin Mehta explaining how best to conduct each piece.

One of the major new research and development challenges in this version was to create a video pipeline architecture that can display full HD video at rapidly varying speeds from 0 to 200% without any stuttering or other artifacts. The system was based on the advanced video rendering subsystem inside macOS.

 (Foto: (c) Anna Rauchenberger / fotodienst.at)

More images from the launch are available at fotodienst.at.

Team:

And from TU Darmstadt:


Personal Orchestra 3: Maestro! (2003–2006 )

The Betty Brinn Children's Museum in Milwaukee, USA

This system was developed for a children's museum. The main new feature developed for this project was an adaptive gesture recognition to accommodate both regular conducting gestures by adults and irregular conducting movements by young children.

'Maestro!'' at the Betty Brinn Children's Museum in Milwaukee, USA
More information is available at the PO3 home page.
Team:

Personal Orchestra 2: You're the Conductor (2002–2003)

The Children's Museum in Boston, USA

You're the Conductor was our second generation conducting system designed specifically for children (although it worked great for adults too). It featured "Stars and Stripes Forever" performed by the Boston Pops/Boston Symphony Orchestra.

You're the Conductor offered an improved technical framework, especially in the area of time-stretching audio. Audio was processed in real-time using an improved phase vocoder algorithm, resulting in increased responsiveness, audio quality and tempo range over the Virtual Conductor. Movies were stored as standard QuickTime movies with uncompressed PCM audio.

The key research contributions were:

  • First conducting system specifically designed for children, featuring an audio and video recording for an orchestra.
  • Design and implementation of a technical framework for real-time interactive audio time stretching without pitch shifting artifacts.
More information is available at http://www.actibits.com.

People:

Personal Orchestra 1: The Virtual Conductor (1998–2000)

The House of Music in Vienna, Austria

The Virtual Conductor was the world's first system that allowed users to conduct an actual audio and video recording of an orchestra. It has become a highlight of the HOUSE OF MUSIC VIENNA where it has been a permanent exhibit and museum highlight ever since the museum's opening in June 2000. It features exclusive performances from the world-famous Vienna Philharmonic Orchestra.

Users conduct the Vienna Philharmonic using a Buchla Lightning II infrared baton that sends MIDI data to a host computer performing the gesture recognition. Gestures are interpreted as simple up-down motions and translated to speed, volume and instrument emphasis which are then sent to a second computer responsible for processing the audio and video for display. Movies are stored as a multi-track QuickTime movie including a series of pre-processed, pitch-shifted tracks. By switching between these different audio tracks and varying the playback speed, the audio is time-stretched without any pitch-shifting artifacts.

The key research contribution was:

  • First system to let a human conduct an actual audio and video recording.
More information is available at http://www.actibits.com.

Team:


Related Publications:

    2022

  • Janosch Hübner. Improving Indoor Location Awareness using Bluetooth Low Energy Beacons. Bachelor's Thesis, RWTH Aachen University, Aachen, June 2022.
    PDF DocumentBibTeX Entry
  • 2015

  • Florian Heller, Simon Voelker, Chat Wacharamanotham and Jan Borchers. Transporters: Vision & Touch Transitive Widgets for Capacitive Screens.  In CHI '15: Extended Abstracts of the SIGCHI Conference on Human Factors in Computing Systems, pages 1603–1608,April 2015.
    HomepagePDF DocumentBibTeX Entry
  • Tatiana Smirnova. Grippo: Using Grip Gestures to Repurpose Everyday Objects as Controllers. Master's Thesis, RWTH Aachen University, Aachen, February 2015.
    PDF DocumentBibTeX Entry
  • 2014

  • Hesham Omran. Opposable Thumbs: A Bare-hands Text Input Technique. Master's Thesis, RWTH Aachen University, Aachen, August 2014.
    PDF DocumentBibTeX Entry
  • 2013

  • Pierre Schoonbrood. PowerSocket: Implementation of a Small-Scale Power Consumption Measurement Platform. Bachelor's Thesis, RWTH Aachen University, Aachen, December 2013.
    PDF DocumentBibTeX Entry
  • Kashyap Todi. Sniper Pointing: Above the Surface Pointing with Multiple Resolutions. Master's Thesis, RWTH Aachen University, Aachen, February 2013.
    PDF DocumentBibTeX Entry
  • 2012

  • Florian Heller and Jan Borchers. Physical prototyping of an on-outlet power-consumption display. In interactions, pages 14–17, ACM, New York, NY, USA, Jan 2012.
    HomepagePDF DocumentBibTeX Entry
  • 2011

  • Wolfgang Kluth. PowerSocket - Implementation and Evaluation of an On-Socket Power Consumption Display. Bachelor's Thesis, RWTH Aachen University,April 2011.
    PDF DocumentBibTeX Entry
  • Simon Voelker, Malte Weiss, Chat Wacharamanotham and Jan Borchers. Dynamic Portals: A Lightweight Metaphor for Fast Object Transfer on Interactive Surfaces.  In ITS '11: Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces, pages 158–161, ACM Press, New York, NY, USA, 2011.
    MoviePDF DocumentBibTeX Entry
  • Florian Heller and Jan Borchers. PowerSocket: Towards On-Outlet Power Consumption Visualization.  In CHI '11: Extended Abstracts of the CHI 2011 Conference on Human Factors in Computing Systems, pages 1981–1986,2011.
    PDF DocumentBibTeX Entry
  • 2008

  • Anna Koster. WiiCon:Acceleration Based Real-Time Conducting Gesture Recognition for Personal Orchestra. Diploma Thesis, RWTH Aachen University, Aachen, December 2008.
    PDF DocumentBibTeX Entry
  • 2007

  • Eric Lee, Thorsten Karrer and Jan Borchers. An Analysis of Startup and Dynamic Latency in Phase Vocoder-Based Time-Stretching Algorithms.  In ICMC '07: Proceedings of the International Computer Music Conference, pages 73–80,Copenhagen, Denmark, August 2007.
    HomepagePDF DocumentBibTeX Entry
  • 2006

  • Thorsten Karrer, Eric Lee and Jan Borchers. PhaVoRIT: A Phase Vocoder for Real-Time Interactive Time-Stretching.  In ICMC '06: Proceedings of the International Computer Music Conference, pages 708–715,New Orleans, USA, November 2006.
    HomepagePDF DocumentBibTeX Entry
  • Eric Lee, Ingo Grüll, Henning Kiel and Jan Borchers. conga: A Framework for Adaptive Conducting Gesture Analysis.  In NIME 2006 International Conference on New Interfaces for Musical Expression, pages 260–265,Paris, France, June 2006.
    HomepagePDF DocumentBibTeX Entry
  • Jan Borchers, Aristotelis Hadjakos and Max Mühlhäuser. MICON: A Music Stand for Interactive Conducting.  In NIME 2006 International Conference on New Interfaces for Musical Expression, pages 254–259,Paris, France, June 2006.
    HomepagePDF DocumentBibTeX Entry
  • Eric Lee, Henning Kiel, Saskia Dedenbach, Ingo Gruell, Thorsten Karrer, Marius Wolf and Jan Borchers. iSymphony: An Adaptive Interactive Orchestral Conducting System for Conducting Digital Audio and Video Streams.  In CHI '06: Extended Abstracts on Human Factors in Computing Systems, pages 259–262, ACM Press, Montréal, Canada, April 2006.
    HomepagePDF DocumentBibTeX Entry
  • Eric Lee, Thorsten Karrer and Jan Borchers. Toward a Framework for Interactive Systems to Conduct Digital Audio and Video Streams. In Computer Music Journal, pages 21–36, number 1, March 2006.
    HomepagePDF DocumentBibTeX Entry
  • 2005

  • Thorsten Karrer. PhaVoRIT: A Phase Vocoder for Real-Time Interactive Time-Stretching. Diploma Thesis, RWTH Aachen University, Aachen, November 2005.
    BibTeX Entry
  • Eric Lee and Jan Borchers. The Role of Time in Engineering Computer Music Systems.  In NIME 2005 International Conference on New Interfaces for Musical Expression, pages 204–207,Vancouver, Canada, May 2005.
    HomepagePDF DocumentBibTeX Entry
  • Eric Lee, Marius Wolf and Jan Borchers. Improving orchestral conducting systems in public spaces: examining the temporal characteristics and conceptual models of conducting gestures.  In Proceedings of the CHI 2005 conference on Human factors in computing systems, pages 731–740, ACM Press, Portland, Oregon, April 2005.
    HomepageMoviePDF DocumentBibTeX Entry
  • 2004

  • Eric Lee, Teresa Marrin Nakra and Jan Borchers. You're the Conductor: A Realistic Interactive Conducting System for Children.  In NIME 2004 International Conference on New Interfaces for Musical Expression, pages 68–73,Hamamatsu, Japan, June 2004.
    HomepagePDF DocumentBibTeX Entry
  • Jan Borchers, Eric Lee, Wolfgang Samminger and Max Mühlhäuser. Personal Orchestra: A real-time audio/video system for interactive conducting. In ACM Multimedia Systems Journal Special Issue on Multimedia Software Engineering, pages 458–465, ACM Press,  number 5, March 2004.
    HomepageBibTeX Entry
  • 2002

  • Jan Borchers, Wolfgang Samminger and Max Mühlhäuser. Personal Orchestra: Conducting Audio/Video Music Recordings.  In WEDELMUSIC 2002 International Conference On Web Delivering of Music, Conference, December 2002.
    HomepageMoviePDF DocumentBibTeX Entry
  • Jan Borchers, Wolfgang Samminger and Max Mühlhäuser. Engineering a realistic real-time conducting system for the audio/video rendering of a real orchestra.  In IEEE MSE 2002 Fourth International Symposium on Multimedia Software Engineering, Conference, December 2002.
    HomepagePDF DocumentBibTeX Entry
  • 2001

  • Jan Borchers, Wolfgang Samminger and Max Mühlhäuser. Conducting A Realistic Electronic Orchestra.  In 14th Annual Symposium on User Interface Software and Technology, pages 161–162, Conference, Orlando, Florida, November 2001.
    HomepagePDF DocumentBibTeX Entry


Hardware checklist for PO demonstrations (internal access)

Personal Orchestra Documentation

)

Attachments:
File Description File size Downloads Last modified
PO_low.mov Report on PO1 in Abenteuer Erde April 19 2006 19118 kB 7906 2006-08-04 23:52

We use cookies on our website. Some of them are essential for the operation of the site, while others help us to improve this site and the user experience (tracking cookies). You can decide for yourself whether you want to allow cookies or not. Please note that if you reject them, you may not be able to use all the functionalities of the site.