Personal Orchestra 3 Documentation


Stable Versions



Software documentation

Overview
Personal Orchestra 3 exists in several revisions and configurations. It was originally developed for the BBCM under the name Maestro!. Since then, though, it has been enhanced and extended. It now works with many different input plugins, and can use the MICON notestand developed by Telis Hadjakos.

The BBCM currently uses the version found at SVNROOT/po/po3tags/v1.1-2006.04.19/


Overall architecture
Overview

The code for the version currently being developed is spread over 2 folders in SVN: SVNROOT/po/po3 and SVNROOT/po/po4. The po3 folder contains most of the code and all Xcode projects, while the po4-folder contains only the files needed for communication with external, alternative output devices like the MICON notestand. This separation stemmed from earlier intentions to separate code used for future exhibits from the code base used for PO3. This never happened, as most additions required changes to PO3-code as well, so the content of the po4 folder should probably be integrated into the po3 folder in the future.

po3 folder
(To see the source code documentation, use podoc/podoc as login)

CONGA: Contains the framework which acts as the basis for gesture recognition, developed by Ingo Grüll. CONGA contains many classes which can be put together as a graph, more or less like Max/MSP works. More info can be found in Ingo's diploma thesis.

CONGASupport: Contains classes which can act as input plugins for a CONGA graph. CONGAInputPlugin is the abstract superclass of which all input plugins must inherit. An input plugin expects a delegate conforming to CONGASupportBatonInputDataListener, which it will continuously update. This delegate then has to update the actual CONGA graph.
CONGASupport also contains classes to help visualize input and calculations of a CONGA graph. For an example on how to use these, see CONGAViz.

CONGApo: Contains classes which contain pre-configured CONGA detectors for different gesture types. Supported gestures are Wiggle (random movement), UpDown and 4 beat legato (see SVNROOT/po/po3/Documentation/4beat neutral legato gesture.pdf, this detector is still not in a usable state). Applications like Maestro!/PO4 or CONGAViz can then use CPOExtendableCongaGraph, which implements logic to use multiple detectors simultaneously and to switch between them according to their current detection quality.

libSTF: Semantic Time Framework developed by Eric Lee. Implements the actual time-stretching of the videos, and synchronization to the current beat, as calculated by CONGA/CONGApo
BatonCam: Contains classes to read frames from USB/Firewire cameras, and to track movement in these using different algorithms. CONGASupport contains input plugins which make use of these trackers.

PO3: Contains the project for the actual PO applications. At the moment there are two targets: Maestro!, which behaves more or less like the BBCM version (no notestand, no menu selection, conducting starts when baton is moved, no complain sequences), and PO4, in which all of the new stuff is activated.
PO3 depends on BatonCam, STF, CONGA, CONGASupport and CONGApo and acts as the central controller to all these. It uses CONGApo to set up gesture detectors based on CONGA objects. It uses all the available input plugins from CONGASupport to feed these detectors. The user can switch between input plugins at run time. It feeds the calculated current beat and tempo into STF, which then speeds up or slows down the speed of video and audio.

po4 folder
The content of this folder is basically everything (regarding code) an external developer needs to implement an alternative output device for PO.

pocp4: Contains the protocol used for communication with the Notestand MICON. The PO3 Xcode project references these, so this folder has to be checked out as well when building.

CocoaClient/CocoaServer: Contains a sample client and server implementing the pocp4 protocol. Can be used for test purposes.



Building
To build PO, first check out SVNROOT/po/po3 and SVNROOT/po/po4 into the same parent folder. You should have a folder hierarchy like
  • trunk
    • po3
    • po4
Then open trunk/po3/PO3/PO3.xcodeproj in Xcode. Most of the time you should select PO4 as target (differences between the two targets are explained above). Click build. All the projects involved are configured to place their build products into trunk/po3/build, which is where you will find the PO4 bundle, too. You can already start PO4 at this time, but to really use it, you'll need movies, which are documented below.


Data files
Personal Orchestra uses 3 types of resources:
  • PO3 Movies, which contain a recording of an orchestra plus metadata describing the time of the beats, location of instrument groups, etc.
  • Selection Screens, which contain graphics used for the selection of the piece to conduct or the requested notestand visualization type.
  • PO3 Movie Collections, which act as a kind of super bundle for both PO Movies and Selection Screens. This is the preferred method of using the above resources in PO. Usable examples can be found at oliver/Public/Research Projects/Personal Orchestra/PO3/Data/PO3 Movie Collections.


Demonstrations
Hardware See the Hardware checklist for PO demonstrations for stuff you have to carry with you.

Software Above Hardware checklist mentions the external HD used for presentations. This drive contains 3 partitions: 2 of them contain bootable OS X installations (one for Intel, one for PPC Macs), the 3rd contains the PO applications and movie files. The OS X installations already have drivers installed for the M-Audio 2x2 USB Midi Interface normally used for demonstrations.

If you don't have access to this drive, or need to rebuild it, you should build both Targets from the PO3 xcode project (Maestro! and PO4). Use a revision which is known to be stable and working (at the time of writing r2330), and do a Release build for both. The resulting Maestro.app/PO4.app contain all frameworks/libraries needed to run.

Then decide what movies/movie collections you want to use. Available movies are in oliver/Public/Research Projects/Personal Orchestra/PO3/Data. Copy the application and all movies/collections you want to use to the PO demo HD. To use a movie/collection, open Maestro! or PO4, and drag it onto the application's dock icon. By dragging instead of just double-clicking you make sure that the correct application opens the file, as you will have several possible candidates (Maestro! and PO4 are different applications, but work on the same file types).

Setup
  • See oliver/Public/Research Projects/Personal Orchestra/PO3/Documentation/Personal Orchestra Setup.graffle for a schematic view on how to connect everything.
  • If possible, place the Buchla tracker above the projection screen. This usually yields the best results.
  • Let the user stand roughly 3m away from the Buchla tracker/projection screen.
  • The Buchla batons work with infrared light, so avoid any direct sunlight.


Evaluation of the components
  • CONGA
    • It is quite complicated to build complex graphs out of CONGA nodes, as currently everything has to be initialized and connected in code. A graphical editor like in Max/MSP might be better to experiment with different graph layouts.
    • It hasn't been shown that this framework really helps building conducting gesture detectors.
  • CONGASupport
    • The 2 CONGAOscilloscopeViews might be helpful for debugging future gesture detectors. They are not dependent on any CONGA-specific features. See CONGAViz for how to use these.
    • The CONGAInputPlugin classes need some refactoring, but the basic techniques contained for polling Buchla batons/USB mice/Video sequence grabber can be easily extracted and used in other input classes.
  • CONGApo
    • As this is basically just a wrapper around CONGA, see above.
  • BatonCam
    • The BatonCam framework, containing various optical detectors, can probably be reused.
    • The BatonCam test app is a mess, and should probably be split in seperate test apps for each optical detector.
  • PO3
    • PO3.app depends heavily on CONGA, CONGApo and libSTF. As all of these frameworks will be replaced in future PO-versions, most of PO3 needs rewriting.
    • The format of the PO3 Movie Bundles is very extensible, and could probably be reused with few modifications.
    • The format of the Selection Screens still relies on "hard-coded" frames in a movie. This might be better done dynamically in code by just overlaying text over an image.
    • pocp4, the protocol used for communicating with the Notestand, works well and stable.




Related papers, theses, code, technical documentation


- PhaVoRIT: A Phase Vocoder for Real-Time Interactive Time-Stretching
- Semantic Time: Representing Time and Temporal Transformations for Digital Audio in Interactive Computer Music Systems
- conga: A Framework for Adaptive Conducting Gesture Analysis
- MICON: A Music Stand for Interactive Conducting
- iSymphony: An Adaptive Interactive Orchestral Conducting System for Conducting Digital Audio and Video Streams
- Toward a Framework for Interactive Systems to Conduct Digital Audio and Video Streams.
- PhaVoRIT: A Phase Vocoder for Real-Time Interactive Time-Stretching
- conga: A Conducting Gesture Analysis Framework
- Improving orchestral conducting systems in public spaces: examining the temporal characteristics and conceptual models of conducting gestures
Details