Phonecam-based Interactions for Large Public Displays


Rafael "Tico" Ballagas
RWTH Aachen Univeristy

Michael Rohs
ETH Zurich

Jennifer G. Sheridan
Lancaster University


We are interested in finding mechanisms for users to use their mobile phones as ubiquitous pointing devices for public and situated displays.

Point & Shoot:

The point & shoot technique allows users to aim the mobile phone camera to select objects on a situated display using a cursor on the live camera image of the mobile phone. This technique uses Visual Codes to establish a coordinate system on the situated display independent of the viewing perspective.


The sweep technique allows the phonecam to be used like an optical mouse. Using optical flow image processing, the phonecam samples successive images and then sequentially compares them to determine relative motion in the (x, y, theta) dimensions thus allowing the camera to be used as a three degrees of freedom input device.

Designing for Serendipity:

To allow serendipitous interactions with displays in public, people must have a way to fluidly establish the communications relationships. In our implementation, we are using Visual Codes to encode the bluetooth address of the public display. Users merely need to take a picture of the display (along with its visual code) to begin interacting with it.

Related Publications:


Rafael Ballagas, Michael Rohs, Jennifer Sheridan, and Jan Borchers. The Smart Phone: A Ubiquitous Input Device. IEEE Pervasive Computing, 5(1):70–77, Jan-Mar 2006. PDF Document BibTeX Entry.


Rafael Ballagas, Michael Rohs, and Jennifer Sheridan. Mobile Phones as Pointing Devices. In Pervasive 2005 Workshop on Pervasive Mobile Interaction Devices (PERMID), Munich, Germany, May 2005. PDF Document BibTeX Entry.
Rafael Ballagas, Michael Rohs, and Jennifer Sheridan. Sweep and Point & Shoot: Phonecam-Based Interactions for Large Public Displays. In CHI '05: CHI '05 extended abstracts on Human factors in computing systems, pages 1200–1203, New York, NY, USA, April 2005. ACM Press. PDF Document Movie BibTeX Entry.

Created by ballagas. Last Modification: Monday 21 of September, 2009 12:36:14 by dennis.

Media Computing Group at RWTH Aachen