Latency in Interactive Computing Systems

When we interact with the physical world around us, whether tapping our fingers on a desk or drawing with a pencil on paper, the resulting effect follows instantly thanks to the laws of physics. However, when we use a computer or other interactive device, there is a delay between our input actions (like tapping a key) and when the system reacts (e.g., by displaying a character).

This "end-to-end latency" is unavoidable because the computer needs time to receive and process what we did and to generate the corresponding output. As a result, our overall reaction time will always be a little longer when using a computer than when interacting with the real world. As long as this delay stays below a millisecond, however, the effect is negligible because our own biological reaction times are on the order of hundreds of milliseconds. When the latency between a discrete input action and its response approaches 50 to 100 ms, we start noticing it visually, and the system feels laggy. For continuous, spatial input like sketching on a touchscreen, much lower latency is noticeable.

Surprisingly, this is an area in which computing technology has not gotten better but worse. For example, when I measured this latency on my ZX Spectrum home computer system from 1982, it was between 0 and 20 ms, as one would expect from a 50 Hz CRT TV. The computer itself was already fast enough to react in code within a few <b>micro</b>seconds, and neither the input device (a Competition Pro joystick) nor the analog TV did any digital processing that would add latency to that baseline.

A modern system, on the other hand, exhibited delays of up to 200 ms. Even worse, the latency varied randomly between about 100 and 200 ms. This is not only noticeable, but it also makes it impossible to perform time-critical tasks, whether in a reaction-time experiment or in a game. Modern digital displays are part of the problem, introducing 40 to 80 ms in my tests depending on the model. USB was the bigger culprit in this case since the particular Speedlink Competition Pro USB joystick model used added a variable delay of up to about 100 ms. 

In the paper (in its first section) and in the talks below, I describe the problem in more detail, and provide some suggestions for solutions, from using microcontrollers instead of computers, to increasing USB polling rates, to using FPGA-based "retro" computers like the MiSTer or Spectrum Next that support near-zero-latency peripherals. For example, the Spectrum Next delivered the same best-in-class latency of 0 to 20 ms when used with an original joystick and CRT.

Publications

2023

• Jan Borchers, Anke Brocker, Sebastian Hueber, Oliver Nowak, René Schäfer, Adrian Wagner, Paul Miles Preuschoff and Lea Emilia Schirp. The Aachen Lab Demo: From Fundamental Perception to Design Tools.  In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, CHI EA '23, pages 4, Association for Computing Machinery, New York, NY, USA, April 2023.
  

Talks

USB & HDMI Considered Harmful: Input Latency in Legacy vs. Modern Computing Architectures. Jan Borchers, Sketching In Hardware '22, Dublin, Oct 1, 2022.

Latency Lecture, Designing Interactive Systems I, Winter Semester 2021/22, Jan Borchers, Jan 19, 2022

USB & HDMI Considered Harmful (in German). Jan Borchers, Aachen Maker Meetup, Nov 17, 2021