Latency in Interactive Computing Systems

When we interact with the physical world around us, whether tapping our fingers on a desk or drawing with a pencil on paper, the effect is instant, thanks to the laws of physics. However, when we use a computer or other digital device, there is always a delay between our input actions (like tapping a key) and when the system reacts (e.g., by displaying a character).

This "end-to-end latency" is unavoidable because of digital processing: the computer and its input and output devices need time to receive and process what we did and to generate the corresponding output. As a result, our overall reaction time will always be a little longer when using a digital system than when interacting with the real world. However, as long as this delay stays below a millisecond, it is negligible because our biological reaction times are on the order of hundreds of milliseconds. When the latency between a discrete input action and its response approaches 50 to 100 ms, we start noticing it visually, and the system feels laggy. For continuous, spatial input like sketching on a touchscreen, much lower latency is noticeable. And even before we notice it consciously, our performance degrades because the system makes it technically impossible to react quite as fast anymore.

Surprisingly, this is an area in which computing technology has gotten worse, not better. For example, when I measured this end-to-end latency on my ZX Spectrum home computer system from 1982, it was between 0 and 20 ms, as you would expect from a 50 Hz CRT TV. Although from the 80s, the computer itself was already fast enough to react in code within a few microseconds, and neither the input device (an original Competition Pro joystick) nor the analog TV did any digital processing that would add latency to that baseline.

A modern system, on the other hand, exhibited delays of up to 200 ms. Even worse, the latency varied randomly between about 100 and 200 ms. This is not only noticeable but also makes it impossible to perform time-critical tasks, whether in a reaction-time experiment or a game. Modern digital displays are part of the problem, introducing 40 to 80 ms in my tests, depending on the model. USB was the bigger culprit in this case since the particular modern-day Speedlink Competition Pro USB joystick model I had purchased online added a variable delay of up to about 100 ms due to a poorly designed USB controller chip.

In the paper (in its first section) and in the talks below, I describe the problem in more detail and provide some suggestions for solutions, from using microcontrollers instead of computers, to increasing USB polling rates, to using FPGA-based "retro" computers like the MiSTer or Spectrum Next that support near-zero-latency peripherals. For example, my Spectrum Next delivered the same best-in-class latency of 0 to 20 ms when used with an original joystick and CRT.

Publications

2023

• Jan Borchers, Anke Brocker, Sebastian Hueber, Oliver Nowak, René Schäfer, Adrian Wagner, Paul Miles Preuschoff and Lea Emilia Schirp. The Aachen Lab Demo: From Fundamental Perception to Design Tools.  In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, CHI EA '23, pages 4, Association for Computing Machinery, New York, NY, USA, April 2023.
  

Talks

USB & HDMI Considered Harmful: Input Latency in Legacy vs. Modern Computing Architectures. Jan Borchers, Sketching In Hardware '22, Dublin, Oct 1, 2022.

Latency Lecture, Designing Interactive Systems I, Winter Semester 2021/22, Jan Borchers, Jan 19, 2022

USB & HDMI Considered Harmful (in German). Jan Borchers, Aachen Maker Meetup, Nov 17, 2021