An in-person eye-tracking study of 26 pairs found that multimodal cues — gaze, pointing, speech — operate as a system, with design implications for real-time collaborative interfaces.
We tracked two people at one screen — their eyes, their pointing gestures, and every word they said, simultaneously. People don’t coordinate through a single channel: they use gaze, gesture, and speech as a system, and when one channel is limited, the others pick up the slack.
Multi-user and collaborative interfaces are increasingly central to product strategy — from shared editing in Figma and Google Docs to co-browsing in customer service and real-time interfaces in games. But how do two people actually coordinate their attention when sharing a screen? The design question is concrete: where should the interface support joint focus, and where should it support independent navigation?
I ran an in-person eye-tracking study of 26 dyads (N=52), capturing gaze, pointing gestures, and speech synchronously as pairs worked through shared tasks. I used AI-assisted transcription to process the speech data at scale, and cross-recurrence quantification analysis to model how the pairs’ attention aligned and diverged over time. I also trained junior researchers on the hardware setup and analysis protocols.
Haraped, L., Gerlofs, D. J., Huang, O. C. H., Hickling, C., Bischof, W. F., Sachse, P., & Kingstone, A. (2025). Coordinating attention in face-to-face collaboration: The dynamics of gaze, pointing, and verbal reference. Cognitive Science, 49(10), Article e70123. https://doi.org/10.1111/cogs.70123
Pairs used gaze, pointing, and speech as a single system — substituting one channel for another when the environment limited it. Whether partners oriented toward a shared screen or separate views meaningfully changed their coordination patterns. The practical upshot: how a collaborative interface handles presence, cursor visibility, and shared-focus affordances shapes how well people can coordinate.
Published in Cognitive Science. The findings inform real-time, multi-user, and multimodal interface design.
Collaborative products often assume that making each user's activity visible to the other is enough for coordination. The reality is richer: people use multiple channels in parallel, and reducing coordination to a single signal — just cursor position, say — misses the system they actually rely on. Interface design for real-time collaboration, especially on shared screens or in XR, benefits from understanding how people orchestrate these channels, and where flattening that richness creates friction.