Coordination between two people had no off-the-shelf measure. Building one meant designing the task, getting four recording devices onto a shared clock, training junior researchers to run the sessions without me in the room, and then aligning everything those devices captured onto a single timeline.
Collaborative products - shared documents, co-browsing, multiplayer, XR - assume that showing each person what the other is doing is enough for them to coordinate. That assumption had not been tested directly, because there was no way to measure coordination between two people as it happened.
Measuring one user is tractable. Measuring the system two users form is a different problem. Coordination is not a state you can sample at a point in time; it is a relationship between behaviours unfolding across modalities, in both directions. No off-the-shelf instrument existed, so we built one.
We seated pairs face-to-face across a table with six impressionist paintings mounted on the partition between them, arranged so each partner could see both the paintings and the other person's face. The task was to agree on a ranking within four minutes. The task is deliberately mundane: it produces natural conversation, disagreement, and frequent pointing, without requiring any expertise from participants.
We instrumented all of it simultaneously. Both participants wore binocular mobile eye-trackers. Fiducial markers between the paintings mapped each person's gaze onto the shared surface rather than only into their own headset video. A microphone captured dialogue and two participant-facing cameras captured pointing gestures. Every deictic gesture and verbal reference to a painting was defined in advance so it could be coded consistently.
Four devices recording the same four minutes do not agree on when anything happened, and that had to be solved before we collected data rather than after. One design decision carried it: an audiovisual cue played on the monitors in front of both participants, positioned so all four feeds captured it. That single shared event is the reference every recording was later synchronized against.
The rest was piloting. We ran the full setup repeatedly to confirm the cue actually landed in every feed, that the marker-based surface mapping held while people moved and talked rather than only while they sat still, and that a session produced recordings we could line up afterwards. Sessions with participants only started once the rig cleared that bar, since a session that fails is a pair you cannot simply run again.
A session meant two computers, two eye-trackers, two cameras, a calibration procedure that could fail silently, and a moderator staying present with two participants throughout. I could not be in the room for all of them, so researchers had to be able to run sessions independently.
I wrote a standardized testing protocol, then rebuilt it around explicit prioritization rather than strict adherence: which steps must be identical every time because the data depends on them, and where a researcher should adapt the procedure to their own workflow. Some wanted printed instruction scripts, some tagged equipment with handling prompts, some reordered steps that had no dependency between them. For researchers new to moderating, I paired that with observation of my sessions first, sitting in on their early ones with an immediate debrief afterwards, and tapering support as their data-quality checks came back clean.
Junior researchers reached independent session-running within one to two weeks, with fewer failed sessions and no loss of rigour on the steps that carried the findings. One went on to supervise their own research assistants using a version of the same protocol.
The other bottleneck was analysis. Everyone on the team wrote their own R code for jobs that were largely the same - cleaning, modelling, plotting. People shared scripts on request, but there was no system, so someone could lose hours tracking down a colleague's code, or wait days for a collaborator to hand back a script before they could continue.
I learned a collaborative Git workflow and taught it to the team: a local repository, pushed to a personal remote, merged into a shared one that everyone syncs from. Presentations first, then hands-on workshops, then co-working on live projects until it was routine.
People on the same project could then write against the same scripts concurrently instead of passing files back and forth, which cut analysis time directly. The less expected effect was social: once the work lived in a shared repository, asking for help became reading a commit and contributing became a merge, so the team collaborated more because it cost less to do so.
Four minutes of conversation per pair across 26 pairs, with every painting reference needing a timestamp accurate to a fraction of a second. Hand-transcribing that is weeks of work and the bottleneck for the whole analysis.
We processed the audio with an AI speech model to produce timestamped, word-aligned transcripts, then imported those into the annotation software alongside the video. That removed the transcription bottleneck.
The model did not decide anything. Judging whether "the realistic one" or "your lower left one" refers to a specific painting is interpretation, so three coders blind to the hypotheses did that by hand, after training on a shared set and cross-checking one pair in common to establish interrater reliability before any of it counted. The model handled the mechanical layer and researchers handled the judgement layer, with the boundary defined before we started rather than justified afterwards.
Getting the hardware onto a shared clock solved when each device thought something happened. It did not solve the second alignment problem, which only appeared once there were recordings to work with: the four streams do not tick at the same speed. The gaze cameras sampled at 200 Hz, scene video at 60 Hz, participant-facing cameras at 25 fps, and speech arrived as timestamped text with no fixed rate. Mobile eye-tracking of a moving, talking pair also loses samples to blinks and head movement; ours averaged 18%, which is normal for the method and still has to be handled rather than ignored.
This pipeline came down to one decision: what temporal resolution can this dataset honestly support? The slowest stream set the answer. Everything was aligned on the shared audiovisual cue, then resampled to a common 5 Hz categorical time series, with mode-based smoothing applied to the noisiest channel. From there, cross-recurrence quantification analysis could test which behaviour leads which in time, both within a person and between the two of them.
Gaze, pointing, and speech were all coupled above chance, within each person and between the pair - one system rather than three separate signals.
Collaborative interfaces routinely reduce coordination to one signal, usually cursor position. That is a lossy compression of what people actually use. Deictic gesture - the "look at this" move - is the strongest and most reliable pointer to a partner's attention, and it leads their gaze by roughly a second, which is a concrete latency budget for presence features. Speech carries attention too, and pairing an explicit pointer with a verbal reference holds joint attention longer than either alone.
For shared screens, co-browsing, multiplayer, and XR, a coordination affordance is not decoration on top of presence. It does the work that gesture does in person.
Haraped, L., Gerlofs, D. J., Huang, O. C. H., Hickling, C., Bischof, W. F., Sachse, P., & Kingstone, A. (2025). Coordinating attention in face-to-face collaboration: The dynamics of gaze, pointing, and verbal reference. Cognitive Science, 49(10), Article e70123. https://doi.org/10.1111/cogs.70123