Measuring attention coordination between two users, and building the team that ran it.

Measuring coordination between two users

Collaborative products - shared documents, co-browsing, multiplayer, XR - assume that showing each person what the other is doing is enough for them to coordinate. That assumption had not been tested directly, because there was no way to measure coordination between two people as it happened.

Measuring one user is tractable. Measuring the system two users form is a different problem. Coordination is not a state you can sample at a point in time; it is a relationship between behaviours unfolding across modalities, in both directions. No off-the-shelf instrument existed, so we built one.

Designing the task and the instrumentation

We seated pairs face-to-face across a table with six impressionist paintings mounted on the partition between them, arranged so each partner could see both the paintings and the other person's face. The task was to agree on a ranking within four minutes. The task is deliberately mundane: it produces natural conversation, disagreement, and frequent pointing, without requiring any expertise from participants.

We instrumented all of it simultaneously. Both participants wore binocular mobile eye-trackers. Fiducial markers between the paintings mapped each person's gaze onto the shared surface rather than only into their own headset video. A microphone captured dialogue and two participant-facing cameras captured pointing gestures. Every deictic gesture and verbal reference to a painting was defined in advance so it could be coded consistently.

One design decision made the rest possible: an audiovisual cue played on the monitors in front of both participants and captured by all four video feeds. That single shared event is what everything else was later synchronized against.

Adapting protocols and training researchers to run sessions

A session meant two computers, two eye-trackers, two cameras, a calibration procedure that could fail silently, and a moderator staying present with two participants throughout. I could not be in the room for all of them, so researchers had to be able to run sessions independently.

I wrote a standardized testing protocol, then rebuilt it around explicit prioritization rather than strict adherence: which steps must be identical every time because the data depends on them, and where a researcher should adapt the procedure to their own workflow. Some wanted printed instruction scripts, some tagged equipment with handling prompts, some reordered steps that had no dependency between them. For researchers new to moderating, I paired that with observation of my sessions first, sitting in on their early ones with an immediate debrief afterwards, and tapering support as their data-quality checks came back clean.

Researchers reached independent session-running within a few weeks, with fewer failed sessions and no loss of rigour on the steps that carried the findings. One went on to supervise their own research assistants using a version of the same protocol.

Deploying a collaborative version-control workflow

The other bottleneck was analysis. Everyone on the team wrote their own R code for jobs that were largely the same - cleaning, modelling, plotting. People shared scripts on request, but there was no system, so someone could lose hours tracking down a colleague's code, or wait days for a collaborator to hand back a script before they could continue.

I learned a collaborative Git workflow and taught it to the team: a local repository, pushed to a personal remote, merged into a shared one that everyone syncs from. Presentations first, then hands-on workshops, then co-working on live projects until it was routine.

People on the same project could then write against the same scripts concurrently instead of passing files back and forth, which cut analysis time directly. The less expected effect was social: once the work lived in a shared repository, asking for help became reading a commit and contributing became a merge, so the team collaborated more because it cost less to do so.

Aligning four data streams at different sampling rates

With sessions running, the remaining difficulty was in the data. The gaze cameras sampled at 200 Hz, scene video at 60 Hz, participant-facing cameras at 25 fps, and speech arrived as timestamped text with no fixed rate. Mobile eye-tracking of a moving, talking pair also loses samples to blinks and head movement; ours averaged 18%, which is normal for the method and still has to be handled rather than ignored.

The pipeline came down to one decision: what temporal resolution can this dataset honestly support? The slowest stream set the answer. Everything was aligned on the shared audiovisual cue, then resampled to a common 5 Hz categorical time series, with mode-based smoothing applied to the noisiest channel. From there, cross-recurrence quantification analysis could test which behaviour leads which in time, both within a person and between the two of them.

AI transcription with human coding

Four minutes of conversation per pair across 26 pairs, with every painting reference needing a timestamp accurate to a fraction of a second. Hand-transcribing that is weeks of work and the bottleneck for the whole analysis.

We processed the audio with an AI speech model to produce timestamped, word-aligned transcripts, then imported those into the annotation software alongside the video. That removed the transcription bottleneck.

The model did not decide anything. Judging whether "the realistic one" or "your lower left one" refers to a specific painting is interpretation, so three coders blind to the hypotheses did that by hand, after training on a shared set and cross-checking one pair in common to establish interrater reliability before any of it counted. The model handled the mechanical layer and researchers handled the judgement layer, with the boundary defined before we started rather than justified afterwards.

Results

Gaze, pointing, and speech were all coupled above chance, within each person and between the pair - one system rather than three separate signals.

• Pointing leads the partner's gaze. A partner is most likely to look at a painting about a second after the other person points at it. Verbal references lead gaze as well, but less tightly and with more variability between people.
• Gaze-to-gaze coupling is symmetric, with a lag. Pairs looked at the same things at similar times, most often about a second apart rather than simultaneously, with neither person consistently leading.
• Conversational gaze followed the canonical pattern. Participants spent about 28% of listening time looking at their partner's face, against about 12% while speaking.
• Pointing and speaking together sustained coupling longer than pointing alone, so the channels compound rather than substitute.

Implications for collaborative interfaces

Collaborative interfaces routinely reduce coordination to one signal, usually cursor position. That is a lossy compression of what people actually use. Deictic gesture - the "look at this" move - is the strongest and most reliable pointer to a partner's attention, and it leads their gaze by roughly a second, which is a concrete latency budget for presence features. Speech carries attention too, and pairing an explicit pointer with a verbal reference holds joint attention longer than either alone.

For shared screens, co-browsing, multiplayer, and XR, a coordination affordance is not decoration on top of presence. It does the work that gesture does in person.