Miles Apart, Brainwaves Aligned – Exploring Remote Hyperscanning in Telepresence Scenarios

If you had ever thought of doing remote hyperscanning in telepresence settings, take a look at the study that shows how this can look.

For this purpose, authors used the OVBOX device – what is this tool?

In 2020, due to the pandemic, we all remember concerts and live shows being cancelled for safety reasons. ORLANDOviols are one such group that couldn’t hold live concerts anymore, but also couldn’t rehearse together.

So, they created the ORLANDOviols Consort Box (OVBOX) that allows musicians to connect, no matter the distance! The system is optimized for low latency and high quality spatial audio. This means that they can stay in harmony while playing miles away.

The authors of the study wanted to expand the boundaries of this device to a different signal – EEG in a remote hyperscanning setup. Keep reading to see the results!

Network structure of the experiment

In the experiment, they monitored EEG in two labs that are 2km away. Smarting PRO amplifiers collected EEG from both participants, and also motion data. All this + audio streams from mics and e-piano are routed through a central session server located ~350km away from the labs.

In total, authors used 4x OVBOX devices. Two of them are used to transmit previously mentioned multimodal data. OVBOX 3 is in the control room, which is physically in the same space as Lab 2. OVBOX 4 was actually not necessary, since it was a way for the experimenter to give instructions to the participants. But, the authors wanted to show that it’s possible to include more people in this remote hyperscanning setup.

You can also take a look at the network inside both labs for better understanding:

LAB1

LAB2

Four conditions to test remote Hyperscanning setup

First, authors wanted to look at evoked potentials in EEG and their timing. For this condition, they had participant 1 play a note on the piano at irregular times, while participant 2 just listened.

In the second condition, one of the participants would take lead in joint-clapping task. The other one would need to listen and try to follow the rhythm. The leader is always instructed to clap at the pace around 60 claps/min.

The last two tasks had to do with automatic labelling of speech activities during either free conversation, or during a turn-taking game. The turn-taking task is that one of the participants names a city, and then the other needs to name a different city starting with the same letter.

Authors noted that the results only serve as a proof of concept, which is why there’s only one pair of participants.

Measuring end-to-end network delay

Many clocks are involved in the setup, so establishing time delays is crucial to determine if this protocol is suitable for remote hyperscanning.

To measure the delay, scientists deployed 2 methods. First one looks at the round-trip time (often called ping time). This means that we take half of the round-trip time as the delay. Downside of this method is that it considers end-to-end delay to be symmetrical, no matter the direction of the signal flow.

Second method looks at the system time of the labs. It compares the recording system time and the lab system time.

First method showed a delay of 12.3ms from LAB1 -> control room, and 11.7ms from LAB2 -> control room. All times were measured via the central sessions server.

Second method showed a delay of ~8ms LAB1 -> control room for all conditions, and 9-10ms for LAB2.

Authors are not sure why the 2 methods gave two different estimations. One of the possibilities is the fact they’re not using the full NTP (Network Time Protocol) system, but only a simplified client for customer-grade systems. With a full NTP setup and consistent time servers, synchronization could likely improve.

Synchronized EEG responses proved the system capable of supporting remote hyperscanning

Scientists found a motor related potential in the participant who played the piano, during finder movement.

Also, in both participants we can expect to see an auditory evoked activity. Let’s take a look at the ERP plot:

This time shift (N1: 32 ms, P2: 44 ms) is slightly larger than the transmission delay of the acoustic signal (24 ms). It could be due to older age of participant 1 (56 versus 25 years) but this is purely speculative since there’s only one pair of participants.

What about the joint clapping task? The leader role alternated between labs, and you can see the lag results in both cases here:

We can see that participant in Lab1 was early compared to the leader in the second lab. Authors also found ERP responses during this task, for both participants. They would use the clap timing of each participant as the event.

Aside from EEG, the system also proved capable of supporting natural turn-taking communication, with minimal delay. This is essential for interpersonal synchrony studies.

OVBOX is developed for low-delay audio applications. However it does lack integrated video transmission, which limits its use. Still, virtual avatars and mixed reality setups could simulate audio-visual interactions. And external tools could manage video at the cost of potential delay.

Hyperscanning – close proximity and telepresence settings

If you’re looking to do any type of Hyperscanning in your study (remote or not), feel free to reach out to us. Also, check out our Hyperscanning guide for more info on the setup.

The original study can be found here – https://ieeexplore.ieee.org/abstract/document/10704205