The Guide to Hyperscanning with EEG

Hyperscanning is the name for the method of recording and analyzing multiple human brains at the same time. Instead of just studying one person’s brain, hyperscanning lets us dive into the neural dynamics of a group or pair of individuals engaged in social interactions.

Hyperscanning techniques are getting more and more popular in the field of neuroscience, as the researchers are getting increasingly interested in understanding how social interactions affect mental states. 

But hyperscanning is not an easy study to set up. There are still plenty of challenges that scientists and technology companies are grappling with. They’re working hard to figure out the best ways to accurately capture and make sense of the complex data that hyperscanning generates.

In the upcoming sections, we’re going to dig deeper into these challenges, explore the methods being used, and take a look at some real-life use-cases. So, have fun learning about it!

But before we jump right into the topic, let’s give you a rundown of what we’re going to cover on this page.

Hyperscanning Scenarios

Now we’ve set the stage and introduced hyperscanning as a method of studying multiple human brains at the same time. But, what insights do we get from this?

The pioneering work on hyperscanning was addressing the questions of imitation, cooperation and competition. In the following years, the experiments diversified significantly and now they cover more complex interactions.

This diversification was to take the social interactions to their more natural settings (education, workplace, solving creative problems in teams, music and performing arts). The reason was that researchers wanted to understand how people interact in these real-life scenarios.

Finally, the most recent research took it to situations where the communication may be challenging or unsuccessful.  The examples are the mother-infant connection, or to to enhance therapist-patient communication. This is studied in autism, schizophrenia, psychotherapy, neurofeedback therapy, couples therapy and others.

The researchers concluded that during the performance of all those tasks, two major neural systems are largely involved. The mirror neuron system (MNS), which plays an important role in tasks involving movements, such as imitation and coordination/joint tasks. 

The other is a mentalizing system (MS), which is engaged in tasks pertaining to the inferences of yourself or others’ intentions or thoughts, such as the economic game and natural social interactions.

More thorough review of different types of studies can be in Meng-Yun Wang et al.

But what makes hyperscanning studies truly distinctive and invaluable? What scientific questions can we answer using hyperscanning that regular brain recording paradigms cannot fully address?

Why are Hyperscanning Studies Important?

To address this question, we organized a series of webinar/podcasts where we talked with our guests who have experience in hyperscanning studies. Among the guests we had Guillaume Dumas, Suzanne Dikker, Martin imhof and Karl-Phillip Flösch.

The most important reason to do hyperscanning is that laboratory setting lacks ecological validity. It removes natural environment for social interaction, which is the exact aspect we are interested in.

In the words of Guillaume Dumas:

I believe that embodiment is not only about the body of the person, but also about the embodiment of the social realm. And we should adopt cognition not only from the individual point of view, but consciousness and cognition is a public affair and we have to bring back the social dimension in the study of human cognition.

Our social diynamics and social interaction are both constitutive of our individual cognition…

In the end, he noted that “there are also a lot of people who would not necessarily agree with that”.

Guillaume pointed out that, paradoxically, the social neuroscience has stayed blind to the large part of researching social interaction in humans, which is the actual interactive context (interesting reading is Pfeiffer et al.). They call it the “dark matter of neuroscience”, as there were very few papers covering 2 individuals engaged in a dynamic interaction. 

But does it mean that we should abandon the labs and do social neuroscience studies exclusively in a natural environment? Absolutely not. We should still keep the luxury of laboratory-controlled conditions to test our hypotheses and validate our real-world conclusions. As Martin and Suzanne put it:

it’s not the question either or (laboratory and real-world recordings), it’s really the two things that can inform each other. 

It’s the cyclical process where you’re testing in real world the questions you might not be able to answer in lab, or you are testing your laboratory model in the real world. The laboratory studies generate findings that you might only be able to decipher by controlled laboratory study. 

But in the end the only way to get at the mechanisms underlying social interaction is to do real-world research.

As Guillaume points out, hyperscanning is complex and it is not always the best way to go. Before you start, you should always ask yourself the question whether or not are you able to test your scientific hypothesis without going into hyperscanning setups. If the answer is yes, just spare yourself the pain and do an in-lab study.

But if the answer is “no”, here are some guidelines on how you can do it and challenges you will have to overcome. 

Popularity of Hyperscanning in Science

The popularity of hyperscanning methods grows over time, as the devices and systems become more developed to cover for this complex topic. Our guests agree that it was a pain to publish a hyperscanning paper in 2010. Despite the fact those were the pioneering studies in this now expanding field. 

Today, the papers get more easily published in high impact journals, as the topic became more mainstream. 

The consequence of the easier publication is that sometimes the results get overinterpreted and the conclusions may not always be firmly scientifically grounded. This is where the hyperscanning hype becomes a little more concerning.

Further in the text we will list the challenges of the environment and what may cause the spurious results and how to carefully avoid misinterpretations.

What Modalities can you use with Hyperscanning?

Normally, hyperscanning can involve different neuroimaging techniques. Those are electroencephalography (EEG), functional near-infrared spectroscopy (fNIRS) and functioinal magnetic resonance imaging (fMRI). 

Now, each technique has its own strengths and its drawbacks; for instance, fMRI can capture really detailed functional images of the brain, but it’s really slow (think about 1 measurement every 2 seconds). And the machinery is extremely bulky. That’s not great for studying real-time social interactions, which often require more spontaneity and movement.

That’s where EEG and fNIRS come in. These techniques can record brain activity quickly and accurately, and can be mobile, so they’re much better for studying social scenarios. Given our team’s expertise in mobile EEG, we’ll be focusing on this method in the discussion that follows.

Stay tuned as we delve further into the world of EEG for hyperscanning!

Hyperscanning with EEG

Why is EEG the tool of choice for hyperscanning studies?

One of the basic assumptions in hyperscanning is the so-called, inter-brain synchrony. This basically means that researchers expect alignment of the brain oscillations when two brains socially interact. As EEG, by definition, captures brain oscillations, it is a perfect tool to explore the inter-brain synchrony.

EEG is one of the most powerful technologies for non-invasive brain activity monitoring in general. When populations of neurons simultaneously fire in the cortex, they produce brain oscillations. As neurons fire electrical impulses, those oscillations are captured by electroencephalography (EEG). 

The temporal resolution (speed) of EEG recordings allows to track the social interaction on the moment-by-moment basis, and to instantaneously relate brain activity to the social interaction of the participants.

Another important aspect is mobility. Today, the portable EEG devices (such as the ones produced by mBrainTrain) are extremely portable and can be easily used in a social setting. Contrary to that, fMRI machines are stationary, and cannot fit multiple people, so the social interaction can only be digital (through a video link for example). 

Finally, even though EEG signal can be corrupted by movement artifacts, they can be cleaned with various (automatic or manual) artifact rejection techniques (such as ASR). On the other hand, motion artifact rejection in fMRI is much harder. As the humans gesticulate while communicating, EEG allows for this body talk, whereas fMRI is much more restricted in that sense.

How to setup a hyperscanning experiment

What are the challenges to use EEG for hyperscanning?

To address the challenges of hyperscanning setups, we will cover a number of topics:

• Challenges from natural environment
• Challenges from natural similarity
• Challenges from movements
• Time synchronization 
• Using wired vs wireless systems

The influence of the natural environment

We mentioned earlier that the environmental sources may introduce signal artifacts (in terms of oscillations). As these signals will be captured by all the recorded systems (in all the subjects), that may appear to show correlations between participants. If not careful enough, these correlations may be mistaken for brain synchronization.

The simple example of this may be an electromagnetic field (the powerline source) that produces oscillations at the frequency of 60Hz (in USA, or 50Hz in Europe). Older EEG systems without common mode rejection ratio (CMRR) didn’t get rid of this noise by design, and so in the earlier studies (e.g. Dumas 2010), they needed 1 Faraday cage per participant to make sure the recorded oscillations do not have the origin in the environment. This, in turn, limited the naturalistic environment for the subjects.

Fortunately, more modern systems (including the EEG devices from Smarting PRO Line), get rid of some of these environmental noise through high CMMR.

The influence of the natural similarity (shared anatomy)

In the end, we are all humans, and the similarities between our brains are already such that the correlations between the EEG signals are already significant. Especially when the subjects are placed in the same environment. Guillaume jokes that it is less unexpected to detect correlation than not to identify any. 

He also explains why setting up the experimental paradigm correctly is very important:

“Carefully crafting the scientific question is crucial to be able to distinguish the synchronization related to interbrain interaction, from the synchronization coming from just being in the same environment.

Now, beyond neurobiological similarity, there is also a behavioral similarity. This is a consequence of two people doing the same task, for example. In some cases, we might be interested in exactly this phenomenon. Or even further, 2 people doing the same task, coming from the same culture are even more similar. This similarity aspect we sometimes call shared anatomy.

And then there is another aspect, which is the communication aspect. And this aspect is what I am mostly interested in. So, what are the physiological changes passed through unidirectional or bidirectional communication.”

To study the interaction (communication) aspect, we should carefully remove the correlation coming from the baseline (shared anatomy) and study only the correlation that is left.

As Guillaume says, hyperscanning studies allow us to move from studying:

• Social perception to social interaction
• Sequential interaction to real-time interaction
• Intra-brain correlations to inter-brain correlations
• Externally driven synchronization (through external metronome for example) to endogenous synchronization.

Wired vs Wireless

When the experimental paradigm is carefully set, we should decide on the system to use. Generally, today the EEG systems may be divided in two groups – wired and wireless systems. Here are some pros and cons for using any of them and how did we address these challenges in Smarting PRO Line systems.

In mbt, when we design EEG systems, we usually perform interviews with our users, and researchers who we believe may potentially become the users of our systems. We do this to gain experience that we don’t have, because we design our systems for researchers and not for ourselves. And our goal is to help them do their job easier and better.

We talked with our guests Suzanne Dikker, Martin Imhoff and Karl Phillip Flösch on our hyperscanning webinar, and one of the topics was wired vs wireless systems.

Challenges when using Wired Systems

The drawbacks of the wired systems are obvious. 

First, it is very difficult to set it up. As a researcher, you normally need multiple EEG systems, then you need to attach synchronization apparatus, and then connect all of that to a PC by wire. This creates a very much unnatural context (which is in contrast to the idea of hyperscanning study – to observe people in the social environment). 

But even if we put it aside, moving in between the blocks, is a nightmare (for example if people want to visit toilet, or just the experimental paradigm requires them to switch places).

As Karl-Phillip says:

wireless is crucial for us. We have an experiment where participants play a game, and at some point 2 of the 8 have to switch positions. Doing this with a wired system is impossible – you completely ruin the real-world social setting you try to set up.

Another reason to go wireless is that setting up multiple people (sometimes even 8 or more, as in the case of Karl-Phillip’s study) can take forever with wired systems. And it is not convenient for people to wait that much. 

In the study of Suzanne Dikker (reference), she did a classroom study. In that case, they have 55 minutes in total to setup the experiment, collect the data and set the subjects free. As you know, this is a huge challenge for EEG studies. With wired systems, this would be close to impossible.

A regular schematics of connecting 2 wired systems in a hyperscanning setup is shown in the figure below. The complexity of the setup significantly increases with the increased number of subjects. The schematics is based on figures from the paper by Paulo Barraza et. al.

Challenges when using Wireless Systems

All of the above are the reasons to go with the wireless EEG systems. But these systems, on the other hand, may come with the set of their own challenges.

Bandwidth

First of all, by definition, wireless devices use no cables. Thus wireless, right? And so the connection stability and reliability is a challenge that equipment vendors (like mBrainTrain) are facing.

The connection reliability is even more vulnerable due to interference of multiple devices streaming continuous data at the same time. In addition, in order to synchronize all this data, all the EEG systems should stream to the same recording device, that takes care that all the recordings are in sync.

Range

Further to the bandwidth challenges, there is an issue of the streaming range. In one of the studies of Suzanne Dikker (reference), they had people playing in an orchestra on stage. And what happens is that a subject walks out (to the corridor) and goes out of range. 

It’s impossible to just get on the stage to fix the connection, and you lose the whole recording. So, then what is the point of wireless systems, if the subjects cannot freely walk around? And as everyone already has a mobile device, why not just connect to them?

These 2 issues lead sometimes to data loss, which makes recording multiple subjects in a wireless setup challenging. Suzanne says that in one of the recordings, out of 5000 recorded subjects, they could use only around 200 pairs. So, 80% of the data was discarded, which is huge.

Synchronization

To ensure the success of a hyperscanning experiment, it’s essential to synchronize the EEG data from all the subjects. There are several methods available for synchronizing EEG data with other modalities or other EEG recordings with TTL and LSL being the most used ones.

TTL, which stands for Transistor-Transistor Logic, is a hardware-based synchronization method. It offers a straightforward approach to synchronize multiple data streams. However, TTL typically requires wired synchronization, which can limit the subjects’ freedom during social interactions. On the positive side, TTL provides the highest level of synchronization accuracy as triggers are sent through a wired protocol, such as analog parallel or digital RS232.

In contrast, the Labstreaming layer (LSL) enables software-based synchronization between multiple devices (the LSL GitHub repository can be found here). LSL allows for wireless data synchronization, handled by the software platform. The synchronization precision achieved with LSL is typically at the millisecond level, which is optimal for most experiment setups. 

However, it’s important to note that achieving high synchronization accuracy relies on a reliable wireless connection. If the Bluetooth connection or the WiFi network interconnecting the devices becomes unstable, the synchronization accuracy may be compromised.

At mbt, we strongly recommend using LSL with our systems due to the robustness of our Bluetooth connections. However, TTL serves as a reliable fallback alternative and is available with all our PRO Line solutions.

The way we solve these at mBrainTrain?

mBrainTrain was born with the purpose to enable researchers to do out-of-lab EEG recordings with ease. And to do that, we put a lot of attention around the wireless connection and communication. 

To address these challenges, we implement a number of solutions:

1. We use Bluetooth 5.0 technology in our Smarting PRO Line devices. The throughput of Bluetooth 5.0 is much higher than that of the previous Bluetooth modules due to PHY 2M connection.
2. We enable recording on mobile phones. This makes it easier to always be in range (as you the subjects can just always have their phone with them.
3. We connect our EEG systems remotely – we don’t have any buttons on our EEG devices, and so you don’t need the direct access to the subject in order to connect the device.
4. We store data to the internal memory (or an SD card), so the data is always accessible, even after the connection may have been lost.
5. We stream a restricted necessary subset of data wirelessly – in this way we reduce the needed bandwidth. The full data set is stored internally. So, when the experiment is finished, the experimenter simply goes takes the data from the device, and they can simply be synchronized offline.

All these make Smarting PRO Line systems resilient to harsh conditions. And we are still continuously improving them to remove this trouble from the researchers back.

For comparison with the wired EEG systems setup shown above, the figure below represents the standard hyperscanning setup with wireless EEG systems. Increasing the number of participants does not proportionally increase the complexity of the study setup. 

Processing Hyperscanning data

The complexity of the data

But recording and collecting data is only part of the issue. Another part comes from processing this complex data. The data is recorded from multiple subjects, that are mutually synchronized. Then there is usually some type of external stimulus (a video for example) that is then also synchronized with EEG data. 

And as the data becomes more complex, so does the analysis approach. As Martin says:

“In the beginning we used the tools that we developed during our previous study when the data was collected sequentially – people watching the same movie, but not at the same time. But now, new tools are to be developed”

As hyperscanning is a new field, not many tools have been developed so far and the researchers are mostly on their own when it comes to processing the data. However, we did identify one toolbox (still in beta) that might be useful for new (but also experienced) researchers stepping into the field. If you stumble upon other software packages for processing hyperscanning data, please share them with us, and we will include them in this text. 

HyPyP processing pipeline

Until a few years back, there is no agreed-upon analysis approach to carry out inter-brain connectivity analysis. One such Python based processing pipeline (HyPyP) was introduced by Anaël Ayrolles, Suzanne Dikker and Guillaume Dumas et al. in 2020. 

You can also find the github repositorium on this link. 

We recommend you look at the package and try it out. You may also provide your own contribution. If you have something more you would like us to share here, get in touch with us and we will update this post.

MoBILAB for EEGLAB

If you are using Matlab for EEG processing, the EEGLAB extention MoBILAB implements loading .xdf files containing multiple EEG streams. If those streams are recorded during a hyperscanning session, MoBILAB automatically aligns these streams into an EEGLAB set and they can be further processed in EEGLAB or MATLAB.

MoBILAB is originally made for synchronized multimodal recordings but can be exploited for hyperscanning data.

Visit their github repositorium or read more of MoBILAB documemtation.

Why mBrainTrain systems

In mbraintrain, we develop wireless systems since the beginning of the company. Our purpose is to enable researchers to do out-of-lab EEG recordings with ease. And to do that, a lot of attention was put around the wireless connection and communication. 

To address the challenges above we do a number of things:

1. We use Bluetooth 5.0 technology in our Smarting PRO Line devices. The throughput of Bluetooth 5.0 is much higher than that of the previous Bluetooth modules due to PHY 2M connection.
2. We enable recording on mobile phones. This makes it easier to always be in range (as you the subjects can just always have their phone with them.
3. We connect our EEG systems remotely – we don’t have any buttons on our EEG devices, and so you don’t need the direct access to the subject in order to connect the device.
4. We store data to the internal memory (or an SD card), so the data is always accessible, even after the connection may have been lost.
5. We stream a restricted necessary subset of data wirelessly – in this way we reduce the needed bandwidth. The full data set is stored internally. So, when the experiment is finished, the experimenter simply goes takes the data from the device, and they can simply be synchronized offline.

All these make Smarting PRO Line systems resilient to harsh conditions. And we are still continuously improving them to remove this trouble from the researchers back.