Cutting-edge research in attention decoding is changing how we approach hearing technology. Using brain-computer interface (BCI) systems, they let users control sound focus with their thoughts. By analyzing brainwaves – EEG in real time, these innovations promise more natural, seamless hearing assistance. But significant challenges remain before they reach everyday use.
Breaking down the real-time auditory attention decoding system. EEG and auditory sources are fed to an attention decoder in real-time. Based on their characteristics, the decoder communicates which speaker to amplify, and who to tone down. The decoder tries to find out which speaker is the participant focusing on based on these measurements. Go to the next few chapters for details!About the WHISPER hardware platform
Hearing aids can struggle in noisy places. Their built-in microphones can’t fully separate speech from background noise.
To pick up mixed sounds in the experiment, researchers developed WHISPER – a wireless microphone array.
WHISPER consists of multiple nodes that can contain up to four microphones. When you place these nodes in a room, they work together to pinpoint and amplify the sound you want to hear. The system can dynamically adapt to different microphone positions, allowing for larger distances and better spatial filtering.
Key features of the system include:
- Wireless synchronization between nodes;
- On-device beamforming* which minimizes latency;
- Scalability: performance gets better with more nodes.
*Beamforming: signal processing technique that combines audio inputs to amplify sounds from specific direction – an “acoustic spotlight”.
WHISPER platform is described in detail in Kiselev et al. (2017) and Ceolini et al. (2020b).
WHISPER module: on the left, we can see the look of the components. On the right, an illustration on how the module works.Real-time attention decoding system – overview
The WHISPER device is then utilized in the attention decoding system. Let’s take a look at the rest of system components:
Decoding pipeline using EEG and auditory inputs
The decoder receives two inputs: one comes from the previously described WHISPER module – auditory. And the other is EEG data that’s acquired using the Smarting EEG system.
The goal of the system is to find which auditory source the participant is focused and listening to. That way, it can adjust the gain of that source i.e. amplify the sound for better hearing. Here are the stages that the data goes through:
- Both EEG and auditory signals are pre-processed
- Linear decoder based on canonical correlation analysis (CCA)
- Classifier that gives the final output
The CCA model needs to be trained prior to real-time application. So authors of the study used training data to define CCA parameters – both EEG and audio. In the image below, you can see detailed steps of the pipeline.
Blue – pre-processing of EEG and audio. Yellow – CCA transformation of EEG/audio data. We can see that the parameters for CCA are pre-defined in offline analysis. Green – classification of attention. Grey – audio gain control which we will go over next.Audio control – how to amplify the auditory source?
In the study, there are two different ways of adjusting the volume in real-time:
- Instant control – leave the gain unchanged unless the classifier reached some level of certainty. Then, change the gain of the source accordingly. The downside of this simple method is sudden volume jumps. This can be very bad for the listener.
- State tracker control – it takes into account previous states i.e. outputs of the classifier. It will change volume gradually when unsure, and faster when confident.
Live demonstration with real talkers
Take a look at the system in action in the videos below. First we have mixed audio recorded from the room before applying source separation:
Then the speech separated audio, where it’s demonstrated how the final output sounds. When a speaker raises their hand, the participant is suppose to shift their attention to them:
Experiment I: pre-live demonstration with normal- and impaired-hearing listeners
In this experiment, authors used pre-recorded voices instead of real-time speakers. Let’s take a look at the attention changes during this experiment:
(A) Participants are suppose to switch their attention to a different speaker when cued. (B) Output of the classifier as a function of time – at time 45s, is when the speaker switch happens! We can see results for both hearing-impaired (HI) and normal-hearing (NH) participants. (C) Final gain applied to the two speakers.We can see that the model performed similarly for HI and NH participants. However, in this phase authors applied the Instant control of the gain. As expected, participants reported very erratic changes in the volume. They characterized these as “highly disturbing”.
So even though the model performed very well, the real-time application has low tolerance for mistakes. Even if this error is around 10% of the time..
Experiment II: pre-live demonstration with audio-visual speech
Auditory attention decoding can be done while only hearing the speakers. But in a real life situations, people usually look at the speakers while listening. In this experiment, half of the participants got a visual of the speakers, in addition to the recorded voices. The other half had no visual of the speakers.
In this experiment, they used the State tracker method for controlling gain. The transitions in gain were a bit smoother here, but the speed of this method is much slower.
Even though offline data showed better system accuracy when the speaker is visible, in online application that wasn’t the case. Both groups of participants “steered” the BCI system in the same fashion.
What are the next challenges for this promising attention decoding system?
This proof-of-concept system successfully enabled hearing-impaired participants to naturally control audio focus – by only using their listening attention! Unlike traditional systems, researchers here analyzed both EEG and acoustic features to get the result.
However, there’s a big challenge – the trade off between response speed and accuracy. Fast detection equals frequent errors, but increased processing time can’t follow the pace of a regular conversion. This was obvious to both researchers and the participants.
One more thing to consider is (real) real-world applications. Even though the setup resembled a naturalistic conversation including two speakers, what about more? Or in a noisier environment? If we went to a different location, how do we distribute the WHISPER hardware modules?
Looking forward, authors suggest promising directions of improvement. These include incorporating deep learning, hybrid control systems (automatic and manual), training protocols for participants and more. Alternative audio processing could also go beyond simple gain adjustments.
This research is a significant milestone for auditory attention decoding technology. But it also offers a look into the real world challenges that need to be addressed.
All videos and images in this article are from the original paper by Jens Hjortkjær et al 2025. Make sure to take a look at their study in detail.
Curious if this setup is right for your study?
Our team works with researchers across neuroscience, sports science and clinical applications - we'll help you figure out if mobile EEG fits your study design.
