Brain-computer interfaces (BCIs) are becoming increasingly sophisticated. BCI with AR combines neural decoding with augmented reality to create intuitive and immersive control systems.
In this blog, we explore how BCI with AR can enable users to operate a robotic arm using eye tracking and motor imagery. It highlights the potential of this technology in many important fields, such as neurorehabilitation.
Importance of picking the right system components in a BCI setup
Authors highlighted the key factors in choosing the right BCI components. Getting the intended functionality requires good coordination between multiple elements. Here’s the requirements researchers had in their study:
A suitable EEG device is essential for measuring participants’ neural activity. First, the system should be compatible with the AR headwear. It should allow access to the raw, real-time EEG data for custom decoding in the BCI system. The EEG device should be mobile to ensure the BCI with AR system is not restricted in any way.
For UI interaction they need an AR head-mounted display with eye-tracking. The device should be portable, ideally stand-alone, and provide access to built-in sensors for spatial computing.
The assistive robot should be a cobot that can safely operate around humans. It must be capable of picking up and placing objects precisely.
(top – a) Schematic representation of the hardware components and their collaboration (bottom – b) physical setup of the equipment in the study.Final selection of the hardware elements:
- mbt’s Smarting PRO X 64-channel system for EEG collection
- Microsoft HoloLens 2.0 for the head-mounted AR display
- The Franka Research robot was employed as the cobot
- Real-time EEG decoding was handled on a personal laptop
Control strategy of the system
Because of eye-tracking, participants’ gaze can be followed to identify which object they’re looking at in the AR environment. If a participant fixated their gaze on an object, that object is selected.Next, the system will suggest possible actions for the object. To choose an action, a participant is supposed to imagine the movement. The model then tries to predict the movement based on the EEG data. Once a decision threshold is met, the system will ask for confirmation before executing the action.
If the prediction is incorrect, the user needs to try again. If it’s correct – the robotic arm will execute the movement. You can find more detail on the tasks in the Study Phases description.
Software architecture of the BCI with AR system
Two main components in the software architecture are:
- EEG decoding application
- AR UI application
They communicate by a messaging software – ZeroMQ. This communication is ensured through a publish/subscribe strategy. That is, one component (the publisher) sends messages to a que provided by ZMQ, while the other listens (subscriber).
The purpose of a message is determined by assigning it to a specific queue. There are two que topics in the software – “decoding control” and “predictions”.
The EEG-decoding app listens to the “decoding control“ topic and sends results to “the predictions“. The AR UI app sends messages to the “decoding control“ topic, and receives results from the “predictions“.
Schematic representation of the software components in the study.System validation study in 3 phases
To allow improvement of the prototype control system, researchers defined 3 phases. During the phases, participants had different tasks to do.
Task 1: Calibration and Training (phase 1 and 2)
Participants completed a calibration task to train the system on their individual EEG data. During this phase, they wore an EEG cap and an AR headset. A fixation cross appeared followed by a cue instructing them to imagine either a left- or right-hand movement. This process repeated until enough data is collected to for the personalized model.
Task 2: Robot Control Tasks (phase 1 and 2)
When calibration is over, participants used their trained model to operate a robotic arm. The first task is a sorting exercise. They controlled a virtual robotic arm to move colored cubes into corresponding baskets. They are guided through AR instructions, and the task repeated until the end of required number of trials. The purpose is to see how well the BCI with AR system could decode the user’s intent.
Phase 3: Real-World Implementation
Authors introduced a second task in phase 3. It was a pick-and-place scenario, where participants selected objects and assigned actions for the robot to execute.
The final phase introduced a real robotic arm, adding a a more realistic and noisy environment. The process followed the same structure as earlier sessions, starting with calibration, followed by pick-and-place runs. To end the session successfully, participants had to execute the sequence of object placements within a time frame.
BCI with AR model success rate stayed similar across study phases
In Phase 1, all participants were able to complete at least a single sorting task. This showed feasibility of the system and going further to phase 2. Let’s take a look at the model performance metrics during this phase:
Online decoding performance in Phase 2, for each participant. The first 3 participants were a part of phase 1, and then participants 4-8 came into phase 2. You can take a look at the original paper to see in detail how different metrics are defined.The decoding performance is very low for most participants. For P5 and P6 the k value goes to ~0 indicating that the system repeatedly made wrong predictions. This is also reflected in their low success rate. However participants 4, 7 and 8 had a success rate of 1, showing that they were always able to complete the task withing the time limit.
Phase 3 performance metrics during the pick-and-place evaluation task.The decoding performance in Phase 3 showed slight improvements over Phase 2. Accuracy is higher with lower variance, and k values are also higher. However, the mean F1 score seems to be lower. So the system sometimes struggled to reliably distinguish between movement classes. This lead to more random misclassifications. Despite this, the overall success rate remained similar between the two phases.
How do different time limits for the task completion affect the system’s effectiveness?
Let’s first take a look at how different time limits affected the task success rates.
Participant success rates against different cutoff times. Phase 2 shows the sorting task, and Phase 3 the pick-and-place success rates.Results showed that most participants (83%) in phase 3 were able to complete their assigned task successfully. In phase 2 that’s 73%. The pick-and-place task in Phase 3 is finished more quickly than the sorting task. This is likely because it required fewer actions (4 vs. 10).
The sorting task did take longer due to its structure, with most participants needing over 10 minutes. Compared to the pick-and-place task that took around 6 minutes. The large standard deviation in completion times highlight individual differences in system performance.
Regarding system efficiency in these cases, we can see the results below:
(a) Mean completion time over all participants; (b) individual participant’s best completion time.Completion times are lower for the pick-and-place task than for the sorting task. This is as expected due to the fewer required actions. To add, individual performance varied significantly. This indicates that while some users struggled, others were able to complete the tasks efficiently.
To summarize the strengths and limitations of the proposed BCI with AR system
The study demonstrates the potential of a portable, multimodal BCI system for robotic arm control. By combining fast calibration, shared control approach and a simple UI, the system offers a user-friendly way to interact with the environment. This system can have great real-life application, such as in neurorehabilitation.
Some challenges do remain. The system’s efficiency is limited by occasional misclassifications, and participants had little time to improve their performance. Authors note that future work will focus on optimizing the decoding pipeline. This includes integrating advanced learning methods, and improving user experience through gamification.
A long-term goal is to develop an all-in-one device—a compact AR headset with built-in EEG sensors and processing hardware. If made accessible and affordable, the system could bring consumer-friendly BCI technology closer to reality.
Find the original work linked here!
Curious if this setup is right for your study?
Our team works with researchers across neuroscience, sports science and clinical applications - we'll help you figure out if mobile EEG fits your study design.
