Optimizing Brain-Controlled Augmented Hearing for Noisy Environments with Moving Talkers

In a significant advancement for assistive hearing technology, Columbia Engineering students Vishal Choudhari (EE PhD ‘25) and Cong Han (EE PhD ‘23) have introduced a new generation of brain-controlled hearing systems that could assist individuals with hearing impairments and augment hearing capabilities for individuals with normal hearing. Under the guidance of Nima Mesgarani, Principal Investigator at Columbia's Zuckerman Institute and Associate Professor of Electrical Engineering, the research was published in Advanced Science, which delves into the development of brain-computer interfaces (BCIs) designed to help people better navigate challenging auditory environments. This pioneering study has garnered third place in the 2024 International BCI Award, earning them recognition among 80 global submissions from over 30 countries, as well as a $1,000 cash prize.

At the heart of their innovation is the ability to decode auditory attention, using neural signals and combining them with speech separation to enhance the speech of the person to whom the listener is paying attention. This breakthrough system, based on auditory attention decoding (AAD), addresses a critical need for individuals with hearing loss: the challenge of understanding specific conversations in noisy environments, where multiple interfering speakers are present. This difficulty, often referred to as the "cocktail party problem," has long been a major hurdle in the development of effective hearing aids. Most traditional hearing devices, while proficient at amplifying sounds and suppressing background noise, struggle to isolate the speech of a specific person in a crowd without knowing whom the listener is focusing on.

“Our study shows that using invasive human brain signals, we can decode listening intent—essentially identifying who the person with the implant is paying attention to in a noisy environment with multiple talkers,” says Choudhari. “This decoded information can then be used to selectively enhance the speech of the attended talker. A major feature of this system is its ability to work in real-life settings where there is background noise and moving talkers in space.”

The technology behind this system relies on neural signals captured through invasive intracranial electroencephalography (iEEG), collected from three neurosurgical patients. These signals are then processed to decode auditory attention and identify which speaker the listener is focusing on. The brain-controlled hearing system combines AAD with a state-of-the-art binaural speech separation model that isolates individual voices while preserving their spatial locations. This ensures that the listener can track conversations naturally as speakers move around.

A key innovation of the study is the introduction of a binaural speech separation model that works in dynamic, real-world scenarios. Unlike previous AAD studies, which have primarily focused on stationary talkers in controlled settings, Choudhari and Han’s system mimics realistic environments where speakers move around and take turns in conversations amidst background noise. This approach more closely mirrors everyday listening experiences, making the system far more applicable in real-world situations.

In their experiments, the patients focused on one of two moving conversations in the presence of background noise. The binaural model they developed successfully unmixed the speech streams, preserved spatial cues, and provided speaker trajectories that significantly improved the system’s accuracy in decoding auditory attention. This allowed the listener to track the attended talker more effectively, resulting in improved speech intelligibility and reduced listening effort in acoustically challenging environments.

One of the major findings of the study is the system's ability to enhance speech intelligibility in noisy conditions while maintaining the spatial characteristics and voice quality of the talkers. Subjective and objective evaluations of the system demonstrated its effectiveness in facilitating conversation tracking, even in the presence of background noise and competing talkers.

This breakthrough research represents a major step forward in developing brain-controlled assistive hearing devices that can adapt to the complex dynamics of everyday auditory environments. By combining neural decoding with advanced speech separation technology, Choudhari and Han’s system offers a solution that could revolutionize the way individuals with hearing impairments experience the world around them.

Moreover, the study highlights the importance of incorporating realistic experimental conditions in AAD research. Previous studies have often relied on oversimplified acoustic environments that fail to capture the complexity of real-world scenarios. In contrast, Choudhari and Han’s research introduces a more challenging and realistic task, involving multiple moving speakers and varying background noise levels. This approach not only improves the applicability of their system but also sets a new standard for future research in the field of auditory attention decoding and brain-computer interfaces.

“I believe that this technology has the potential to not just assist people who are hard of hearing but can also enable augmented hearing capabilities for individuals with normal hearing,” says Choudhari. “As seen in the evaluations, people with normal hearing saw a significant increase in ease of listening in noisy environments. Oftentimes, I go to conferences with poster sessions and there are 10s - 100s of conversations all around me. As a normal-hearing listener, I find it draining to keep up a conversation with a visitor to my poster. This technology would drastically reduce listening effort for both the poster presenter as well as the visitor in such scenarios.” 

The full study can be accessed through Columbia University’s e-library and a demonstration video of the system is available on YouTube, providing an insight into how this cutting-edge technology works in practice.