Research on Smart Headphones featured on IEEE Signal Processing Magazine

March 26, 2018

Immersive technology makes audio more realistic and engaging, yet for some listeners audio can sometimes become a bit too immersive. At Columbia University in New York, a professor is leading the design of an intelligent headphone system that will warn distracted pedestrians of imminent dangers posed by oncoming vehicles.

Stephen Xia

With input from miniature microphones, the system uses intelligent signal processing to detect the sounds of approaching vehicles. If a hazard appears imminent, the system sends an audio alert—a beep—to the headphones, interrupting the audio stream and alerting the wearer that danger is rapidly approaching,” explains Fred Jiang, a professor of electrical engineering in the university’s Electrical Engineering Department. The threat’s direction and distance is represented by the relative loudness—in stereo—of the beep. “The beep will happen while the car is still 50 meters away, giving the pedestrian plenty of time to react and avoid a potential fatal encounter,” he notes.

While the technology’s main goal is warning pedestrians and joggers about approaching vehicles, there are also other potential applications. Construction workers, for instance, often wear headsets to muffle the sounds of heavy machinery. Yet conventional hearing protection products also prevent them from detecting sounds that might indicate an imminent threat. “Our system can be adapted to alert danger for them,” Jiang says. The system can also be adapted to fit on bike helmets and backpacks.

Daniel De Godoy Peixoto

While the technology’s main goal is warning pedestrians and joggers about approaching vehicles, there are also other potential applications. Construction workers, for instance, often wear headsets to muffle the sounds of heavy machinery. Yet conventional hearing protection products also prevent them from detecting sounds that might indicate an imminent threat. “Our system can be adapted to alert danger for them,” Jiang says. The system can also be adapted to fit on bike helmets and backpacks.

The four-year research project is supported by a grant from the U.S. National Science Foundation. The research team includes Peter Kinget, chair of Columbia’s Electrical Engineering Department; Shahriar Nirjon, a computer science professor at the University of North Carolina at Chapel Hill; and Joshua New, a psychology professor at Barnard College in New York. Graduate students from both Columbia and the University of North Carolina are also working on the project.

The technology involves embedding multiple miniature microphones into the headset as well as developing a data pipeline to process nearby sounds and extract the correct cues that signal impending danger. The system features a new, ultralow power, custom-integrated circuit (Figure 3) that extracts the relevant features from the sounds while using very little battery power.

Jiang says a prototype system offers promising early results (Figure 4). “The system includes a modified headset with embedded sensors, signal processing algorithms, and machine-learning (ML) classifiers in a smartphone application,” he says. “Signal processing is integral to our research, since our entire system is all about processing the sound and ultrasound signals from vehicles into useful alerts,” Jiang says. “The signals we use include the sounds of tires against gravel, the sound of air displacement as vehicles approach, engine sounds, and also honks.”

The signal processing pipeline starts in the analog domain with four channels of microelectromechanical system (MEMS) microphones. Signals from the analog sensors are amplified, sampled, filtered, and used to compute a number of useful features, such as cross-correction, zero-crossing, and signal power. On the digital side, frequency domain features relating to sound are computed using fast Fourier transforms, cosine transforms, and other techniques. “In particular, we created a set of frequency domain features specific to cars,” Jiang explains.