System for detection and amplification of target speech in multi-talker environments

This technology separates multi-source audio signals and filters the signals based on the listener’s brain activity.

Unmet Need: Methods to detect and isolate a target audio signal from multiple interfering sources

Speech perception in crowded environments is challenging particularly for hearing-impaired listeners. Assistive hearing devices have seen substantial progress in suppressing background noises that are acoustically different from speech, but they cannot enhance a target speaker without knowing with whom the listener is conversing.

The Technology: Real-time tracking of a listener’s brainwave to detect and amplify a target speaker in multi-talker listening conditions

This technology is a system for auditory attention decoding in which the brainwaves of listeners are compared with sound sources to determine the attended source, which can then be amplified to facilitate hearing. In realistic situations, however, only mixed audio is available. The inventors created a speech separation algorithm to automatically separate speakers in mixed audio, with no need for the speakers to have prior training. The results have shown that auditory attention decoding with automatically separated speakers is as accurate and fast as using clean speech sounds. The proposed technology significantly improves the subjective and objective quality of the attended speaker. The technology addresses a major obstacle in actualization of auditory attention decoding that can assist hearing-impaired listeners and reduce listening effort for normal-hearing subjects.

This technology has been demonstrated to cleanly separate and amplify a target speaker relative to the interfering sources. Learn more here.

Applications:

• Hearing aids and other speech enhancement devices
• Speech recognition software
• Noise reduction in voice communications and noise-canceling headphones
• Music enhancement (vocal track isolation)
• Security software or auditory forensics

Advantages:

• Brain-controlled speech separation
• Real-time implementation
• Insensitive to speaker number or order
• Faster processing and better performance than competitors

Lead Inventor:

Nima Mesgarani, Ph.D.

Patent Information:

Patent Status

Related Publications:

• Han, C., O’Sullivan, J., Luo, Y., Herrero, J., Mehta, A., Mesgarani, N., Speaker independent auditory attention decoding without access to clean sources, Science Advances, 2019. / Demo / Press Release / In the news

• Luo, Y., Mesgarani, N., Conv-TasNet: surpassing ideal time-frequency magnitude masking for speech separation, IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2019 / Audio Samples

• Han. C., Luo, Y., Mesgarani, N., Online deep attractor network for real-time single-channel speech separation, in Proc. IEEE Int. Conf. Acoustic, Speech, and Signal Process, 2019.

• O’Sullivan J, Chen Z, Herrero J, McKhann GM, Sheth SA, Mehta AD, Mesgarani N. “Neural decoding of attentional selection in multi-speaker environments without access to clean sources” J Neural Eng. 2017 Oct; 14(5): 056001.

Tech Ventures Reference:

• IR CU17276, CU18145, CU16343

• Licensing Contact: Beth Kauderer