{"id":"CU25368","slug":"neural-attention-driven--CU25368","source":{"id":"CU25368","dataset":"techtransfer","title":"Neural Attention-Driven Auditory LLM for Selective Speech Understanding","description_":"<p>This technology is an auditory large language model that infers which sound source a person is attending to in complex environments and generates listener-aligned responses for assistive hearing and adaptive voice assistants.</p>\r\r<h2>Unmet Need: Listener-intent decoding for selective speech in noisy environments</h2>\r\r<p>Current auditory AI and speech-processing solutions primarily optimize source-agnostic transcription and comprehension, treating all concurrent speakers as equally relevant. In multi-talker, noisy environments, this approach cannot determine the user’s intended talker, leading to inaccurate or irrelevant outputs and reduced effectiveness in assistive hearing and voice-interface applications. A critical unmet need is the ability to align system outputs with the listener’s intended auditory focus to improve reliability, usability, and real-world performance.</p>\r\r<h2>The Technology: Neural attention-driven auditory LLM for aid in noisy environments</h2>\r\r<p>This technology is an auditory large language model (LLM) system that integrates brain signals to infer listener’s auditory attention in noisy environments. Using time-synchronized intracranial EEG (iEEG) recordings, the system identifies the sound source a listener is attending to and uses this information to guide the LLM toward the intended speech stream while suppressing competing background sounds. The platform can also incorporate auxiliary cues such as head orientation and eye gaze, to improve performance in dynamic, multi-speaker environments.</p>\r\r<p>This technology has been validated through real-time testing in human participants.</p>\r\r<h2>Applications:</h2>\r\r<ul>\r<li>Speech-processing technologies, including assistive hearing technologies and voice assistants</li>\r<li>Assistive technologies for hearing impairment</li>\r<li>Audio source separation in complex acoustic environments</li>\r<li>Neuroadaptive audio systems for high-noise workplaces</li>\r<li>Clinical neurotechnology for auditory attention monitoring</li>\r<li>Communication systems for aviation, defense and emergency response</li>\r<li>Cognitive neuroscience research on selective auditory attention</li>\r<li>Meeting and conferencing platforms with attention-guided transcription, summarization, and question answering</li>\r</ul>\r\r<h2>Advantages:</h2>\r\r<ul>\r<li>Listener-specific speech processing in multi-speaker environments</li>\r<li>Improves speech understanding in noisy environments</li>\r<li>Real-time adaptation to attention shifts</li>\r<li>Integrates multimodal intent cues including eye gaze and head orientation</li>\r</ul>\r\r<h2>Lead Inventor:</h2>\r\r<p><a href=\"https://nima.ee.columbia.edu/\">Nima Mesgarani, Ph.D.</a> </p>\r\r<h2>Patent Information:</h2>\r\r<p>Patent pending</p>\r\r<h2>Related Publications:</h2>\r\r<ul>\r<li><a href=\"https://pubmed.ncbi.nlm.nih.gov/42115433/\">Choudhari V, Nentwich M, Johnson S, Herrero JL, Bickel S, Mehta AD, Friedman D, Flinker A, Chang EF, Mesgarani N. “Real-time brain-controlled selective hearing enhances speech perception in multi-talker environments” Nat Neurosci. 2026 May 11.</a> </li>\r</ul>\r\r<h2>Tech Ventures Reference:</h2>\r\r<ul>\r<li><p>IR CU25368</p></li>\r<li><p>Licensing Contact: <a href=\"mailto:techtransfer@columbia.edu\">Kristin Neuman</a> </p></li>\r</ul>\r","tags":["Acoustics","Artificial intelligence","Assistive technology","Cognitive neuroscience","Hearing loss","Language model","Speech perception","Speech processing"],"file_number":"CU25368","collections":[],"meta_description":"Neural attention-driven auditory LLM decodes listener focus via brain signals to enhance selective speech understanding in noisy, multi-speaker settings.","apriori_judge_output":"{\"scores\":{\"novelty\":5.0,\"potential_impact\":5.0,\"readiness\":4.0,\"scalability\":4.0,\"timeliness\":4.0},\"weighted_score\":4.65,\"risks\":[\"Ethical/privacy concerns with brain signal data\",\"Regulatory approvals for medical/assistive use\",\"Hardware requirements limit deployment\",\"Reliance on invasive iEEG may restrict clinical/real-world adoption\",\"Computational latency in real-time multimodal integration\"],\"one_sentence_take\":\"High novelty and potential impact with strong readiness, but practical deployment hinges on invasive data handling, regulatory clearance, and latency management.\"}","inventors":["Nima Mesgarani"],"manager":"Kristin Neuman","depts":["Electrical Engineering"],"divs":["Fu Foundation School of Engineering and Applied Science (SEAS)"],"date_released":"2026-08-21"},"highlight":{},"matched_queries":null,"score":0.0}