LSAI: Language and Speech AI
Project Overview
Speech, language and voice carry clinically useful signals about child development and about neurological and respiratory health, but assessing them is labor-intensive and often subjective. LSAI develops AI methods along two lines. Child speech and language assessment turns language samples, acoustic recordings and articulatory ultrasound into interpretable measures for speech-language pathologists (SLPs), including automated child language-sample analysis, phoneme-level error detection and multimodal assessment of speech sound disorders. Voice biomarkers derive acoustic and multimodal markers from everyday speech and body sounds for Parkinson's disease, Alzheimer's disease and related dementias (ADRD), and respiratory conditions such as COVID-19. The goal is objective evidence for screening, assessment and longitudinal monitoring in clinics and everyday settings, while keeping clinical judgment central.
ESC Group Team
- Professor
- Wenyao Xu
- Postdoctoral researchers
- Wei Bo; Anarghya Das
- Ph.D. students
- Shuwei Hou; Varun Shijo; Chuhui Liu
- Undergraduate student
- Long Cao
- Former ESC members
- Hanbin Zhang; Aosen Wang; Matthew Rubino; Puru Soni
- Former K-12 members
- Sanjana Singh; Soumyadeep Bhattacharjee
Collaborators
- Ling-Yu Guo — Associate Professor, Communicative Disorders and Sciences, University at Buffalo
- Jun Xia — Professor, Biomedical Engineering, University at Buffalo
Publications
Child Speech and Language Assessment
- Speaker-Independent Place-of-Articulation Classification from Tongue Pose in Child Ultrasound using Deep Learning.
- Privacy-Enhancing Federated Child Speech Recognition using Machine Learning.
- Speech Annotation and Transcription Enhancer (SATE): An Automated System for Child Language Sample Analysis.
- Correct Then Detect: Zero-Shot FVMC Annotation for Child Language Sample Analysis.
- PULSE: A Principled Framework for Multimodal Speech Sound Disorder Assessment Using Ultrasound Tongue Imaging.
- A Mispronunciation-Based Voice-Omics Representation Framework for Screening Specific Language Impairments in Children.
Voice Biomarkers
- Event-Duration Distribution Modeling and Co-Adapted Ensembles for Speech-Based ADRD Screening.
- An Explainable COVID-19 Detection System based on Human Sounds.
- VoiceLens: A Multi-view Multi-class Disease Classification Model through Daily-Life Speech Data.
- mHealth Technologies towards Parkinson's Disease Detection and Monitoring in Daily Life: A Comprehensive Review.
- PDLens: Smartphone Knows Drug Effectiveness among Parkinson's via Daily-Life Activity Fusion.
- Robust Detection of Parkinson's Disease Using Harvested Smartphone Voice Data: A Telemedicine Approach.
- PDVocal: Towards Privacy-preserving Parkinson's Disease Detection using Non-speech Body Sounds.
- DeepVoice: A Voiceprint-Based Mobile Health Framework for Parkinson's Disease Identification.
Related Work
- Optimizing Deep Neural Networks for EEG-Based Speech Recognition: A Multimodal Approach to Assistive Communication.