Collaborative Research: Exploiting Voice Assistant Systems For Early Detection Of Cognitive Decline

National Institute On Aging (NIA), R01AG067416

9/30/2019 – 5/31/2024

Xiaohui Liang, PhD (University of Massachusetts, Boston)

John A. Batsis, MD (D-H Geriatrician, Associate Professor of Medicine, Geisel School of Medicine at Dartmouth); Robert Roth, PhD (D-H Neuropsychologist, Associate Professor of Psychiatry at Geisel); Brian MacWhinney, PhD (Carnegie Mellon University); and David Kotz, PhD (Dartmouth College)

Early detection of the cognitive decline involved in Alzheimer’s Disease and Related Dementias (ADRD) in older adults living alone is essential for developing, planning, and initiating interventions and support systems to improve patients’ everyday function and quality of life. Conventional, clinic-based methods for early diagnosis are expensive, time-consuming, and impractical for large-scale screening. This project aims to develop a low-cost, passive, and practical home-based assessment method using Voice Assistant Systems (VAS) for early detection of cognitive decline, including a set of novel data mining techniques for sparse time-series speech. The project has three specific aims: 1. Using a recurrent neural network (RNN) and a softmax regression model, we will develop a transfer learning technique to investigate the link between the speech from in-lab VAS tasks and cognitive decline. The Pitt corpus from the DementiaBank database will be used to optimize the RNN parameters and thereby overcome the limited data problem of VAS. The softmax regression model will allow us to align the feature distributions from the previous speech data and in-lab VAS speech. 2. We will develop a novel “many-to-difference” prediction model with a symmetric RNN structure to predict the cognitive difference at two ends of a time period from the sparse time-series data. The proposed model is different from previous ones as the learning focus is shifted from the short-term pattern differences across users to the pattern difference over time for an individual user. The proposed model accommodates well for the highly dynamic nature of the inputs and maximally removes individual characteristics from the prediction result. To analyze the sparse time-series speech, a new data sampling technique will be used to address the imbalanced data problem, and a data quality metric will be developed for the proposed model. 3. The team will conduct an 18-month in-lab evaluation and a 28-month in-home evaluation with a focus on whether the VAS tasks and features from the in-lab evaluation and the repetition features of the in-home VAS data can measure and predict cognitive decline in the in-home participants over time. The proposed methods will be integrated into an interactive system to enable efficient communication on cognitive decline among patients, caregivers, and clinicians. If successful, the outcomes of this project will provide an opportunity to provide supportive evidence to clinicians for the early detection of cognitive impairment outside of a clinic-based setting. RELEVANCE (See instructions): This project aims to develop a low-cost, passive, and practical cognitive assessment method using Voice Assistant Systems (VAS) for early detection of cognitive decline. If successful, the proposed system may be widely disseminated for the early diagnosis of cognitive impairment to complement existing diagnostic modalities that could ultimately enable long-term patient and caregiver planning to maintain individual’s independence at home.

The proposed research will explore passively collected sparse time-series speech using transfer learning and deep learning techniques. If successful, the proposed techniques will make an immediate and significant impact on mitigating the dementia problem in the older adult community. The proposed research activities will trigger additional data mining research on exploiting sparse time-series data for monitoring health conditions, such as stress and depression within the older adult community.