Digital Facial Analysis Correlates with Mood and Cognitive Outcomes with People with MS
Alyssa Nylander1, Kyra Henderson1, Kanishka Koshal1, Nikki Sisodia2, Jaeleene Wijangco1, Shane Poole1, Jim Rowson1, Cathra Halabi3, Ethan Brown4, Katherine Possin5, Adam Staffaroni1, Riley Bove4
1UCSF, 2University of California San Francisco, 3UCSF Department of Neurology, 4University of California, San Francisco, 5U of CA San Francisco, Neurology
Objective:
To correlate facial movements as assessed by digital tools with cognition and patient-reported mood in adults with multiple sclerosis (MS).
Background:
Facial expressivity is a dimension of emotional expression that has largely not been captured in individuals with MS. Digital facial analysis presents a novel, objective, non-invasive approach to capturing facial movements, which could augment the recognition of subtle indicators of emotional state or cognitive processing.
Design/Methods:
Adults with MS participating in an ongoing transdiagnostic digital phenotyping study completed: a validated tablet-based cognitive assessment (TabCat) and Hospital Anxiety and Depression Scales (HADS). Videos of their faces were recorded during a spontaneous language task and processed using OpenFace 2.0, an open-access digital tool pre-trained for facial landmark detection and facial action unit (AU) recognition. Instances of AU activation were averaged across frame number. Current analyses explored smiling (“cheek-raiser,” “lip corner-puller”) and eye expression (“brow-furrow,” “blink”) AUs. Correlations between AUs, cognitive testing, and PROs were assessed with univariate regression.
Results:
Among participants (n=98, 76% female, 80% relapsing-onset MS), mean age was 47 (SD 12.6), median EDSS 2 (range 0-6.5). HADS-Anxiety scores were ≥8 in 15.6%; HADS-Depression scores were ≥8 in 18.7%. HADS-Anxiety scores correlated with less brow-furrowing (r -0.39, p=0.03) and more smiling (r 0.37, p=0.03); smiling activation was higher in individuals with HADS-Anxiety scores ≥8 (difference 0.36, 95%CI 0.06-0.67, p=0.019). HADS-depression and AUS showed no associations. For t
he TabCat global cognition measure, Brain Health Assessment Composite (BHA-CS), 37.7% had z-score < -1.5. BHA-CS positively correlated with eye-crinkling (r
0.28, p=0.04) and smiling (r
0.30, p=0.03). The TabCat associative memory test, “
Favorites”, positively correlated with eye-crinkling (r
0.30, p=0.03) and smiling (r
0.33, p=0.015).
Conclusions:
Digitally identified components of facial expression correlate with metrics of mood and cognition in adults with MS, opening the door for novel modes of data capture in remote evaluation and monitoring.