Using an Intracortical Brain Computer Interface to Decode Contextual Modulation from Speech
Daniel Rubin1, Claire Nicolas1, John Simeral2, Sydney Cash1, Leigh Hochberg3
1Massachusetts General Hospital, 2Dept Veterans Affairs and Brown Univ., 3Dept Veterans Affairs, Massachusetts General Hospital, and Brown Univ.
Objective:
To understand how contextual elements of speech are encoded in human motor cortex, toward the development of an intracortical brain-computer interface (iBCI) for speech restoration.
Background:
Loss of communication is one of the most disabling symptoms of ALS and other neurologic conditions that cause paralysis. Recently, iBCIs have been used to decode the neural activity associated with intended phoneme production to restore communication. While these neuroprostheses accurately render users’ intended speech content, little is known regarding how cortex encodes contextual speech elements, such as volume, prosody, or inflection.
Design/Methods:
This research is conducted under an IDE from FDA and permission from the MGH IRB. A 39-year-old with quadriplegia (with preserved speech) from cervical spinal cord injury had two 96-channel microelectrode arrays placed chronically in left precentral gyrus. During recording sessions at his home, the participant read sentences by either whispering, speaking, or shouting (based on an onscreen cue) while neural activity was recorded. Neural signals were preprocessed and binned into 20ms blocks; power within the 250-5000Hz band and threshold-crossing spiking events were used for analyses.
Results:
The participant read 40 sentences at each of three volumes. Simultaneous audio recordings were used to determine the onset/offset of each spoken word. Using word-aligned neural data, we trained a multi-layer recurrent neural network to predict, in an offline analysis of held-out testing data, (1) the categorical volume class and (2) the RMS amplitude of each spoken word. Word volume category could be predicted with 77.7% accuracy from intracortical signals alone and the model accounted for 42.1% of the variance in recorded amplitude.
Conclusions:
In this preliminary study, an iBCI system was used to classify speech volume directly from motor cortical neural activity. Ongoing work focusses on expanding the range of contextual modulations decoded from cortex to restore fluent speech-based communication to people with severe dysarthria and anarthria.