Media release

From: Springer Nature

Neuroscience: Developing a real-time thought-to-speech device for patients with paralysis 

A demonstration of online naturalistic streaming speech synthesis with synchronised text decoding from brain activity. Credit: Chang et al.
A demonstration of online streaming text-decoding and incremental text-to-speech synthesis from brain activity. Credit: Chang et al.

A new device capable of translating speech activity in the brain into spoken words in real-time is presented in a Nature Neuroscience paper. This technology could help people with speech loss to regain their ability to communicate more fluently in real time.

Current brain–computer interfaces for speech typically have a delay of a few seconds between the person silently attempting to say sentences and the computer’s verbal output, which prevents fluent and articulate communication. This can lead to miscommunication and frustration between the listener and speaker. A real-time system has the potential to restore the natural flow of conversation, which could improve the quality of life for patients who cannot speak.

Edward Chang, Gopala Anumanchipalli, and colleagues developed a silent brain–computer interface and implanted it into a 47-year-old woman with quadriplegia (paralysis of the limbs and torso), who could not speak or vocalize for 18 years after experiencing a stroke, as part of a clinical trial. The authors trained a deep learning neural network using the participant’s brain activity, which was recorded using electrodes implanted over her speech sensorimotor cortex, while she internally spoke complete sentences in her brain which contained 1,024 unique words. This model was then used to decode online speech in 80-millisecond increments, simultaneously with the participant’s vocal intent, and then produce audio mimicking the participant’s voice trained using a clip from her speaking before the injury. The brain–computer interface could also generalize to words the participant had not been exposed to during training, and the authors found that the device could operate continuously as opposed to in increments of a few seconds each.

Although further research is needed in more participants, the device could potentially help patients with speech paralysis to speak more naturally and seamlessly in real time and improve their quality of life, the authors suggest.