Research progress
Brain interface pairs speech and gestures in a virtual avatar
A small early-feasibility study found that a brain-computer interface decoded attempted speech and gestures to control a virtual avatar.
AI-assisted summary and sourcing. Official and research sources are linked below; this article is informational and is not personal medical advice.
CareNews
Key takeaways
- The peer-reviewed study involved three participants in a single-center early-feasibility trial.
- The system used a surgically implanted electrode array and a wired connection to a virtual avatar.
What researchers tested
Speech and body language usually work together in conversation, but earlier brain-computer interfaces generally decoded one or the other. Researchers at the University of California, San Francisco tested whether signals from one implant could support both forms of communication at once.
Three adults with severe vocal-tract paralysis took part. A high-density electrocorticography array recorded activity from the motor cortex while participants attempted speech, facial movements, and upper-body gestures; software translated those signals into a personalized virtual avatar.
Sources for this section: Nature Neuroscience – Simultaneous speech and gesture decoding for multimodal communication in paralysis, NIH Research Matters – Device helps people with paralysis communicate
What the early study found
The researchers reported that the system decoded isolated movements of the upper body, mouth, and face, as well as attempted speech and gestures made together. Participants used the avatar to reproduce actions and to answer conversational prompts.
Training the decoding models with both isolated and simultaneous attempts improved performance across those communication contexts. This matters because natural expression often combines words with visible gestures rather than relying on either alone.
Sources for this section: Nature Neuroscience – Simultaneous speech and gesture decoding for multimodal communication in paralysis, NIH Research Matters – Device helps people with paralysis communicate
A promising, still early communication tool
The advance is a proof of concept for assistive technology that could eventually make digital communication more expressive for some people with paralysis. It does not restore a person’s own speech or movement, and the study did not test whether the approach improves long-term daily communication or quality of life.
This was a phase I, single-center early-feasibility study of three people, using an investigational implanted device and a wired setup. Larger studies, safety follow-up, and development work, including a possible wireless version, are needed before broad use can be considered.
Sources for this section: Nature Neuroscience – Simultaneous speech and gesture decoding for multimodal communication in paralysis, NIH Research Matters – Device helps people with paralysis communicate
Sources