For the first time in history, a team of neuroengineers from Columbia University (USA) has created a system that translates thought into an intelligible and recognizable speech . By monitoring a person’s brain activity, this innovative technology can reconstruct the words an individual hears with unprecedented clarity.
This breakthrough, harnessing the power of speech synthesizers and artificial intelligence, could lead to new ways for machines to communicate directly with the brain. It also lays the foundation for helping people who cannot speak, such as those living with amyotrophic lateral sclerosis (ALS) or recovering from a stroke, to regain their ability to communicate with the outside world.
“Our voices help us connect with our friends, family and the world around us, so losing the power of the voice due to injury or illness is devastating,” said researcher Nima Mesgarani. “With today’s study, we have a potential way to restore that power. ” We have shown that, with the right technology, any listener can decipher and understand these people’s thoughts. “
The study, which has been published in the journal Scientific Reports , began with a computer algorithm capable of reconstructing what a person is hearing and transmitting it with a robotic voice like human speech.
“This is the same technology used by the Amazon Echo and Apple Siri to provide verbal answers to our questions,” Mesgarani explained.
Tests
The researchers then asked five epilepsy patients undergoing brain surgery to listen to someone utter sentences while the researchers looked for neural patterns in the patients’ brain activity. The scientists used those patterns to train their algorithm, which they called a “vocoder,” which translates brain waves into speech.
After this, it was time to test how well the already trained machine translated thoughts. So they asked a group of epilepsy patients to listen to someone recite the numbers zero to nine, while the team recorded their brain signals. The researchers made the recordings through the algorithm, then had a specially trained neural network analyze and “clean” the output.
Finally, the team asked 11 people to listen to the speech to test its intelligibility.
” We found that people could understand and repeat sounds about 75% of the time, which is far above and beyond any previous attempt ,” Mesgarani clarified. “Vocoder and the powerful neural networks represented the sounds that the patients had originally heard with surprising precision,” says the researcher.
Reading the mind
The current scope of the machine is limited, so the next research goal is to use phrases and sentences more complex than numbers and to do the same tests not only when a person listens, but also when they speak or imagine that they speak. The goal is to translate all thoughts into words.
“In this scenario, if a user wearing this implant thinks ‘I need a glass of water’, our system can identify the brain signals generated with that thought, and convert them into a sentence,” says Mesgarani. ” This would change everything. It would give anyone who has lost their ability to speak, a new opportunity to connect with the world around them,” he concludes.
Referencia: Towards reconstructing intelligible speech from the human auditory cortex. Hassan Akbari, Bahar Khalighinejad, Jose L. Herrero, Ashesh D. Mehta & Nima Mesgarani. Scientific Reports volume 9, Article number: 874 (2019) DOI: https://doi.org/10.1038/s41598-018-37359-z
