The development of an AI-based decoder that translates brain activity into coherent text, plots a groundbreaking advancement in non-invasive mind-reading technology. This innovation enables the reconstruction of speech with remarkable precision, leveraging fMRI scans to decode thoughts without the need for surgical implants.
The technology uses large language model artificial intelligence. OpenAI’s ChatGPT pushed a new way in. These large language models are able to represent, in numbers, the semantic meaning of speech, allowing the scientists to look at which patterns of neuronal activity corresponded to strings of words with a particular meaning rather than attempting to read out activity word by word.
“For a non-invasive method, this is a real leap forward compared to what’s been done before, which is typically single words or short sentences,” Research Dr. Huth said. Sometimes the decoder got the wrong end of the stick and it struggled with certain aspects of language, including pronouns. “It doesn’t know if it’s first-person or third-person, male or female,” said Huth. “Why it’s bad at this we don’t know.” The decoder was personalized. When the model was tested on another person the readout was unintelligible. It was also possible for participants on whom the decoder had been trained to thwart the system. By thinking of animals or quietly imagining another story, the system was confused.
Jerry Tang, a doctoral student at the University of Texas at Austin and a co-author, said: “We take very seriously the concerns that it could be used for bad purposes and have worked to avoid that. We want to make sure people only use these types of technologies when they want to and that it helps them.”
Professor Tim Behrens, a computational neuroscientist at the University of Oxford who was not involved in the work, described it as “technically extremely impressive” and said it opened up a host of experimental possibilities, including reading thoughts from someone dreaming or investigating how new ideas spring up from background brain activity. “These generative models are letting you see what’s in the brain at a new level,” he said. “It means you can really read out something deep from the fMRI.” Prof Shinji Nishimoto, of Osaka University, who has pioneered the reconstruction of visual images from brain activity, described the paper as a “significant advance”. “The research showed that the brain represents continuous language information during perception and imagination in a compatible way,” he said. “This is a non-trivial finding and can be a basis for the development of brain-computerinterfaces.
The system currently is not practical for use outside of the laboratory because of its reliance on the time need on an fMRI machine. Functional MRI, magnetic resonance imaging, relies on detecting small changes in the signals used to produce magnetic resonance images that are associated with neuronal activity in the brain. Functional magnetic resonance finds functions of the brain. Functions such as speech or memory have general locations in the brain, but the exact location can change slightly between individuals. The scan is safe. The future tech would resemble a shower cap. The MRI tech is in a capsuled machine that scans the individual, while laying down.
Researchers think this work could transfer to other, more portable brain-imaging systems, such as functional near-infrared spectroscopy (fNIRS). “fNIRS measures where there’s more or less blood flow in the brain at different points in time, which, it turns out, is exactly the same kind of signal that fMRI is measuring,” Huth said. “So, our exact kind of approach should translate to fNIRS,” although, he noted, the resolution with fNIRS would be lower.
The technology could aid communication for people who are unable to speak due to illness or injury, including stroke or paralysis. It could also enable seamless communication between humans and machines, such as the operation of a bionic arm or robot.
