Brain signals decoded to determine what a person sees
Technology has potential to help people affected by stroke or other conditions communicate
Edward Del RosarioAreas in the back of the brain become active when a person looks at an image. Researchers at Washington University School of Medicine in St. Louis have used light to decode brain signals and identify what image a person sees.
Some people are trapped within their own minds, able to think and feel but unable to express themselves because brain injury or disease has damaged their lines of communication with the outside world.
As a step toward helping people in such situations communicate, scientists at Washington University School of Medicine in St. Louis have demonstrated that they can use light to detect what is going on inside someone’s head. The researchers use LED light beamed from the outside of the head inward to detect activity in the area of the brain responsible for visual processing, and then decode brain signals to determine what the person sees. Accomplishing this feat required the development of neuroimaging tools and analysis techniques that move the field a step closer to solving the much more complex problem of decoding language.
The study, available online in the journal NeuroImage, demonstrates that high-density diffuse optical tomography (HD-DOT) — a noninvasive, wearable, light-based brain imaging technology — is sensitive and precise enough to be potentially useful in applications such as augmented communication that are not well suited to other imaging methods.
“MRI could be used for decoding, but it requires a scanner, and you can’t expect someone to go lie in a scanner every time they want to communicate,” said senior author Joseph P. Culver, PhD, the Sherwood Moore Professor of Radiology at Washington University’s Mallinckrodt Institute of Radiology. “With this optical method, users would be able to sit in a chair, put on a cap and potentially use this technology to communicate with people. We’re not quite there yet, but we’re making progress. What we’ve shown in this paper is that, using optical tomography, we can decode some brain signals with an accuracy above 90%, which is very promising.”
When the neuronal activity increases in any region of the brain, oxygenated blood rushes in to fuel the activity. HD-DOT uses light to detect the rush of blood. Participants wear a cap fitted with dozens of fibers that relay light from tiny LEDs to the head. After the light is transmitted through the head, detectors capture dynamic changes in the colors of the brain tissue as a result of changes in blood flow.
Culver, first author and graduate student Kalyan Tripathy, and colleagues set out to evaluate the potential of HD-DOT for decoding brain signals. They started with the visual system because it is one of the best understood brain functions. Neuroscientists long ago worked out a detailed map of the visual part of the brain by showing participants flashing checkerboard patterns on a screen and identifying the 3D units, known as voxels, in the brain that became active in response to each pattern. Decoding is the attempt to reverse the process: Detecting active voxels and then deducing which checkerboard pattern triggered that pattern of brain activity.
“We know what the participant is seeing, so we can verify how well our decoding matches up to reality,” said Culver, also a professor of physics, of electrical and systems engineering, and of biomedical engineering. “By going to something that was well validated, we could optimize the experimental design, push harder on the statistics of the decoding and obtain performance that is really very high.”
The researchers started simple. They recruited five participants for multiple five- to ten-minute runs in which the participants were shown a checkerboard pattern on either the left or the right side of the visual field for a few seconds at a time, interspersed with breaks during which no image was shown.
Using one run as the template, the researchers analyzed the data from another run to determine when the checkerboard was on which side of the screen. They repeated this analysis using different runs as the template and the test until they had analyzed all possible pairings.
The researchers were able to identify the correct position of the checkerboard — left, right or not visible at all — with 75% to 98% accuracy. While decoding was more successful when the same person was used for the template run and the test run, the patterns from one person could be used to decode the brain activity of another person.
Then, the researchers made the problem more complex. They showed participants a checkerboard wedge that rotated at 10 degrees a second. Three participants sat for six seven-minute runs on two separate days. Using the same template and test run strategy, the researchers were able to pinpoint the position of the wedge within 26 degrees.
The results are a first step toward the ultimate goal of facilitating communication for people who struggle to express themselves because of cerebral palsy, stroke or other conditions that result in locked-in syndrome, the researchers said.
“It sounds like a huge jump, from checkerboards to figuring out what words somebody is internally verbalizing to oneself,” Culver said. “But a lot of the principles are the same. The goal is to help people communicate, and what we’ve learned by decoding these visual stimuli is a solid step toward that goal.”
