Blind since birth, Marie-Laure Martin had always thought that candle flames were big balls of fire. The 39-year-old woman couldn’t see the flames themselves, but she could sense the candle’s aura of heat.
Last October, she saw a candle flame for the first time. She was stunned by how small it actually was and how it danced. There’s a second marvel here: She saw it all with her tongue.
The tongue, an organ of taste and touch, may seem like an unlikely substitute for the eyes. After all, it’s usually hidden inside the mouth, insensitive to light, and not connected to optic nerves. However, a growing body of research indicates that the tongue may in fact be the second-best place on the body for receiving visual information from the world and transmitting it to the brain.
Researchers at the University of WisconsinMadison are developing this tongue-stimulating system, which translates images detected by a camera into a pattern of electric pulses that trigger touch receptors. The scientists say that volunteers testing the prototype soon lose awareness of on-the-tongue sensations. They then perceive the stimulation as shapes and features in space. Their tongue becomes a surrogate eye.
Earlier research had used the skin as a route for images to reach the nervous system. That people can decode nerve pulses as visual information when they come from sources other than the eyes shows how adaptable, or plastic, the brain is, says Wisconsin neuroscientist and physician Paul Bach-y-Rita, one of the device’s inventors.
“You don’t see with the eyes. You see with the brain,” he contends. An image, once it reaches an eye’s retina, “becomes nerve pulses no different from those from the big toe,” he says. To see, people rely on the brain’s ability to interpret those signals correctly.
With that in mind, he and his colleagues propose that restoring sight is only one of the many trajectories for their research. Restoring stability to those with balance disorders is another. So is bestowing people with brand new senses, such as the capability to use heat to see in the dark.
Restoring lost vision
First things first, however, and for the Wisconsin scientists that means restoring lost vision. Swapping the sense of touch for sight is not a new idea. In the 1960s, Bach-y-Rita, his colleagues, and other scientists began developing and testing devices that enable the skin of blind people to pick up visual information.
For Bach-y-Rita, the experiments also provided insight into the brain’s plasticity. His more general goal has been to find out how well one sense can take the place of another.
Until the 1980s, “one of the axioms of neuroscience was that there was no plasticity in the adult central nervous system,” says Edward Taub of the University of Alabama in Birmingham. Today, the field has turned around in response to many studies, including Bach-y-Rita’s. Now, scientists view the brain as almost as malleable in old age as in youth, he adds.
The idea of tongue as eye evolved from the earlier skin-as-eye studies. Bach-y-Rita and his coworkers had been placing touch-stimulating arrays on areas of people’s skin, such as the back and the abdomen. The scientists used either electrodes or little buzzers to excite nerve endings of the skin in a pattern that corresponded to visual images.
They found that after receiving training, blind people using these systems could recognize shapes and track motion. Some subjects could perceive the motion of a ball rolling down an inclined plane and bat it as it rolled off the plane’s edge. Others could carry out an assembly-line task at an electronics plant. It required them to recognize glass tubes lacking solder and then to deposit some solder into those tubes.
These results impressed Bach-y-Rita and his colleagues enough to begin trying to apply their basic research toward designing aids for the blind, he says.
The researchers’ early systems had the look and feel of what they were–experiments. The buzzers were noisy, heavy, and power hungry. Although electrodes could stimulate nerves quietly and efficiently, high voltages and currents were necessary to drive signals through the skin. That sometimes led to uncomfortable shocks.
Because of these drawbacks, Bach-y-Rita began thinking about the tongue. “We brushed him off,” recalls coworker Kurt A. Kaczmarek, an electrical engineer and perception researcher, also at the University of Wisconsin. “He tends to be a bit ahead of his day.”
In time, however, Kaczmarek was convinced. “One day, I said ‘Okay, Paul. Let’s go up to the lab and try it.’ It turns out, it worked quite well,” he says.
Tongue stimulation, however, isn’t the only way to circumvent blindness. One competing approach, for example, is to implant microchips in the eyes or brain (SN: 4/12/97, p. 221). Another scheme, devised by a Dutch scientist, converts images to what he calls soundscapes, which are piped to a blind person’s ears.
Tongue stimulation
To Bach-y-Rita, his team’s switch from skin to tongue stimulation was crucial. “We now, for the first time, have the possibility of a really practical [touch-based] human-machine interface,” he declares. He and his coworkers founded the Madison-based company Wicab, to exploit the potential.
Kaczmarek points out the fledgling company may be in for some competition, since a German inventor already has been granted a U.S. patent for a tongue-vision system.
“Using the tongue for seeing is a whole new approach. . . . I think it has great promise,” says Michael D. Oberdorfer, program director for visual neuroscience at the National Eye Institute in Bethesda, Md. His office has been funding some of the Wisconsin group’s work.
The tongue is a better sensor than skin for several reasons, says Bach-y-Rita. For one, it’s coated in saliva–an electrically conductive fluid.
So, stimulation can be applied with much lower voltage and current than is required for the skin.
Also, the tongue is more densely populated with touch-sensitive nerves than most other parts of the body. That opens up the possibility that the tongue can convey higher-resolution data than the skin can.
What’s more, the tongue is ordinarily out of sight and out of the way. “With visual aids to the blind, there are cosmetic issues,” says Oberdorfer. “And you’d want something easy to wear that doesn’t interfere with everyday activities.”
Currently, the Wisconsin researchers’ tongue-display system begins with a camera about the size of a deck of cards. Cables connect it with a toaster-size control box. Extending from the box is another cable made of flat, flexible plastic laced with copper wires. It narrows at the end to form the flat, 12-by-12, gold-plated electrode array the size of a dessert fork. The person lays it like a lollipop on his or her tongue. Stimulation from electrodes produces sensations that subjects describe as tingling or bubbling.
The Wisconsin researchers say that the whole apparatus could shrink dramatically, becoming both hidden and easily portable. The camera would vanish into an eyeglass frame. From there, it would wirelessly transmit visual data to a dental retainer in the mouth that would house the signal-translating electronics. The retainer would also hold the electrode against the tongue.
The tongue display still has a long way to go in terms of performance, the researchers admit. In the July 13 Brain Research, Bach-y-Rita and his colleagues Eliana Sampaio and Stéphane Maris, both of the Université Louis Pasteur in Strasbourg, France, report results from the first clinical study of the tongue display.
After an initial, brief training period, 12 first-time users–6 sighted but blindfolded and 6 congenitally blind, including Marie-Laure Martin–tried to determine the orientation of the E’s of a standard Snellen eye chart. On average, they scored 20/860 in visual acuity.
The cutoff for legal blindness is 20/200 with corrected vision.
“It’s not normal sight,” comments Taub. “It’s like very dim shadows. But it’s remarkable. It’s a beginning.”
One obstacle to better vision with the device is the low resolution of its 144-electrode display. Engineers on the team say they expect to quadruple the array density in the next few years.
A more serious problem is the range of contrast that can be replicated on the tongue, Kaczmarek notes. In a typical image, the eye may simultaneously see lighted regions that are 1,000 times brighter than the dimmest ones. But the ratio of strongest to weakest tongue stimulation can only be about 3 to 1. “That’s one of the things we’re struggling with,” Kaczmarek says.
Visual sensations
Exactly how the tongue supplies the brain with images remains a focus of the Wisconsin team’s research. In his 1993 book, The Man Who Tasted Shapes (Putnam), Washington, D.C.based neurologist Richard E. Cytowic made much of how flavors stimulating the tongue of a friend and, later, an experimental subject, would elicit visual sensations. However, that type of involuntary and poorly understood sensory blending, which is known as synesthesia, probably goes beyond what’s needed to explain the operation of the tongue display, Bach-y-Rita says.
Instead, there’s plenty of evidence, he says, that even those brain regions devoted almost exclusively to a certain sense actually receive a variety of sensory signals. “We showed many years ago that even in the specialized eye region, auditory and tactile signals also arrive,” he notes.
Also, many studies over the past 40 years indicate that the brain is capable of massively reorganizing itself in response to loss or injury. When it comes to seeing via the sense of touch, reorganization may involve switching portions of the visual cortex to the processing of touch sensations, Bach-y-Rita says.
In that vein, the first clinical study of the tongue device showed that users got better with practice. Of the dozen subjects in the initial evaluation, two went on to receive an additional 9 hours each of training. When retested, they had doubled their visual acuity, scoring an average of 20/430.
The brain’s apparent ability to shunt data for one sense through the customary pathways of another may enable the Wisconsin researchers to apply their device beyond vision replacement. “It’s not just about vision,” says Mitchell E. Tyler, a biomedical engineer with the group. “That’s the obvious one, but it’s by no means the only game in town.”
The team began tests this summer of a modified system that’s intended to assist people who have lost their sense of balance because of injury, disease, or reactions to antibiotics. The unit gathers signals from accelerometers mounted on a person that indicate when he or she is tilting and in what direction. By stimulating the tongue with patterns representing the degree and direction of tilt, such a device may act as an artificial vestibular system. Then, the person might be able to correct bodily position and avoid falling, Tyler explains.
Although the main emphasis of the Wisconsin research has been rehabilitation, the group also foresees using its technology to aid people who don’t have sensory deficits.
Interest in enhancement of the senses has come primarily from the military. While Bach-y-Rita and his colleagues were using external skin as a receiver of light-derived images, the Defense Advanced Research Projects Agency in Arlington, Va., funded them to develop a sonar-based system to help Navy commandos orient themselves in pitch darkness. The prototype worked, Bach-y-Rita says.
Tyler proposes that ground soldiers could also receive data by means of infrared cameras or other sensors that would alert them, through the tongue, to the presence and positions of enemy troops or tanks. Civilian workers, such as firefighters, might also benefit from such interfaces.
That’s pure speculation right now. Martin’s bouts of vision; however, are much more than that. In a new film that aired on Canadian television in June, a smile spreads across Martin’s face as she gets her first glimpse of a candle flame.
The film, Touch: The Forgotten Sense, highlights some of the Wisconsin work. Its message is this: Touch works in a thousand ways, often without people even being aware of its roles.
By taking this sense into new arenas, such as the tongue display, Bach-y-Rita and his coworkers intend to extend touch’s repertoire even more.
