These robots can follow how-to diagrams

Robots imbued with a certain kind of common sense may soon be able to follow instructional diagrams to build things.

When studying pictures for assembling IKEA furniture or LEGO villages, humans are naturally good at inferring how to get from A to B. Robots, on the other hand, normally have to be painstakingly programmed with exact instructions for how to move. “Even when you try to teach robots by demonstration, they’re just repeating the exact same motions you show them, not the concept underlying them,” says Dileep George, an artificial intelligence and neuroscience researcher at the San Francisco company Vicarious AI.

George and colleagues have now designed a robot operating system that can understand the basic ideas conveyed in schematic instructions and translate those ideas into action. These common sense robots, described online January 16 in Science Robotics, could work on a wider variety of tasks under different conditions than machines restricted to explicitly coded instructions or physical demonstrations.

The new robotic system learned more than 500 general concepts, such as “stack green objects on the right” and “arrange objects in a circle,” by studying before and after images for each type of action. When given a new set of instructions with a before-and-after diagram, the fully trained system considers all the concepts it has learned, and chooses and executes the maneuvers that will help it reach its goal.

George’s team tested the operating system in two gripper arm robots that moved objects across a tabletop. The robots examined picture instructions and then completed the tasks, such as separating lemons from limes and arranging different colored cans in a row. The machines also worked well when researchers changed conditions like the type of objects to be moved or the tabletop color.