High-Tech Sign Language Could Replace the Mouse

G-speak's technology allows users to interact with computers using hand gestures.

May 31, 2006|Dawn C. Chmielewski | Times Staff Writer

John Underkoffler wants to build a better mouse.

Working in a downtown Los Angeles loft, the co-founder of G-speak is developing technology to replace the ubiquitous computer mouse with a more natural interface: human hands.

Riffs on Underkoffler's technology have featured prominently in movies and television shows. The 2002 sci-fi film "Minority Report," for example, included a dramatic scene in which Tom Cruise gestures with gloved hands to navigate and manipulate crime scene information on translucent screens.

That scene was the product of movie industry special effects. But it marked the birth of G-speak.

Underkoffler moved to Los Angeles to serve as a science and technology advisor on the Steven Spielberg movie. He came from the Massachusetts Institute of Technology Media Lab, where he was a doctoral candidate researching new ways for people to interact with computers.

"The film consultancy stuff in a way was a detour," Underkoffler said. "I'd been building stuff like G-speak before 'Minority Report,' and it was very natural to return to that after. Except that the movie also had the interesting effect of, in some sense, validating a specific form of those ideas. Audiences really responded to those sequences. So in a weird way we'd been able to use the film mechanism to prototype a technology."

In May 2005, Underkoffler and other former MIT colleagues pooled their own money and formed G-speak to find commercial applications for what's called gestural technology. The eight-person company has contracts with aerospace companies and federal agencies interested in the technology. It won't disclose revenue.

"It seems to us that manipulating things with our hands is a fundamental way to interact with the world," said Kwindla Hultman Kramer, G-speak's director of business development.

To demonstrate how the technology works, Underkoffler donned a pair of black gloves with reflective beads attached to the back of the hand, the thumb and two fingers. He drew a pair of heavy black curtains to block light from entering the loft and stepped onto the center of a makeshift stage, surrounded by a scaffolding holding eight infrared-sensitive cameras.

The cameras use light reflected off the BB-sized beads to calculate the position of Underkoffler's hands and interpret each of about two dozen core gestures as a computer command. He faced a 16-foot screen and held his hands like a kid holding imaginary twin six-shooters (index finger pointed, thumb raised) to tag objects on screen and depressed the thumb to grab the object -- in this case a letter of the alphabet.

Holding his hands vertically, thumbs touching, and gesturing to the left like a little league coach signaling a base runner to lead off first base causes the data to sweep to the left edge of the screen. Making an OK gesture to the screen with the thumb and index finger causes the text to form a circle.

Underkoffler said gestures were a more nuanced way to interact with images.

He displayed a panoramic scene of downtown shot from the roof of G-speak's building. He panned the landscape to the left with a simple sweeping gesture and zoomed in on a distant skyscraper by pushing his hand toward the screen.

The technology affords TiVo-like control over video. Underkoffler holds his hands upright in a gesture to "stop" and the video of traffic moving through a nearby intersection freezes. Making a chopping motion to the left, hands parallel to the floor, reverses the footage; gesturing to the right causes it to fast-forward.

Unlike a computer mouse, whose motion is limited to a flat two-dimensional plane, this system of gestures takes advantage of the position of the hands and fingers in space to manipulate images in three dimensions.

Kramer said the technology had attracted interest from Hollywood studios, which are considering ways to incorporate it into film editing, and the video game industry. It also has potential applications in other areas, including computer-aided design, medical imaging, air-traffic control, shipping logistics and homeland security.

Alex McDowell, production designer on "Minority Report" and several other films, said the G-speak technology has applications in the pre-visualization process of filmmaking, in which three-dimensional virtual environments are created on a computer to help the director evaluate where to place characters or the camera within a scene.

The director currently relies on an intermediary -- a character modeler or camera operator -- to maneuver the camera or manipulate placement of characters, using a keyboard and mouse, McDowell said. Gestural technology would give the director the ability to use his hands to manipulate objects within the computer-generated space or position the camera just-so.

Los Angeles Times Articles