New Robotic Telekinesis System Translates Actions Via Camera

Controllable robots hold a promising future for us, but today’s current models can be tricky to control for untrained users. They often involve complex electronics to manipulate, but that is about to change with the system started by a group of researchers at Carnegie Mellon University – robotic telekinesis.

Besides the ability to control the robot remotely, another exciting news about this robotic telekinesis system is its ability to translate the user’s action via an RGB camera. The robotic hand and arm will replicate the action by simply demonstrating a specific action. This makes it different from the previous inventions that commonly utilize complicated systems to control robots, like remote controllers and other devices.

“Prior works in this area rely either on gloves, motion markers or a calibrated multi-camera setup,” Deepak Pathak, one of the researchers of the team told TechXplore. “Instead, our system works using a single uncalibrated camera. Since no calibration is needed, the user can be standing anywhere and still successfully teleoperate the robot.”

According to Pathak, what even makes the robot’s system remarkable is the fact that it was entirely trained with just a series of YouTube videos showing individuals performing arm actions. Pathak said it can analyze simple two-dimensional images, making it an incredibly straightforward tech for anyone – even an unpracticed person. He said that the team had to focus on computer vision and 3D human pose estimation.

“The diversity of huge passive video data helps it work across untrained users, tasks, and objects,” Pathak stated. “Our system offers a low-cost and natural way to teach robots via demonstrations, as opposed to kinesthetically holding the robot or wearing a glove or motion capture suit… Hopefully, our recent work is just one of many future directions in which internet videos act as a rich source of supervision for robotic control, in addition to robotic vision.”

With all of these things, the researchers believe that the tech can be useful in everyday situations, especially to tasks that can be dangerous for a human’s presence. 

“Robotic telekinesis and similar technologies will enable robot teaching in a larger variety of settings, including in households, where they will be expected to perform everyday tasks.” Pathak said, “Using only a single uncalibrated camera, our system can in theory be controlled from anywhere in the world, thus it makes robot teaching more accessible to anyone. We are now collecting large-scale data using our robotic telekinesis system to teach the robot to act and adapt autonomously in the real world.”