Robotic Perception via Contact

Humans are capable of performing many tasks relying just on the sense of touch.  To enable robots to do the same, we explore robotic perception via contact.  By sensing the object with the end-effector we are able to determine the object’s position and orientation with very high precision.  This allows us to perform fine manipulation tasks such as picking up a box and manipulating a door handle.


Taking this a step further, we want robots to utilize information from contacts made along entire length of manipulator links.


New Videos (June 2011):

These videos start with tactile probing procedure, which has been sped up x8.  The robot localizes the object based on the tactile data and begins the interaction scenario, which is shown in real time.

· Puma using a cash register

· Puma playing a guitar

· Puma toasting bagels


The same method can be used to localize and track moving objects based on Kinect data. In these experiments, we used the same object models and algorithms as for the tactile experiments.

· Original video sequence

· Range data provided by Kinect

· Object tracking results


Older Videos:

(Note: these videos have been sped up for better viewing)


· Puma robot picking up a box  Fully autonomous experiment.

· STAIR robot opening a door  In this experiment probing and manipulation of the door handle are autonomous, but motion of the mobile platform is controlled with a joystick.

· Estimating link contact  The contact is estimated using active sensing strategy and geometric constraints (see ICRA07 paper below).

· Estimating contact in a multi-contact scenario  The experiment shows that the same active sensing strategy can be used with multiple contacts.


Relevant publications:


Towards Dependable Robotic Perception. Anna Petrovskaya. PhD Thesis,  Computer Science Department, Stanford University, June 2011. [pdf]

Global Localization of Objects via Touch. Anna Petrovskaya and Oussama Khatib. IEEE Transactions on Robotics,  vol. 27(3), pp. 569--585, June 2011. [pdf] [IEEE version] [more]

Guaranteed Inference for Global State Estimation in Human Environments. Anna Petrovskaya, Sebastian Thrun, Daphne Koller, and Oussama Khatib. [MMW pdf] [DRHE pdf]


Touch Based Perception for Object Manipulation.  Anna Petrovskaya, Oussama Khatib, Sebastian Thrun, and Andrew Y. Ng.   Robotics Science and Systems Conference (RSS), Robot Manipulation Workshop, Atlanta, GA, 2007. [pdf]

Probabilistic Estimation of Whole Body Contacts for Multi-Contact Robot Control.  Anna Petrovskaya, Jaeheung Park, Oussama Khatib.  IEEE International Conference on Robotics and Automation (ICRA), Rome, Italy, 2007. [pdf]

Bayesian Estimation for Autonomous Object Manipulation Based on Tactile Sensors.  Anna Petrovskaya, Oussama Khatib, Sebastian Thrun, and Andrew Y. Ng. IEEE International Conference on Robotics and Automation (ICRA), Orlando, Florida, 2006. [pdf]  Also poster version from WiML06 [ppt]