Suraj Nair

I work at Physical Intelligence building brains for robots.

I completed my Ph.D. in Computer Science from the Stanford AI Lab, where I was co-advised by Professors Chelsea Finn and Silvio Savarese. My Ph.D. thesis was on Scaling Deep Robotic Learning to Broad Real-World Data. Before that, I completed my Bachelors in Computer Science at the California Institute of Technology (Caltech), where I worked with Yisong Yue on multi-agent reinforcement learning. I have also spent time at the Toyota Research Institute, Facebook AI Research, Google Brain, and GE.

Email  /  CV  /  Research Statement (May 2022)  /  Google Scholar  /  Twitter  /  LinkedIn

The goal of my research is to enable robots that can operate in unstructured, real-world environments. Towards this goal I study how robots can generalize effectively across tasks, objects, and environments by learning from large datasets. Specifically, my research focuses on methods for:

• Scalably [1, 2] and safely [3] collecting real world robot datasets.
• Self-supervised learning of visual models from offline data [1,2,3] and using them for robotic manipulation [4,5,6,7].
• Leveraging human video datasets [1,3] and natural language annotations [2,3] to enable better robot learning.

We present Voltron, a multi-modal foundation model for robotics trained on human videos and language to produce reusable representations and rewards. We train a single model with many downstream capabilities from features for control to expression grounding and reward/intent inference.

Often, robot data isn't shared across projects. We present a new way that past project data can be used to improve downstream learning. We use a learned model select relevant data from a large dataset of robot interactions, which augments a small set of task demonstrations for use in a behavior cloning algorithm for more efficient learning.

We pre-train a generalizable visual representation on diverse human videos and language, and show it enables far more efficient learning across a wide range of robotic manipulation tasks.

We propose a method to enable robots to tackle challenging visually occluded manipulation tasks (like extracting keys from a bag), via end-to-end interactive imitation learning from vision and sound. .

We learn language-conditioned visuomotor skills on real robots from entirely offline, pre-collected datasets and crowdsourced language annotation.

EMBR is a model-based RL algorithm that learns visuomotor skills and their groundings, which can then be sequenced with symbolic planners to complete long-horizon, multi-stage manipulation tasks on real robots.

We propose a variational video prediction model that is capable of severe overfitting on common video prediction benchmarks while having similar parameter count as the current SOTA models.

We propose a technique for learning multi-task reward functions from a small amount of robot data and large amounts of in-the-wild human videos. By leveraging diverse human data, the learned reward function is able to generalize to new environments and tasks.

We propose a technique for video prediction which trains a hierarchy of action-conditioned VAEs in a greedy fashion, enabling efficient training of large video prediction models.

We propose a method for offline model-based RL which learns a video prediction model and a Q function based distance metric, and uses them to accomplish visually specified goals.

We propose a framework for leveraging weak human superivision to enable better robotic exploration. Using just a few minutes of human supervision, the robot collects high quality data while unsupervised, providing better data for downstream offline RL.

An algorithm for safe reinforcement learning which utilizes a set of offline data to learn about constraints before policy learning and a pair of policies which seperate the often conflicting objectives of task directed exploration and constraint satisfaction to learn contact rich and visuomotor control tasks.

We explore learning visual dynamics models which are conditioned on goals, and learn to model only goal relevant quantities.

We study how we can learn long horizon vision-based tasks in self-supervised settings. Our approach, hierarchical visual foresight, can optimize for a sequence of subgoals that break down the task into easy to complete subsegments.

We propose a technique that uses time-reversal to learn goals and provide a high level plan to reach them. In particular, our approach explores outward from a set of goal states, "unsolving" a task, which then enables solving the task from new initializations at test time.

We explore how to effectively predict causal graphs from a small set of visual observations, and how to encorporate the learned graphs into downstream goal conditioned policy learning.

We collect a dataset of robotic experience across 4 institutions and 7 robots, and demonstrate that robot learning algorithms leveraging this data can adapt to new environments faster than training from scratch.

NTG learns to produce a task graph from a single video demonstration of an unseen task, and leverages it for one-shot imitation learning.

Neural Task Programming (NTP) is a meta-learning framework that learns to generate robot-executable neural programs from task demonstration video.

Efficient prediction of real local earthquake signals from impulsive signals for earthquake early warning (EEW) alerts.

Reconstruct 3D voxel representations of a scene with object labels from RGB images captured from a drone, and use it for exporatory motion planning