Learning Compositional and Actionable Visual Representations for 3D Shapes

The representation of 3D objects, or 3D shapes, is one of the cornerstones of human-level 3D vision and robotics systems. This is notoriously challenging given that 3D shapes have extraordinarily diverse geometries, rich functional semantics, and complicated part structures. In the meantime, the complex nature of 3D tasks in vision, graphics and robotics is another challenge we are facing. To this end, this thesis explores to tackle the core problems of building compositional and actionable visual representations for diverse 3D shapes and designing scalable learning frameworks for various 3D tasks.

GIMO: Gaze-Informed Human Motion Prediction in Context

We propose a large-scale human motion dataset that delivers high-quality body pose sequences, scene scans, as well as ego-centric views with eye gaze that serves as a surrogate for inferring human intent. After adapting and benchmarking existing state-of-the-art methods, we introduce a novel network architecture that enables bidirectional communication between the gaze and motion branches for better ego-centric human motion predictions.

AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactions

Taking only passive observations as inputs, Where2Act ignores many hidden but important kinematic constraints (e.g., joint location and limits) and dynamic factors (e.g., joint friction and restitution), therefore losing significant accuracy for test cases with such uncertainties. In this paper, we propose a novel framework, named AdaAfford, that learns to perform very few test-time interactions for quickly adapting the affordance priors to more accurate instance-specific posteriors.

RoboAssembly: Learning Generalizable Furniture Assembly Policy in a Novel Multi-robot Contact-rich Simulation Environment

We develop a robotic assembly simulation environment for furniture assembly, and formulate the part assembly problem into a concrete reinforcement learning problem along with a learning pipeline to assemble a diverse set of chairs. Experiments show that when testing with unseen chairs, our approach achieves 74.5% success rate on our object-centric setting (without robots) and 50% success rate on our normal setting (with robots).

Compositionally Generalizable 3D Structure Prediction

We tackle the task of 3D shape structure reconstruction from single images in a zero-shot setting, where we train our model on chairs only and show its generalizability to unseen novel categories (e.g. tables, cabinets) without any new training data. We bring in the concept of compositional generalizability and propose a novel framework that factorizes the 3D shape reconstruction problem into proper sub-problems, each of which is tackled by a carefully designed neural sub-module with generalizability guarantee.

The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigation

We propose an indoor navigation dataset for visual navigation and deep reinforcement learning research. We show that the current mapless DRL-based method suffers from the target generalization and scene generalization issues. We propose methods to improve target generalization.

Fixing Malfunctional Objects With Learned Physical Simulation and Functional Prediction

This paper studies the problem of fixing malfunctional 3D objects. Given a malfunctional object, humans can perform mental simulations to reason about its functionality and figure out how to fix it. We propose FixIt, a dataset that contains around 5k poorly-designed 3D physical objects paired with choices to fix them. We present FixNet, a novel framework that seamlessly incorporates perception and physical dynamics. Specifically, FixNet consists of a perception module to extract the structured representation from the 3D point cloud, a physical dynamics prediction module to simulate the results of interactions on 3D objects, and a functionality prediction module to evaluate the functionality and choose the correct fix.

IFR-Explore: Learning Inter-object Functional Relationships in 3D Indoor Scenes

We take the first step in building AI system learning inter-object functional relationships in 3D indoor environments (e.g., a switch on the wall turns on or off the light, a remote control operates the TV). The key technical contributions are modeling prior knowledge by training over large-scale scenes and designing interactive policies for effectively exploring the training scenes and quickly adapting to novel test scenes.

Object Pursuit: Building a Space of Objects via Discriminative Weight Generation

We propose a framework to continuously learn object-centric representations for visual learning and understanding. Our method leverages interactions to effectively sample diverse variations of an object and the corresponding training signals while learning the object-centric representations. Throughout learning, objects are streamed one by one in random order with unknown identities, and are associated with latent codes that can synthesize discriminative weights for each object through a convolutional hypernetwork.

VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects

We propose object-centric actionable visual priors as a novel perception-interaction handshaking point that the perception system outputs more actionable guidance than kinematic structure estimation, by predicting dense geometry-aware, interaction-aware, and task-aware visual action affordance and trajectory proposals. We design an interaction-for-perception framework VAT-Mart to learn such actionable visual representations by simultaneously training a curiosity-driven reinforcement learning policy exploring diverse interaction trajectories and a perception module summarizing and generalizing the explored knowledge for pointwise predictions among diverse shapes.

DSG-Net: Learning Disentangled Structure and Geometry for 3D Shape Generation

We introduce DSG-Net, a deep neural network that learns a disentangled structured mesh representation for 3D shapes, where two key aspects of shapes, geometry and structure, are encoded in a synergistic manner to ensure plausibility of the generated shapes, while also being disentangled as much as possible. This supports a range of novel shape generation applications with intuitive control, such as interpolation of structure (geometry) while keeping geometry (structure) unchanged. Our method not only supports controllable generation applications, but also produces high-quality synthesized shapes.

O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learning

Contrary to the vast literature in modeling, perceiving, and understanding agent-object interaction in computer vision and robotics, very few past works have studied the task of object-object interaction, which also plays an important role in robotic manipulation and planning tasks. There is a rich space of object-object interaction scenarios in our daily life, such as placing an object on a messy tabletop, fitting an object inside a drawer, pushing an object using a tool, etc. In this paper, we propose a large-scale object-object affordance learning framework that requires no human annotation or demonstration.

Learning to Regrasp by Learning to Place

Regrasping is needed whenever a robot's current grasp pose fails to perform desired manipulation tasks. In this paper, we propose a system for robots to take partial point clouds of an object and the supporting environment as inputs and output a sequence of pick-and-place operations to transform an initial object grasp pose to the desired object grasp poses. The key technique includes a neural stable placement predictor and a regrasp graph based solution through leveraging and changing surrounding environment.

Where2Act: From Pixels to Actions for Articulated 3D Objects

One of the fundamental goals of visual perception is to allow agents to meaningfully interact with their environment. In this paper, we take a step towards that long-term goal -- we extract highly localized actionable information related to elementary actions such as pushing or pulling for articulated objects with movable parts. We propose, discuss, and evaluate novel network architectures that given image and depth data, predict the set of actions possible at each pixel, and the regions over articulated parts that are likely to move under the force. We propose a learning-from-interaction framework with an online data sampling strategy that allows us to train the network in simulation (SAPIEN) and generalizes across categories.

Rethinking Sampling in 3D Point Cloud Generative Adversarial Networks

We show that sampling-insensitive discriminators (e.g.PointNet-Max) produce shape point clouds with point clustering artifacts while sampling-oversensitive discriminators (e.g.PointNet++, DGCNN) fail to guide valid shape generation. We propose the concept of sampling spectrum to depict the different sampling sensitivities of discriminators. We point out that, though recent research has been focused on the generator design, the main bottleneck of point cloud GAN actually lies in the discriminator design.

Generative 3D Part Assembly via Dynamic Graph Learning

Analogous to buying an IKEA furniture, given a set of 3D part point clouds, we predict 6-Dof part poses to assemble a 3D shape. To tackle this problem, we propose an assembly-oriented dynamic graph learning framework that leverages an iterative graph neural network as a backbone. It explicitly conducts sequential part assembly refinements in a coarse-to-fine manner, exploits a pair of part relation reasoning module and part aggregation module for dynamically adjusting both part features and their relations in the part graph.

Learning 3D Part Assembly from a Single Image

We introduce a novel problem, single-image-guided 3D part assembly, that assembles 3D shapes from parts given a complete set of part point cloud scans and a single 2D image depicting the object. The task is motivated by the robotic assembly setting and the estimated per-part poses serve as a vision-based initialization before robotic planning and control components.

PT2PC: Learning to Generate 3D Point Cloud Shapes from Part Tree Conditions

This paper investigates the novel problem of generating 3D shape point cloud geometry from a symbolic part tree representation. In order to learn such a conditional shape generation procedure in an end-to-end fashion, we propose a conditional GAN "part tree"-to-"point cloud" model (PT2PC) that disentangles the structural and geometric factors.

SAPIEN: A SimulAted Part-based Interactive ENvironment

We propose a realistic and physics-rich simulation environment hosting large-scale 3D articulated objects from ShapeNet and PartNet. Our PartNet-Mobility dataset contains 14,068 articulated parts with part motion information for 2,346 object models from 46 common indoor object categories. SAPIEN enables various robotic vision and interaction tasks that require detailed part-level understanding.

StructEdit: Learning Structural Shape Variations

We learn local shape edits (shape deltas) space that captures both discrete structural changes and continuous variations. Our approach is based on a conditional variational autoencoder (cVAE) for encoding and decoding shape deltas, conditioned on a source shape. The learned shape delta spaces support shape edit suggestions, shape analogy, and shape edit transfer, much better than StructureNet, on the PartNet dataset.

Learning to Group: A Bottom-Up Framework for 3D Part Discovery in Unseen Categories

We address the problem of learning to discover 3D parts for objects in unseen categories under the zero-shot learning setting. We propose a learning-based iterative grouping framework which learns a grouping policy to progressively merge small part proposals into bigger ones in a bottom-up fashion. On PartNet, we demonstrate that our method can transfer knowledge of parts learned from 3 training categories to 21 unseen categories.

StructureNet: Hierarchical Graph Networks for 3D Shape Generation

We introduce a hierarchical graph network for learning structure-aware shape generation which (i) can directly encode shape parts represented as such n-ary graphs; (ii) can be robustly trained on large and complex shape families such as PartNet; and (iii) can be used to generate a great diversity of realistic structured shape geometries with both both continuous geometric and discrete structural variations.

PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding

We propose a 3D object database with fine-grained and hierarchical part annotation, to assist segmentation and affordance research. We benchmark three part-level object understanding tasks: fine-grained semantic segmentation, hierarchical semantic segmentation and instance segmentation. We also propose a novel method for instance segmentation.

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

We propose novel neural networks to directly consume an unordered point cloud as input, without converting to other 3D representations such as voxel grids first. Rich theoretical and empirical analyses are provided.

Accelerating Random Kaczmarz Algorithm Based on Clustering Information

Using the property of randomly sampled data in high-dimensional space, we propose an accelerated algorithm based on clustering information to improve block Kaczmarz and Kaczmarz via Johnson-Lindenstrauss lemma. Additionally, we theoretically demonstrate convergence improvement on block Kaczmarz algorithm.

CS Ph.D. Candidate, Geometric Computation Group and Artificial Intelligence Lab, Stanford University
2016.9 - 2022.6

Advisor: Prof. Leonidas J. Guibas

Visiting Graduate, Su Lab, University of California, San Diego
2019.7 - 9

Advisor: Prof. Hao Su

Exchange Research Student, Graphics and Vision Lab, Cornell University
2015.6 - 2015.12

Research Advisor: Prof. Kavita Bala
Academic Advisor: Prof. John E. Hopcroft

Bachelor of Engineering in Science, ACM Honored Class, Zhiyuan College, Shanghai Jiao Tong University
2012.9 - 2016.6

Research Advisor: Prof. Zhihua Zhang
Academic Advisor: Prof. Yong Yu

Research Intern, Facebook AI Research (Pittsburgh Remote from California)
2020.6 - 2020.9

Mentor: Shubham Tulsiani, Mustafa Mukadam and Prof. Abhinav Gupta

Research Intern, AI Lab, Autodesk Research
2018.6 - 2018.9

Mentor: Mike Haley

Research Intern, Creative Intelligence Lab, Adobe Research
2017.6 - 2017.9

Mentors: Haoxiang Li, Joon-Young Lee, Zhe Lin and Ersin Yumer