Where2Act: From Pixels to Actions for Articulated 3D Objects

One of the fundamental goals of visual perception is to allow agents to meaningfully interact with their environment. In this paper, we take a step towards that long-term goal -- we extract highly localized actionable information related to elementary actions such as pushing or pulling for articulated objects with movable parts. For example, given a drawer, our network predicts that applying a pulling force on the handle opens the drawer. We propose, discuss, and evaluate novel network architectures that given image and depth data, predict the set of actions possible at each pixel, and the regions over articulated parts that are likely to move under the force. We propose a learning-from-interaction framework with an online data sampling strategy that allows us to train the network in simulation (SAPIEN) and generalizes across categories. But more importantly, our learned models even transfer to real-world data.

Compositionally Generalizable 3D Structure Prediction

We tackle the task of 3D shape structure reconstruction from single images in a zero-shot setting, where we train our model on chairs only and show its generalizability to unseen novel categories (e.g. tables, cabinets) without any new training data. We bring in the concept of compositional generalizability and propose a novel framework that factorizes the 3D shape reconstruction problem into proper sub-problems, each of which is tackled by a carefully designed neural sub-module with generalizability guarantee. The intuition behind our formulation is that object parts (slates and cylindrical parts), their relationships (adjacency, equal-length, and parallelism) and shape substructures (T-junctions and a symmetric group of parts) are mostly shared across object categories, even though the object geometry may look very different.

Generative 3D Part Assembly via Dynamic Graph Learning

Analogous to buying an IKEA furniture, given a set of 3D part point clouds, we predict 6-Dof part poses to assemble a 3D shape. To tackle this problem, we propose an assembly-oriented dynamic graph learning framework that leverages an iterative graph neural network as a backbone. It explicitly conducts sequential part assembly refinements in a coarse-to-fine manner, exploits a pair of part relation reasoning module and part aggregation module for dynamically adjusting both part features and their relations in the part graph.

DSM-Net: Disentangled Structured Mesh Net for Controllable Generation of Fine Geometry

We introduce DSM-Net, a deep neural network that learns a disentangled structured mesh representation for 3D shapes, where two key aspects of shapes, geometry and structure, are encoded in a synergistic manner to ensure plausibility of the generated shapes, while also being disentangled as much as possible. This supports a range of novel shape generation applications with intuitive control, such as interpolation of structure (geometry) while keeping geometry (structure) unchanged. Our method not only supports controllable generation applications, but also produces high-quality synthesized shapes, outperforming state-of-the-art methods.

Rethinking Sampling in 3D Point Cloud Generative Adversarial Networks

We show that sampling-insensitive discriminators (e.g.PointNet-Max) produce shape point clouds with point clustering artifacts while sampling-oversensitive discriminators (e.g.PointNet++, DGCNN) fail to guide valid shape generation. We propose the concept of sampling spectrum to depict the different sampling sensitivities of discriminators. We point out that, though recent research has been focused on the generator design, the main bottleneck of point cloud GAN actually lies in the discriminator design.

Learning 3D Part Assembly from a Single Image

We introduce a novel problem, single-image-guided 3D part assembly, that assembles 3D shapes from parts given a complete set of part point cloud scans and a single 2D image depicting the object. The task is motivated by the robotic assembly setting and the estimated per-part poses serve as a vision-based initialization before robotic planning and control components.

PT2PC: Learning to Generate 3D Point Cloud Shapes from Part Tree Conditions

This paper investigates the novel problem of generating 3D shape point cloud geometry from a symbolic part tree representation. In order to learn such a conditional shape generation procedure in an end-to-end fashion, we propose a conditional GAN "part tree"-to-"point cloud" model (PT2PC) that disentangles the structural and geometric factors.

SAPIEN: A SimulAted Part-based Interactive ENvironment

We propose a realistic and physics-rich simulation environment hosting large-scale 3D articulated objects from ShapeNet and PartNet. Our PartNet-Mobility dataset contains 14,068 articulated parts with part motion information for 2,346 object models from 46 common indoor object categories. SAPIEN enables various robotic vision and interaction tasks that require detailed part-level understanding.

StructEdit: Learning Structural Shape Variations

We learn local shape edits (shape deltas) space that captures both discrete structural changes and continuous variations. Our approach is based on a conditional variational autoencoder (cVAE) for encoding and decoding shape deltas, conditioned on a source shape. The learned shape delta spaces support shape edit suggestions, shape analogy, and shape edit transfer, much better than StructureNet, on the PartNet dataset.

Learning to Group: A Bottom-Up Framework for 3D Part Discovery in Unseen Categories

We address the problem of learning to discover 3D parts for objects in unseen categories under the zero-shot learning setting. We propose a learning-based iterative grouping framework which learns a grouping policy to progressively merge small part proposals into bigger ones in a bottom-up fashion. On PartNet, we demonstrate that our method can transfer knowledge of parts learned from 3 training categories to 21 unseen categories.

StructureNet: Hierarchical Graph Networks for 3D Shape Generation

We introduce a hierarchical graph network for learning structure-aware shape generation which (i) can directly encode shape parts represented as such n-ary graphs; (ii) can be robustly trained on large and complex shape families such as PartNet; and (iii) can be used to generate a great diversity of realistic structured shape geometries with both both continuous geometric and discrete structural variations.

PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding

We propose a 3D object database with fine-grained and hierarchical part annotation, to assist segmentation and affordance research. We benchmark three part-level object understanding tasks: fine-grained semantic segmentation, hierarchical semantic segmentation and instance segmentation. We also propose a novel method for instance segmentation.

The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigation

We propose an indoor navigation dataset for visual navigation and deep reinforcement learning research. We show that the current mapless DRL-based method suffers from the target generalization and scene generalization issues. We propose methods to improve target generalization.

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

We propose novel neural networks to directly consume an unordered point cloud as input, without converting to other 3D representations such as voxel grids first. Rich theoretical and empirical analyses are provided.

Accelerating Random Kaczmarz Algorithm Based on Clustering Information

Using the property of randomly sampled data in high-dimensional space, we propose an accelerated algorithm based on clustering information to improve block Kaczmarz and Kaczmarz via Johnson-Lindenstrauss lemma. Additionally, we theoretically demonstrate convergence improvement on block Kaczmarz algorithm.

CS Ph.D. Candidate, Geometric Computation Group and Artificial Intelligence Lab, Stanford University
2016.9 - Present

Advisor: Prof. Leonidas J. Guibas

Visiting Graduate, Su Lab, University of California, San Diego
2019.7 - 9

Advisor: Prof. Hao Su

Exchange Research Student, Graphics and Vision Lab, Cornell University
2015.6 - 2015.12

Research Advisor: Prof. Kavita Bala
Academic Advisor: Prof. John E. Hopcroft

Bachelor of Engineering in Science, ACM Honored Class, Zhiyuan College, Shanghai Jiao Tong University
2012.9 - 2016.6

Research Advisor: Prof. Zhihua Zhang
Academic Advisor: Prof. Yong Yu

Research Intern, Facebook AI Research (Pittsburgh Remote from California)
2020.6 - 2020.9

Mentor: Shubham Tulsiani, Mustafa Mukadam and Prof. Abhinav Gupta

Research Intern, AI Lab, Autodesk Research
2018.6 - 2018.9

Mentor: Mike Haley

Research Intern, Creative Intelligence Lab, Adobe Research
2017.6 - 2017.9

Mentors: Haoxiang Li, Joon-Young Lee, Zhe Lin and Ersin Yumer