Learning 3D Part Assembly from a Single Image

We introduce a novel problem, single-image-guided 3D part assembly, that assembles 3D shapes from parts given a complete set of part point cloud scans and a single 2D image depicting the object. The task is motivated by the robotic assembly setting and the estimated per-part poses serve as a vision-based initialization before robotic planning and control components.

PT2PC: Learning to Generate 3D Point Cloud Shapes from Part Tree Conditions

This paper investigates the novel problem of generating 3D shape point cloud geometry from a symbolic part tree representation. In order to learn such a conditional shape generation procedure in an end-to-end fashion, we propose a conditional GAN "part tree"-to-"point cloud" model (PT2PC) that disentangles the structural and geometric factors.

SAPIEN: A SimulAted Part-based Interactive ENvironment

We propose a realistic and physics-rich simulation environment hosting large-scale 3D articulated objects from ShapeNet and PartNet. Our PartNet-Mobility dataset contains 14,068 articulated parts with part motion information for 2,346 object models from 46 common indoor object categories. SAPIEN enables various robotic vision and interaction tasks that require detailed part-level understanding.

StructEdit: Learning Structural Shape Variations

We learn local shape edits (shape deltas) space that captures both discrete structural changes and continuous variations. Our approach is based on a conditional variational autoencoder (cVAE) for encoding and decoding shape deltas, conditioned on a source shape. The learned shape delta spaces support shape edit suggestions, shape analogy, and shape edit transfer, much better than StructureNet, on the PartNet dataset.

Learning to Group: A Bottom-Up Framework for 3D Part Discovery in Unseen Categories

We address the problem of learning to discover 3D parts for objects in unseen categories under the zero-shot learning setting. We propose a learning-based iterative grouping framework which learns a grouping policy to progressively merge small part proposals into bigger ones in a bottom-up fashion. On PartNet, we demonstrate that our method can transfer knowledge of parts learned from 3 training categories to 21 unseen categories.

StructureNet: Hierarchical Graph Networks for 3D Shape Generation

We introduce a hierarchical graph network for learning structure-aware shape generation which (i) can directly encode shape parts represented as such n-ary graphs; (ii) can be robustly trained on large and complex shape families such as PartNet; and (iii) can be used to generate a great diversity of realistic structured shape geometries with both both continuous geometric and discrete structural variations.

PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding

We propose a 3D object database with fine-grained and hierarchical part annotation, to assist segmentation and affordance research. We benchmark three part-level object understanding tasks: fine-grained semantic segmentation, hierarchical semantic segmentation and instance segmentation. We also propose a novel method for instance segmentation.

The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigation

We propose an indoor navigation dataset for visual navigation and deep reinforcement learning research. We show that the current mapless DRL-based method suffers from the target generalization and scene generalization issues. We propose methods to improve target generalization.

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

We propose novel neural networks to directly consume an unordered point cloud as input, without converting to other 3D representations such as voxel grids first. Rich theoretical and empirical analyses are provided.

Accelerating Random Kaczmarz Algorithm Based on Clustering Information

Using the property of randomly sampled data in high-dimensional space, we propose an accelerated algorithm based on clustering information to improve block Kaczmarz and Kaczmarz via Johnson-Lindenstrauss lemma. Additionally, we theoretically demonstrate convergence improvement on block Kaczmarz algorithm.

Visiting Graduate, Su Lab, University of California, San Diego
2019.7 - 9

Advisor: Prof. Hao Su

CS Ph.D. Candidate, Geometric Computation Group and Artificial Intelligence Lab, Stanford University
2016.9 - Present

Advisor: Prof. Leonidas J. Guibas

Exchange Research Student, Graphics and Vision Lab, Cornell University
2015.6 - 2015.12

Research Advisor: Prof. Kavita Bala
Academic Advisor: Prof. John E. Hopcroft

Bachelor of Engineering in Science, ACM Honored Class, Zhiyuan College, Shanghai Jiao Tong University
2012.9 - 2016.6

Research Advisor: Prof. Zhihua Zhang
Academic Advisor: Prof. Yong Yu

Research Intern, Facebook AI Research (Pittsburgh)
2020.6 - 2020.9

Mentor: Shubham Tulsiani and Prof. Abhinav Gupta

Research Intern, AI Lab, Autodesk Research
2018.6 - 2018.9

Mentor: Mike Haley

Research Intern, Creative Intelligence Lab, Adobe Research
2017.6 - 2017.9

Mentors: Haoxiang Li, Joon-Young Lee, Zhe Lin and Ersin Yumer