PT2PC: Learning to Generate 3D Point Cloud Shapes from Part Tree Conditions

Kaichun Mo1     He Wang1     Xinchen Yan2     Leonidas J. Guibas1    
1 Stanford University     2 Uber ATG

European Conference on Computer Vision (ECCV) 2020

[ArXiv Preprint] [BibTex] [Code] [Slides]
Abstract

3D generative shape modeling is a fundamental research area in computer vision and interactive computer graphics, with many real-world applications. This paper investigates the novel problem of generating 3D shape point cloud geometry from a symbolic part tree representation. In order to learn such a conditional shape generation procedure in an end-to-end fashion, we propose a conditional GAN Part-Tree-to-Point-Cloud model (PT2PC) that disentangles the structural and geometric factors. The proposed model incorporates the part tree condition into the architecture design by passing messages top-down and bottom-up along the part tree hierarchy. Experimental results and user study demonstrate the strengths of our method in generating perceptually plausible and diverse 3D point clouds, given the part tree condition. We also propose a novel structural measure for evaluating if the generated shape point clouds satisfy the part tree conditions.


Figure 1. Task Definition. We formulate the problem of Part-Tree-to-Point-Cloud (PT2PC) as a conditional generation task which takes in a symbolic part tree as condition and generates multiple point clouds with shape variations satisfy the structure defined by the part tree.

Long 10-min Video Presentation
Short 1-min Video Presentation
Conditional-GAN Network Architecture

Figure 2. Our c-GAN PT2PC architecture. Our part-tree conditioned generator first extracts the subtree features by traversing the input symbolic part tree in a bottom-up fashion, and then recursively decodes part features in a top-down way until leaf nodes, where part point clouds are finally generated. Our part-tree conditioned geometry discriminator recursively consumes the generated part tree with leaf node geometry in a bottom-up fashion to generate the final score. The solid arrow indicates a network module while a dashed arrow means to copy the content.

Qualitative Results Comparing to Baselines

Figure 3. Qualitative Comparisons. We show two examples for each of the four categories: chair, table, cabinet and lamp. The leftmost two columns show the real examples illustrating the conditional part tree input. We show three random real examples unless there are only one or two in the dataset. For our method and B-Part we show both the generated part point clouds with colors and the down-sampled shape point clouds, to fairly compare with B-Whole that only produces shape point clouds. One can clearly observe that B-Whole produces holistically reasonable shape geometry but with unclear part structures. For B-Part, it fails severely for chair, table and cabinet examples that it does not assign clear roles for the parts and the generated part point clouds are overlaid with each other. Obviously, our method generates shapes with clearer part structures and boundaries. We also see a reasonable amount of generation diversity even for part trees with only one real data in PartNet, thanks to the knowledge sharing among similar part tree and sub-tree structures when training a unified and conditional network.

More Qualitative Results

Figure 4. Additional qualitative results. We show six more results for each of the four categories. For each block, the top row shows the real shapes and the bottom row shows our generated results.

Decoupling Structure and Geometry

Figure 5. Decoupling Structure and Geometry. Our approach enables disentanglement of geometry and structure factors in point cloud generation. Each row shows shape structural interpolation results while sharing similar shape geometry, and every column presents geometric interpolation results conditioned on the same part tree structure.

Mesh Generation Results

Figure 6. Mesh Generation Results. Since our method deforms a point cloud sampled from a unit cube mesh for each leaf-node part geometry, we naturally obtain the mesh generation results as well. The top rows show the generated shape point clouds and the bottom rows show the corresponding generated mesh results. Since the goal of this work is primarily for point cloud generation, we do not explicitly optimize for the mesh generation results. However, we observe reasonable mesh generation results learned by our method.

Acknowledgements

This research was supported by a Vannevar Bush Faculty Fellowship, grants from the Samsung GRO program and the Stanford SAIL Toyota Research Center, and gifts from Autodesk and Adobe.