Visual knowledge bases such as Visual Genome power numerous applications in computer vision, like visual question answering and captioning, but suffer from sparse, incomplete relationships. All scene graph models to date are limited to training on a small set of visual relationships that have thousands of training labels each. Hiring humans annotators is expensive, and using textual knowledge base completion methods are incompatible with visual data. In this paper, we introduce a semi-supervised method that assigns probabilistic relationship labels to a large number of unlabeled images from a few labeled examples. We analyze visual relationships and suggest two types of image-agnostic features that are used to generate noisy heuristics, whose outputs are aggregated together using a factor graph-based generative model. With as few as 10 labeled examples of a relationship, the generative model creates enough training data to train any existing state-of-the-art scene graph model. We demonstrate that our method for generating training data outperforms other baseline approaches including a transfer learning approach pretrained on a different set of relationships by 8.27 recall@100 points. We further analyze visual relationships and define a complexity metric that serves as an indicator (R^2 = 0.787) for conditions under which our method succeeds over transfer learning, the de-facto approach for training with limited labeled examples.

ICCV 2019


Download the paper here.


Download the Visual Genome dataset here.


Download the code here.


  title={Scene Graph Prediction with Limited Labels},
  author={Chen, Vincent and Varma, Paroma and Krishna, Ranjay and Bernstein, Michael and Re, Christopher and Fei-Fei, Li},
  booktitle={International Conference on Computer Vision},

Blog post coming soon.