Can Small and Synthetic Benchmarks Drive Modeling Innovation?

Nelson F. Liu, Tony Lee, Robin Jia, Percy Liang

Interactively Explore Our Plots

Choose a benchmark from the drop-down below to select a plot to view.

Y-axis benchmark:
MRQA NewsQA MRQA NaturalQuestions MRQA DROP MRQA HotpotQA ReCoRD CBT (CN) CBT (NE) CNN (100K Examples) LAMBADA SQuAD (1K Examples) SQuAD (10K Examples) SQuAD (20K Examples) SQUAD (40K Examples) SQuAD (60K Examples) LAMBADA (10K Examples) LAMBADA (20K Examples) LAMBADA (40K Examples) LAMBADA (60K Examples) LAMBADA (80K Examples) LAMBADA (100K Examples) Unsupervised QA (100K Examples) Synthetic Cloze Synthetic Cloze (with word dropout) Synthetic Fuzzy Pattern-Matching (GloVe) PCFG Fuzzy Pattern-Matching (GloVe) 3-gram Synthetic Fuzzy Pattern-Matching (GloVe) English Wikipedia Fuzzy Pattern-Matching (GloVe) QAMR SQuAD 1.1 Reduplicated (Max Length 1000) Simple WikiData Synthetic (-triple delimiter) WikidataSyntheticQA AWS UnsupervisedQA WikiData Synthetic (10K) (+comb, coref) WikiData Synthetic (10K) (+comb, -coref) WikiData Synthetic (10K) (+comb, -coref, -aliases) WikiData Synthetic (10K) (+comb, coref, no hyponyms) WikiData Synthetic (10K) (+comb, coref, no inverses) WikiData Synthetic (10K) (+comb, coref, simple questions only) WikiData Synthetic (10K) (+coref, -comb) WikiData Synthetic (10K) (-coref, -comb, simple questions only) WikiData Synthetic (10K) (-coref, -comb) bAbI QA #1 bAbI QA #2 bAbI QA #3 bAbI QA #4 bAbI QA #5 bAbI QA #11 bAbI QA #12 bAbI QA #13 bAbI QA #14 bAbI QA #15 bAbI QA #16 bAbI QA #1 (9K Examples) bAbI QA #2 (9K Examples) bAbI QA #3 (9K Examples) bAbI QA #4 (9K Examples) bAbI QA #5 (9K Examples) bAbI QA #11 (9K Examples) bAbI QA #12 (9K Examples) bAbI QA #13 (9K Examples) bAbI QA #14 (9K Examples) bAbI QA #15 (9K Examples) bAbI QA #16 (9K Examples) FuzzySyntheticQA PCFG FuzzySyntheticQA 3-gram LM FuzzySyntheticQA GPT-2 FuzzySyntheticQA English Wikipedia FuzzySyntheticQA Synthetic Fuzzy Pattern-Matching (Near synonyms, FastText) PCFG Fuzzy Pattern-Matching (Near synonyms, FastText) 3-gram Synthetic Fuzzy Pattern-Matching (Near synonyms, FastText) English Wikipedia Fuzzy Pattern-Matching (Near synonyms, FastText)