Publications

Recent Highlights (2024-2025)

[ML] Fairness through Difference Awareness: Measuring Desired Group Discrimination in LLMs
Angelina Wang, Michelle Phan, Daniel E. Ho, Sanmi Koyejo
Association for Computational Linguistics (ACL), 2025 - Outstanding Paper Award
[url]

[BI] Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review
Suhana Bedi, Yutong Liu, Lucy Orr-Ewing, Dev Dash, Sanmi Koyejo, et al.
JAMA, 2025
[url]

[POL] Shaping AI's impact on billions of lives
Mariano-Florentino Cuéllar, Jeff Dean, Finale Doshi-Velez, John Hennessy, Andy Konwinski, Sanmi Koyejo, Pelonomi Moiloa, Emma Pierson, David Patterson
Communications of the ACM, 2025
[preprint]

[POL] Advancing science-and evidence-based AI policy
Rishi Bommasani, Sanjeev Arora, Jennifer Chayes, Yejin Choi, Mariano-Florentino Cuéllar, Li Fei-Fei, Daniel E Ho, Dan Jurafsky, Sanmi Koyejo, Hima Lakkaraju, et al.
Science, 2025
[url]

[ML] Machine Learning from Human Preferences
Sang Truong, Andreas Haupt, Sanmi Koyejo
Stanford Living Textbooks, 2025
[website]

Selected Recent Publications (2023-2025)

[ML] Are Emergent Abilities of Large Language Models a Mirage?
Rylan Schaeffer, Brando Miranda, Sanmi Koyejo
Neural Information Processing Systems (NeurIPS), 2023 - Outstanding Paper Award
[preprint]

[ML] How Do Large Language Monkeys Get Their Power (Laws)?
Rylan Schaeffer, Joshua Kazdan, John Hughes, Jordan Juravsky, Sara Price, Aengus Lynch, Erik Jones, Robert Kirk, Azalia Mirhoseini, Sanmi Koyejo
International Conference on Machine Learning (ICML), 2025
[preprint]

[ML] Reliable and Efficient Amortized Model-based Evaluation
Sang T. Truong, Yuheng Tu, Percy Liang, Bo Li, Sanmi Koyejo
International Conference on Machine Learning (ICML), 2025
[preprint]

[ML] Certified unlearning for neural networks
Anastasia Koloskova, Youssef Allouah, Animesh Jha, Rachid Guerraoui, Sanmi Koyejo
International Conference on Machine Learning (ICML), 2025
[preprint]

[ML] Collapse or thrive? perils and promises of synthetic data in a self-generating world
Joshua Kazdan, Rylan Schaeffer, Apratim Dey, Matthias Gerstgrasser, Rafael Rafailov, David L Donoho, Sanmi Koyejo
International Conference on Machine Learning (ICML), 2025

[ML] Putnam-axiom: A functional and static benchmark for measuring higher level mathematical reasoning
Aryan Gulati, Brando Miranda, Eric Chen, Emily Xia, Kai Fronsdal, Bruno de Moraes Dumont, Sanmi Koyejo
International Conference on Machine Learning (ICML), 2025

[ML] Rethinking machine unlearning for large language models
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, Kush R. Varshney, Mohit Bansal, Sanmi Koyejo, Yang Liu
Nature Machine Intelligence, 2025
[preprint]

[ML] Are Domain Generalization Benchmarks with Accuracy on the Line Misspecified?
Olawale Elijah Salaudeen, Nicole Chiou, Shiny Weng, Sanmi Koyejo
Transactions on Machine Learning Research, 2025
[preprint]

[ML] Scaling laws for downstream task performance of large language models
Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, Sanmi Koyejo
International Conference on Learning Representations (ICLR), 2025
[preprint]

[ML] DecodingTrust: A comprehensive assessment of trustworthiness in GPT models
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, et al.
Neural Information Processing Systems (NeurIPS), 2023 - Outstanding Paper Award
[preprint]

[ML] Transforming and combining rewards for aligning large language models
Zihao Wang, Chirag Nagpal, Jonathan Berant, Jacob Eisenstein, Alexander Nicholas D'Amour, Sanmi Koyejo, Victor Veitch
International Conference on Machine Learning (ICML), 2024
[preprint]

[ML] Causally inspired regularization enables domain general representations
Olawale Salaudeen, Sanmi Koyejo
International Conference on Artificial Intelligence and Statistics (AISTATS), 2024
[preprint]

Selected Preprints

[ML] Let's Measure Information Step-by-Step: LLM-Based Evaluation Beyond Vibes
Zachary Robertson, Sanmi Koyejo
Preprint, 2025
[preprint]

[ML] Measurement to Meaning: A Validity-Centered Framework for AI Evaluation
Olawale Salaudeen, Anka Reuel, Ahmed Ahmed, Suhana Bedi, Zachary Robertson, Sudharsan Sundar, Ben Domingue, Angelina Wang, Sanmi Koyejo
Preprint, 2025
[preprint]

Technical & Scientific Publications (2020-2023)

[ML, BI] Opportunistic Detection of Type 2 Diabetes using Deep Learning from Frontal Chest Radiographs
Ayis Pyrros, Stephen M. Borstelmann, Ramana Mantravadi, et al.
Nature Communications, 2023
[url]

[ML, BI] Toward fairness in artificial intelligence for medical image analysis: identification and mitigation of potential biases in the roadmap from data collection to model deployment
Karen Drukker, Weijie Chen, Judy Gichoya, Nicholas Gruszauskas, Jayashree Kalpathy-Cramer, Sanmi Koyejo, Kyle Myers, Rui C. Sá, Berkman Sahiner, Heather Whitney, Zi Zhang, Maryellen Giger
Journal of Medical Imaging, 2023
[url]

[ML] Cooperative inverse decision theory for uncertain preferences
Zachary Robertson, Hantao Zhang, Sanmi Koyejo
International Conference on Artificial Intelligence and Statistics (AISTATS), 2023
[link]

[ML] Adapting to latent subgroup shifts via concepts and proxies
Ibrahim Alabdulmohsin, Nicole Chiou, Alexander D'Amour, Arthur Gretton, Sanmi Koyejo, Matt Kusner, Stephen Pfohl, Olawale Salaudeen, Jessica Schrouff, Katherine Tsai
International Conference on Artificial Intelligence and Statistics (AISTATS), 2023
[preprint]

[ML, BI] Diagnosing failures of fairness transfer across distribution shift in real-world medical settings
Jessica Schrouff, Natalie Harris, Sanmi Koyejo, Ibrahim Alabdulmohsin, Eva Schnider, et al.
Neural Information Processing Systems (NeurIPS), 2022
[preprint]

[ML] Fair Wrapping for Black-box Predictions
Alexander Soen, Ibrahim Alabdulmohsin, Sanmi Koyejo, Yishay Mansour, Nyalleng Moorosi, Richard Nock, Ke Sun, Lexing Xie
Neural Information Processing Systems (NeurIPS), 2022
[preprint]

[ML] A Reduction to Binary Approach for Debiasing Multiclass Datasets
Ibrahim Alabdulmohsin, Jessica Schrouff, Sanmi Koyejo
Neural Information Processing Systems (NeurIPS), 2022
[preprint]

[ML] Zenops: A distributed learning system integrating communication efficiency and security
Cong Xie, Oluwasanmi Koyejo, Indranil Gupta
Algorithms, 2022
[url]

[ML] Fair Performance Metric Elicitation
Gaurush Hiranandani, Harikrishna Narasimhan, Oluwasanmi Koyejo
Neural Information Processing Systems (NeurIPS), 2020
[preprint]

[ML] Fairness with Overlapping Groups
Forest Yang, Moustapha Cisse, Sanmi Koyejo
Neural Information Processing Systems (NeurIPS), 2020
[preprint]

[ML] Zeno: Robust Asynchronous SGD with an Arbitrary Number of Byzantine Workers
Cong Xie, Sanmi Koyejo, Indranil Gupta
International Conference on Machine Learning (ICML), 2020
[preprint]

Foundational Work (2013-2019)

[ML] Performance metric elicitation from pairwise classifier comparisons
Gaurush Hiranandani, Shant Boodaghians, Ruta Mehta, Oluwasanmi Koyejo
International Conference on Artificial Intelligence and Statistics (AISTATS), 2019
[preprint]

[ML] Examples are not Enough, Learn to Criticize! Criticism for Interpretable Machine Learning
Been Kim, Rajiv Khanna, Oluwasanmi Koyejo
Advances in Neural Information Processing Systems (NIPS), 2016 (Oral)
[url]

[ML] Consistent multilabel classification
Oluwasanmi Koyejo, Nagarajan Natarajan, Pradeep Ravikumar, Inderjit Dhillon
Advances in Neural Information Processing Systems (NIPS), 2015
[url]

[ML] Consistent binary classification with generalized performance metrics
Oluwasanmi Koyejo, Nagarajan Natarajan, Pradeep Ravikumar, Inderjit Dhillon
Advances in Neural Information Processing Systems (NIPS), 2014 (Spotlight)
[url]

[ML, NS] Constrained Bayesian inference for low rank multitask learning
Oluwasanmi Koyejo, Joydeep Ghosh
Proceedings of the 29th conference on Uncertainty in artificial intelligence (UAI), 2013
Outstanding student paper award
[preprint]

Thesis

Constrained relative entropy minimization with applications to multitask learning
Oluwasanmi Koyejo
University of Texas at Austin, 2013
[url]