Sang Michael Xie

I am a Computer Science Ph.D. student studying Machine Learning at Stanford University, advised by Percy Liang and Tengyu Ma.
My research focuses on data-centric methods for language models and understanding pretraining and adaptation to downstream tasks, including emergent behavior such as in-context learning. I have also worked on pretraining and self-training methods for robust machine learning.

Previously, I was a FY2019 NDSEG Fellow and a Student Researcher at Google Brain, working with Adams Wei Yu, Hieu Pham, and Quoc Le. I received both my B.S. with departmental honors and M.S. in Computer Science from Stanford in 2017, where I am grateful to have worked with Stefano Ermon on the first deep learning and transfer learning methods for sustainability, particularly in poverty mapping using satellite imagery. My work has been recognized in Scientific American's 10 World Changing Ideas, publication in flagship venues such as Science, and covered by media outlets including the New York Times, The Washington Post, Reuters, BBC News, IEEE Spectrum, and The Verge.

[Music] Email: xie AT cs.stanford.edu

Selected Publications

DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining [Paper] [Code] [Blog]
Advances in Neural Information Processing Systems (NeurIPS), 2023. Spotlight

BayLearn, 2023. Oral Presentation

Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc Le, Tengyu Ma, Adams Wei Yu
Data Selection for Language Models via Importance Resampling [Paper] [Data and Code]
Advances in Neural Information Processing Systems (NeurIPS), 2023.

Sang Michael Xie, Shibani Santurkar, Tengyu Ma, Percy Liang
An Explanation of In-context Learning as Implicit Bayesian Inference [Paper] [Code] [Video] [Blog]
International Conference on Learning Representations (ICLR), 2022.

Sang Michael Xie, Aditi Raghunathan, Percy Liang, Tengyu Ma
Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation [Paper]
International Conference on Machine Learning (ICML), 2022. Long talk

Kendrick Shen*, Robbie Jones*, Ananya Kumar*, Sang Michael Xie*, Jeff Z. HaoChen, Tengyu Ma, Percy Liang
In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness [Paper] [Code]

International Conference on Learning Representations (ICLR), 2021.

Sang Michael Xie*, Ananya Kumar*, Robbie Jones*, Fereshte Khani, Tengyu Ma, Percy Liang
Understanding and Mitigating the Tradeoff Between Robustness and Accuracy [Paper][Video]

International Conference on Machine Learning (ICML), 2020.

Aditi Raghunathan*, Sang Michael Xie*, Fanny Yang, John C. Duchi, Percy Liang
Combining Satellite Imagery and Machine Learning to Predict Poverty. [Video, Maps, Media, and Links] [Paper] [Code] [Top of Mind radio show segment]

Science, 2016.

Neal Jean*, Marshall Burke*, Michael Xie, William Davis, David Lobell, Stefano Ermon

Papers

2024

A Survey on Data Selection for Language Models [Paper] [Survey Paper List]
arXiv preprint, 2024.

Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, William Yang Wang
Connect Later: Improving Fine-tuning for Robustness with Targeted Augmentations [Paper] [Code]
arXiv preprint, 2024.

Helen Qu, Sang Michael Xie

2023

DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining [Paper] [Code] [Blog]
Advances in Neural Information Processing Systems (NeurIPS), 2023. Spotlight

BayLearn, 2023. Oral Presentation

Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc Le, Tengyu Ma, Adams Wei Yu
Data Selection for Language Models via Importance Resampling [Paper] [Data and Code]
Advances in Neural Information Processing Systems (NeurIPS), 2023.

Sang Michael Xie, Shibani Santurkar, Tengyu Ma, Percy Liang
Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models [Paper]
International Conference on Machine Learning (ICML), 2023. Oral

Hong Liu, Sang Michael Xie, Zhiyuan Li, Tengyu Ma
Reward Design with Language Models [Paper] [Code]
International Conference on Learning Representations (ICLR), 2023.

Minae Kwon, Sang Michael Xie, Kalesha Bullard, Dorsa Sadigh

2022

An Explanation of In-context Learning as Implicit Bayesian Inference [Paper] [Code] [Video] [Blog]
International Conference on Learning Representations (ICLR), 2022.

Sang Michael Xie, Aditi Raghunathan, Percy Liang, Tengyu Ma
Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation [Paper]
International Conference on Machine Learning (ICML), 2022. Long talk

Kendrick Shen*, Robbie Jones*, Ananya Kumar*, Sang Michael Xie*, Jeff Z. HaoChen, Tengyu Ma, Percy Liang
Extending the WILDS benchmark for Unsupervised Adaptation [Paper] [Code]
International Conference on Learning Representations (ICLR), 2022. Oral

Shiori Sagawa*, Pang Wei Koh*, Tony Lee*, Irena Gao*, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, Sara Beery, Etienne David, Ian Stavness, Wei Guo, Jure Leskovec, Kate Saenko, Tatsunori Hashimoto, Sergey Levine, Chelsea Finn, Percy Liang
Holistic Evaluation of Language Models [Paper]
Transactions on Machine Learning Research (TMLR), 2022.

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D Manning, Christopher Ré, Diana Acosta-Navas, Drew A Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, Yuta Koreeda

2021

Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning [Paper][Code]
Advances in Neural Information Processing Systems (NeurIPS), 2021. Spotlight

Colin Wei, Sang Michael Xie, Tengyu Ma
Composed Fine-Tuning: Freezing Pre-Trained Denoising Autoencoders for Improved Generalization [Paper] [Code]

International Conference on Machine Learning (ICML), 2021. Long talk

Sang Michael Xie, Tengyu Ma, Percy Liang
In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness [Paper] [Code]

International Conference on Learning Representations (ICLR), 2021.

Sang Michael Xie*, Ananya Kumar*, Robbie Jones*, Fereshte Khani, Tengyu Ma, Percy Liang
WILDS: A Benchmark of in-the-Wild Distribution Shifts [Paper] [Website]

International Conference on Machine Learning (ICML), 2021. Long talk

Pang Wei Koh*, Shiori Sagawa*, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard L. Phillips, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, Percy Liang
Ensembles and Cocktails: Robust Finetuning for Natural Language Generation [Paper]
NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications , 2021

John Hewitt*, Xiang Lisa Li*, Sang Michael Xie*, Ben Newman, Percy Liang
No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasets [Paper]

ICML Workshop on Uncertainty & Robustness in Deep Learning, 2021.

Fahim Tajwar, Sang Michael Xie, Ananya Kumar, Percy Liang
On the Opportunities and Risks of Foundation Models [Paper]

arXiv, 2021

Bommasani et al.

Robustness section: Sang Michael Xie, Ananya Kumar, Rohan Taori, Tony Lee, Pang Wei Koh, Shiori Sagawa, Tatsu Hashimoto

Reasoning section: Yuhuai Wu, Frieda Rong, Hongyu Ren, Sang Michael Xie, Xuechen Li, Andy Shih, Drew A. Hudson, Omar Khattab

Adaptation section: Xiang Lisa Li*, Eric Mitchell*, Sang Michael Xie, Xuechen Li, Tatsunori Hashimoto

Theory section: Aditi Raghunathan, Sang Michael Xie, Ananya Kumar, Niladri Chatterji, Rohan Taori, Tatsunori Hashimoto, Tengyu Ma
Automated detection of skin reactions in epicutaneous patch testing using machine learning [Paper]

British Journal of Dermatology (BJD), 2021.

Warren Chan, R. Srivastava, N. Damaraju, H. Do, G. Burnett, J. MacFarlane, Sang Michael Xie, J.K. Chen, G. Honari, K.Y. Sarin

2016-2020

Understanding and Mitigating the Tradeoff Between Robustness and Accuracy [Paper][Video]

International Conference on Machine Learning (ICML), 2020.

Aditi Raghunathan*, Sang Michael Xie*, Fanny Yang, John C. Duchi, Percy Liang
Weakly supervised deep learning for segmentation of remote sensing imagery [Paper]

Remote Sensing, 2020.

Sherrie Wang, William Chen, Sang Michael Xie, George Azzari, David Lobell
Adversarial Training Can Hurt Generalization [Paper]

ICML Workshop on Identifying and Understanding Deep Learning Phenomena, 2019.

Aditi Raghunathan*, Sang Michael Xie*, Fanny Yang, John C. Duchi, Percy Liang
Reparameterizable Subset Sampling via Continuous Relaxations [Paper]

International Joint Conferences on Artificial Intelligence (IJCAI), 2019.
Sang Michael Xie, Stefano Ermon
Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variance. [Paper]

Advances in Neural Information Processing Systems (NeurIPS), 2018.

Neal Jean*, Sang Michael Xie*, Stefano Ermon
Incorporating Spatial Context and Fine-grained Detail from Satellite Imagery to Predict Poverty [Paper]

Jae Hyun Kim, Michael Xie, Neal Jean, Stefano Ermon
Combining Satellite Imagery and Machine Learning to Predict Poverty. [Video, Maps, Media, and Links] [Paper] [Code] [Top of Mind radio show segment]

Science, 2016.

Neal Jean*, Marshall Burke*, Michael Xie, William Davis, David Lobell, Stefano Ermon
Transfer Learning from Deep Features for Remote Sensing and Poverty Mapping. [Paper] [Oral Presentation] [Stanford Report] [NYTimes]

Association for the Advancement of Artificial Intelligence (AAAI), 2016. Oral Presentation, NVIDIA Global Impact Award Finalist, Scientific American 10 World Changing Ideas of 2016

Michael Xie, Neal Jean, Marshall Burke, David Lobell, Stefano Ermon
Mapping Poverty with Satellite Imagery. [Paper]

Honors Thesis for B.S. with Honors

Michael Xie

Teaching

Course Assistant for CS 324: Understanding and Developing Large Language Models, Winter 2022
Course Assistant for CS 229: Machine Learning, Spring 2022
Section Leader for ENGR 40M: Intro to Making: What is EE, Winter 2015

I have mentored the following amazing undergraduate/master's researchers:

Kendrick Shen, now ML Research Engineer at Genesis Therapeutics
Robbie Jones, now ML Software Engineer at GridSpace
Fahim Tajwar, now PhD Student at CMU
Ben Newman, now PhD Student at University of Washington

Service

I am co-organizing the ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models (ME-FoMo) and the ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models (DPFM).
I co-organized the ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models (ME-FoMo) with Ananya Kumar, Tiffany Vlaar, Yamini Bansal, Mathilde Caron, Tengyu Ma, Hanie Sedghi, Aditi Raghunathan, and Percy Liang.
I participated in a panel discussion with Ludwig Schmidt, Nathan Lambert, and Megan Ansdell at the Data-centric ML Research (DMLR) workshop at ICML 2023.
I have reviewed for NeurIPS (2019, 2020, 2021, 2022, 2023), ICML (2020, 2022, 2023), ICLR (2021, 2022), IEEE SatML 2023, NeurIPS 2022 RobustSeq Workshop, ICML 2022 First Workshop on Pre-Training, ICML 2022 Principles of Distribution Shift (PODS) workshop, the NeurIPS 2021, 2022, 2023 Workshops on Distribution Shifts (DistShift), the Workshop on Computer Vision for Global Challenges (CV4GC) at CVPR 2019.